2024-04-14T13:42:14.051 INFO:root:teuthology version: 0.0.1.dev267+g6c63784 2024-04-14T13:42:14.051 DEBUG:teuthology.run:Teuthology command: teuthology --owner scheduled_teuthology@teuthology --name teuthology-2024-04-08_20:08:14-orch-main-distro-default-smithi --description orch/cephadm/upgrade/{1-start-distro/1-start-centos_9.stream 2-repo_digest/repo_digest 3-upgrade/staggered 4-wait 5-upgrade-ls agent/off mon_election/connectivity} --archive /home/teuthworker/archive/teuthology-2024-04-08_20:08:14-orch-main-distro-default-smithi/7646001 --verbose -- /home/teuthworker/archive/teuthology-2024-04-08_20:08:14-orch-main-distro-default-smithi/7646001/orig.config.yaml 2024-04-14T13:42:14.100 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-04-14T13:42:14.191 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/teuthology-2024-04-08_20:08:14-orch-main-distro-default-smithi/7646001 branch: main description: orch/cephadm/upgrade/{1-start-distro/1-start-centos_9.stream 2-repo_digest/repo_digest 3-upgrade/staggered 4-wait 5-upgrade-ls agent/off mon_election/connectivity} email: ceph-qa@ceph.com first_in_suite: false job_id: '7646001' kernel: kdb: 1 sha1: distro last_in_suite: false machine_type: smithi name: teuthology-2024-04-08_20:08:14-orch-main-distro-default-smithi no_nested_subset: false nuke-on-error: true os_type: centos os_version: 9.stream overrides: admin_socket: branch: main ceph: conf: global: mon election default strategy: 3 mgr: debug mgr: 20 debug ms: 1 mgr/cephadm/use_agent: false mon: debug mon: 20 debug ms: 1 debug paxos: 20 osd: debug ms: 1 debug osd: 20 flavor: default log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) sha1: a971fd5f5719038c70b817bafacd6d5400b1d888 ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} install: ceph: flavor: default sha1: a971fd5f5719038c70b817bafacd6d5400b1d888 workunit: branch: main sha1: 05c9d46e7de5db2cb434c177a6bbfc56f472d09b owner: scheduled_teuthology@teuthology priority: 950 repo: https://git.ceph.com/ceph.git roles: - - mon.a - mon.c - mgr.y - osd.0 - osd.1 - osd.2 - osd.3 - client.0 - node-exporter.a - alertmanager.a - - mon.b - mgr.x - osd.4 - osd.5 - osd.6 - osd.7 - client.1 - prometheus.a - grafana.a - node-exporter.b seed: 6509 sha1: a971fd5f5719038c70b817bafacd6d5400b1d888 sleep_before_teardown: 0 subset: 21/64 suite: orch suite_branch: main suite_path: /home/teuthworker/src/git.ceph.com_ceph_05c9d46e7de5db2cb434c177a6bbfc56f472d09b/qa suite_relpath: qa suite_repo: https://git.ceph.com/ceph.git suite_sha1: 05c9d46e7de5db2cb434c177a6bbfc56f472d09b targets: smithi069.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBJQaGbtEJpZRB1Ob754T3lFCfEKlgZJ4BF5lU+jqhlWbWzgg878IXsm4Cg2WJdYA925nZ+I/soQQwsXyHC/Vu9g= smithi122.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBHlnOxRm8xz7RAh1WohUOHVd7Uuy9q8Gxh3Mg8872snOHa/PP/YXWGJrm+Jtza+hn+pfOAX7nfikFSM/fKmXzfA= tasks: - cephadm: cephadm_branch: v17.2.0 cephadm_git_url: https://github.com/ceph/ceph image: quay.io/ceph/ceph:v17.2.0 - cephadm.shell: mon.a: - ceph config set mgr mgr/cephadm/use_repo_digest false --force - cephadm.shell: env: - sha1 mon.a: - radosgw-admin realm create --rgw-realm=r --default - radosgw-admin zonegroup create --rgw-zonegroup=default --master --default - radosgw-admin zone create --rgw-zonegroup=default --rgw-zone=z --master --default - radosgw-admin period update --rgw-realm=r --commit - ceph orch apply rgw foo --realm r --zone z --placement=2 --port=8000 - ceph osd pool create foo - rbd pool init foo - ceph orch apply iscsi foo u p - sleep 180 - ceph config set mon mon_warn_on_insecure_global_id_reclaim false --force - ceph config set mon mon_warn_on_insecure_global_id_reclaim_allowed false --force - ceph config set global log_to_journald false --force - ceph orch ps - ceph versions - ceph -s - ceph orch ls - ceph orch daemon redeploy "mgr.$(ceph mgr dump -f json | jq .standbys | jq .[] | jq -r .name)" --image quay.ceph.io/ceph-ci/ceph:$sha1 - ceph orch ps --refresh - sleep 180 - ceph orch ps - ceph versions - ceph -s - ceph health detail - ceph versions | jq -e '.mgr | length == 2' - ceph mgr fail - sleep 180 - ceph orch daemon redeploy "mgr.$(ceph mgr dump -f json | jq .standbys | jq .[] | jq -r .name)" --image quay.ceph.io/ceph-ci/ceph:$sha1 - ceph orch ps --refresh - sleep 180 - ceph orch ps - ceph versions - ceph health detail - ceph -s - ceph mgr fail - sleep 180 - ceph orch ps - ceph versions - ceph -s - ceph health detail - ceph versions | jq -e '.mgr | length == 1' - ceph mgr fail - sleep 180 - ceph orch ps - ceph orch ls - ceph versions - ceph orch upgrade start --image quay.ceph.io/ceph-ci/ceph:$sha1 --daemon-types mgr - while ceph orch upgrade status | jq '.in_progress' | grep true && ! ceph orch upgrade status | jq '.message' | grep Error ; do ceph orch ps ; ceph versions ; ceph orch upgrade status ; sleep 30 ; done - ceph versions | jq -e '.mgr | length == 1' - ceph versions | jq -e '.mgr | keys' | grep $sha1 - ceph versions | jq -e '.overall | length == 2' - ceph orch upgrade check quay.ceph.io/ceph-ci/ceph:$sha1 | jq -e '.up_to_date | length == 2' - ceph orch upgrade status - ceph health detail - ceph orch upgrade start --image quay.ceph.io/ceph-ci/ceph:$sha1 --daemon-types mon --hosts $(ceph orch ps | grep mgr.x | awk '{print $2}') - while ceph orch upgrade status | jq '.in_progress' | grep true && ! ceph orch upgrade status | jq '.message' | grep Error ; do ceph orch ps ; ceph versions ; ceph orch upgrade status ; sleep 30 ; done - ceph orch ps - ceph versions | jq -e '.mon | length == 2' - ceph orch upgrade status - ceph health detail - ceph orch upgrade start --image quay.ceph.io/ceph-ci/ceph:$sha1 --daemon-types mon --hosts $(ceph orch ps | grep mgr.y | awk '{print $2}') - while ceph orch upgrade status | jq '.in_progress' | grep true && ! ceph orch upgrade status | jq '.message' | grep Error ; do ceph orch ps ; ceph versions ; ceph orch upgrade status ; sleep 30 ; done - ceph orch ps - ceph versions | jq -e '.mon | length == 1' - ceph versions | jq -e '.mon | keys' | grep $sha1 - ceph orch upgrade check quay.ceph.io/ceph-ci/ceph:$sha1 | jq -e '.up_to_date | length == 5' - ceph orch upgrade status - ceph health detail - ceph orch upgrade start --image quay.ceph.io/ceph-ci/ceph:$sha1 --daemon-types osd --limit 2 - while ceph orch upgrade status | jq '.in_progress' | grep true && ! ceph orch upgrade status | jq '.message' | grep Error ; do ceph orch ps ; ceph versions ; ceph orch upgrade status ; sleep 30 ; done - ceph orch ps - ceph versions | jq -e '.osd | length == 2' - ceph orch upgrade check quay.ceph.io/ceph-ci/ceph:$sha1 | jq -e '.up_to_date | length == 7' - ceph orch upgrade status - ceph health detail - ceph orch upgrade start --image quay.ceph.io/ceph-ci/ceph:$sha1 --daemon-types crash,osd --limit 1 - while ceph orch upgrade status | jq '.in_progress' | grep true && ! ceph orch upgrade status | jq '.message' | grep Error ; do ceph orch ps ; ceph versions ; ceph orch upgrade status ; sleep 30 ; done - ceph orch ps - ceph versions | jq -e '.osd | length == 2' - ceph orch upgrade check quay.ceph.io/ceph-ci/ceph:$sha1 | jq -e '.up_to_date | length == 8' - ceph orch upgrade status - ceph health detail - ceph orch upgrade start --image quay.ceph.io/ceph-ci/ceph:$sha1 --daemon-types crash,osd - while ceph orch upgrade status | jq '.in_progress' | grep true && ! ceph orch upgrade status | jq '.message' | grep Error ; do ceph orch ps ; ceph versions ; ceph orch upgrade status ; sleep 30 ; done - ceph orch ps - ceph versions | jq -e '.osd | length == 1' - ceph versions | jq -e '.osd | keys' | grep $sha1 - ceph orch upgrade status - ceph health detail - ceph orch upgrade start --image quay.ceph.io/ceph-ci/ceph:$sha1 --services rgw.foo - while ceph orch upgrade status | jq '.in_progress' | grep true && ! ceph orch upgrade status | jq '.message' | grep Error ; do ceph orch ps ; ceph versions ; ceph orch upgrade status ; sleep 30 ; done - ceph orch ps - ceph versions | jq -e '.rgw | length == 1' - ceph versions | jq -e '.rgw | keys' | grep $sha1 - ceph orch upgrade status - ceph health detail - ceph orch upgrade start --image quay.ceph.io/ceph-ci/ceph:$sha1 - cephadm.shell: env: - sha1 mon.a: - while ceph orch upgrade status | jq '.in_progress' | grep true && ! ceph orch upgrade status | jq '.message' | grep Error ; do ceph orch ps ; ceph versions ; ceph orch upgrade status ; ceph health detail ; sleep 30 ; done - ceph orch ps - ceph versions - echo "wait for servicemap items w/ changing names to refresh" - sleep 60 - ceph orch ps - ceph versions - ceph orch upgrade status - ceph health detail - ceph versions | jq -e '.overall | length == 1' - ceph versions | jq -e '.overall | keys' | grep $sha1 - ceph orch ls | grep '^osd ' - cephadm.shell: mon.a: - ceph orch upgrade ls - ceph orch upgrade ls --image quay.io/ceph/ceph --show-all-versions | grep 16.2.0 - ceph orch upgrade ls --image quay.io/ceph/ceph --tags | grep v16.2.2 teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: 6c637841c215537a4502385240412f1966e0faab timestamp: 2024-04-08_20:08:14 tube: smithi user: teuthology verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885 2024-04-14T13:42:14.192 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph_05c9d46e7de5db2cb434c177a6bbfc56f472d09b/qa; will attempt to use it 2024-04-14T13:42:14.192 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph_05c9d46e7de5db2cb434c177a6bbfc56f472d09b/qa/tasks 2024-04-14T13:42:14.192 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-04-14T13:42:14.194 INFO:teuthology.task.internal:Checking packages... 2024-04-14T13:42:14.313 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash 'a971fd5f5719038c70b817bafacd6d5400b1d888' 2024-04-14T13:42:14.314 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-04-14T13:42:14.314 INFO:teuthology.packaging:ref: None 2024-04-14T13:42:14.314 INFO:teuthology.packaging:tag: None 2024-04-14T13:42:14.314 INFO:teuthology.packaging:branch: main 2024-04-14T13:42:14.314 INFO:teuthology.packaging:sha1: a971fd5f5719038c70b817bafacd6d5400b1d888 2024-04-14T13:42:14.314 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=main 2024-04-14T13:42:14.625 INFO:teuthology.task.internal:Found packages for ceph version 19.0.0-3080.gc2d8c315 2024-04-14T13:42:14.628 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-04-14T13:42:14.634 INFO:teuthology.task.internal:no buildpackages task found 2024-04-14T13:42:14.635 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-04-14T13:42:14.661 INFO:teuthology.task.internal:Saving configuration 2024-04-14T13:42:14.678 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-04-14T13:42:14.689 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-04-14T13:42:14.709 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi069.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-04-08_20:08:14-orch-main-distro-default-smithi/7646001', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-14 13:37:21.333171', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBJQaGbtEJpZRB1Ob754T3lFCfEKlgZJ4BF5lU+jqhlWbWzgg878IXsm4Cg2WJdYA925nZ+I/soQQwsXyHC/Vu9g='} 2024-04-14T13:42:14.731 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi122.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-04-08_20:08:14-orch-main-distro-default-smithi/7646001', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-14 13:37:21.334942', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBHlnOxRm8xz7RAh1WohUOHVd7Uuy9q8Gxh3Mg8872snOHa/PP/YXWGJrm+Jtza+hn+pfOAX7nfikFSM/fKmXzfA='} 2024-04-14T13:42:14.731 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-04-14T13:42:14.740 INFO:teuthology.task.internal:roles: ubuntu@smithi069.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a', 'alertmanager.a'] 2024-04-14T13:42:14.740 INFO:teuthology.task.internal:roles: ubuntu@smithi122.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'grafana.a', 'node-exporter.b'] 2024-04-14T13:42:14.740 INFO:teuthology.run_tasks:Running task console_log... 2024-04-14T13:42:14.823 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7ff554d4e0d0>, signals=[15]) 2024-04-14T13:42:14.824 INFO:teuthology.run_tasks:Running task internal.connect... 2024-04-14T13:42:14.835 INFO:teuthology.task.internal:Opening connections... 2024-04-14T13:42:14.835 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi069.front.sepia.ceph.com 2024-04-14T13:42:14.837 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi069.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-14T13:42:14.929 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi122.front.sepia.ceph.com 2024-04-14T13:42:14.930 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi122.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-14T13:42:15.035 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-04-14T13:42:15.046 DEBUG:teuthology.orchestra.run.smithi069:> uname -m 2024-04-14T13:42:15.064 INFO:teuthology.orchestra.run.smithi069.stdout:x86_64 2024-04-14T13:42:15.064 DEBUG:teuthology.orchestra.run.smithi069:> cat /etc/os-release 2024-04-14T13:42:15.120 INFO:teuthology.orchestra.run.smithi069.stdout:NAME="CentOS Stream" 2024-04-14T13:42:15.120 INFO:teuthology.orchestra.run.smithi069.stdout:VERSION="9" 2024-04-14T13:42:15.121 INFO:teuthology.orchestra.run.smithi069.stdout:ID="centos" 2024-04-14T13:42:15.121 INFO:teuthology.orchestra.run.smithi069.stdout:ID_LIKE="rhel fedora" 2024-04-14T13:42:15.121 INFO:teuthology.orchestra.run.smithi069.stdout:VERSION_ID="9" 2024-04-14T13:42:15.121 INFO:teuthology.orchestra.run.smithi069.stdout:PLATFORM_ID="platform:el9" 2024-04-14T13:42:15.121 INFO:teuthology.orchestra.run.smithi069.stdout:PRETTY_NAME="CentOS Stream 9" 2024-04-14T13:42:15.121 INFO:teuthology.orchestra.run.smithi069.stdout:ANSI_COLOR="0;31" 2024-04-14T13:42:15.121 INFO:teuthology.orchestra.run.smithi069.stdout:LOGO="fedora-logo-icon" 2024-04-14T13:42:15.121 INFO:teuthology.orchestra.run.smithi069.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-04-14T13:42:15.121 INFO:teuthology.orchestra.run.smithi069.stdout:HOME_URL="https://centos.org/" 2024-04-14T13:42:15.121 INFO:teuthology.orchestra.run.smithi069.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-04-14T13:42:15.121 INFO:teuthology.orchestra.run.smithi069.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-04-14T13:42:15.121 INFO:teuthology.orchestra.run.smithi069.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-04-14T13:42:15.122 INFO:teuthology.lock.ops:Updating smithi069.front.sepia.ceph.com on lock server 2024-04-14T13:42:15.145 DEBUG:teuthology.orchestra.run.smithi122:> uname -m 2024-04-14T13:42:15.162 INFO:teuthology.orchestra.run.smithi122.stdout:x86_64 2024-04-14T13:42:15.162 DEBUG:teuthology.orchestra.run.smithi122:> cat /etc/os-release 2024-04-14T13:42:15.218 INFO:teuthology.orchestra.run.smithi122.stdout:NAME="CentOS Stream" 2024-04-14T13:42:15.218 INFO:teuthology.orchestra.run.smithi122.stdout:VERSION="9" 2024-04-14T13:42:15.219 INFO:teuthology.orchestra.run.smithi122.stdout:ID="centos" 2024-04-14T13:42:15.219 INFO:teuthology.orchestra.run.smithi122.stdout:ID_LIKE="rhel fedora" 2024-04-14T13:42:15.219 INFO:teuthology.orchestra.run.smithi122.stdout:VERSION_ID="9" 2024-04-14T13:42:15.219 INFO:teuthology.orchestra.run.smithi122.stdout:PLATFORM_ID="platform:el9" 2024-04-14T13:42:15.219 INFO:teuthology.orchestra.run.smithi122.stdout:PRETTY_NAME="CentOS Stream 9" 2024-04-14T13:42:15.219 INFO:teuthology.orchestra.run.smithi122.stdout:ANSI_COLOR="0;31" 2024-04-14T13:42:15.219 INFO:teuthology.orchestra.run.smithi122.stdout:LOGO="fedora-logo-icon" 2024-04-14T13:42:15.219 INFO:teuthology.orchestra.run.smithi122.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-04-14T13:42:15.219 INFO:teuthology.orchestra.run.smithi122.stdout:HOME_URL="https://centos.org/" 2024-04-14T13:42:15.219 INFO:teuthology.orchestra.run.smithi122.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-04-14T13:42:15.219 INFO:teuthology.orchestra.run.smithi122.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-04-14T13:42:15.219 INFO:teuthology.orchestra.run.smithi122.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-04-14T13:42:15.220 INFO:teuthology.lock.ops:Updating smithi122.front.sepia.ceph.com on lock server 2024-04-14T13:42:15.236 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-04-14T13:42:15.248 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-04-14T13:42:15.292 INFO:teuthology.task.internal:Checking for old test directory... 2024-04-14T13:42:15.292 DEBUG:teuthology.orchestra.run.smithi069:> test '!' -e /home/ubuntu/cephtest 2024-04-14T13:42:15.295 DEBUG:teuthology.orchestra.run.smithi122:> test '!' -e /home/ubuntu/cephtest 2024-04-14T13:42:15.310 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-04-14T13:42:15.318 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-04-14T13:42:15.318 DEBUG:teuthology.orchestra.run.smithi069:> test -z $(ls -A /var/lib/ceph) 2024-04-14T13:42:15.352 DEBUG:teuthology.orchestra.run.smithi122:> test -z $(ls -A /var/lib/ceph) 2024-04-14T13:42:15.462 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-04-14T13:42:15.516 INFO:teuthology.run_tasks:Running task kernel... 2024-04-14T13:42:15.532 INFO:teuthology.task.kernel:normalize config orig: {'kdb': 1, 'sha1': 'distro'} 2024-04-14T13:42:15.533 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': 1, 'sha1': 'distro'}, 'mon.b': {'kdb': 1, 'sha1': 'distro'}}, timeout 300 2024-04-14T13:42:15.533 DEBUG:teuthology.orchestra.run.smithi069:> test -f /run/.containerenv -o -f /.dockerenv 2024-04-14T13:42:15.533 DEBUG:teuthology.orchestra.run.smithi122:> test -f /run/.containerenv -o -f /.dockerenv 2024-04-14T13:42:15.549 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T13:42:15.549 DEBUG:teuthology.orchestra.run.smithi122:> uname -r 2024-04-14T13:42:15.550 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T13:42:15.550 DEBUG:teuthology.orchestra.run.smithi069:> uname -r 2024-04-14T13:42:15.605 INFO:teuthology.orchestra.run.smithi122.stdout:5.14.0-361.el9.x86_64 2024-04-14T13:42:15.606 INFO:teuthology.task.kernel:Running kernel on smithi122: 5.14.0-361.el9.x86_64 2024-04-14T13:42:15.606 DEBUG:teuthology.orchestra.run.smithi122:> sudo yum install -y kernel 2024-04-14T13:42:15.608 INFO:teuthology.orchestra.run.smithi069.stdout:5.14.0-361.el9.x86_64 2024-04-14T13:42:15.608 INFO:teuthology.task.kernel:Running kernel on smithi069: 5.14.0-361.el9.x86_64 2024-04-14T13:42:15.608 DEBUG:teuthology.orchestra.run.smithi069:> sudo yum install -y kernel 2024-04-14T13:42:17.842 INFO:teuthology.orchestra.run.smithi069.stdout:CentOS-9 - AppStream 14 MB/s | 8.1 MB 00:00 2024-04-14T13:42:17.863 INFO:teuthology.orchestra.run.smithi122.stdout:CentOS-9 - AppStream 13 MB/s | 8.1 MB 00:00 2024-04-14T13:42:19.643 INFO:teuthology.orchestra.run.smithi069.stdout:CentOS-9 - BaseOS 5.5 MB/s | 2.5 MB 00:00 2024-04-14T13:42:19.796 INFO:teuthology.orchestra.run.smithi122.stdout:CentOS-9 - BaseOS 4.0 MB/s | 2.5 MB 00:00 2024-04-14T13:42:20.625 INFO:teuthology.orchestra.run.smithi069.stdout:CentOS Stream 9 - BaseOS 15 MB/s | 8.0 MB 00:00 2024-04-14T13:42:20.737 INFO:teuthology.orchestra.run.smithi122.stdout:CentOS Stream 9 - BaseOS 16 MB/s | 8.0 MB 00:00 2024-04-14T13:42:22.301 INFO:teuthology.orchestra.run.smithi069.stdout:CentOS Stream 9 - AppStream 33 MB/s | 19 MB 00:00 2024-04-14T13:42:22.614 INFO:teuthology.orchestra.run.smithi122.stdout:CentOS Stream 9 - AppStream 24 MB/s | 19 MB 00:00 2024-04-14T13:42:26.593 INFO:teuthology.orchestra.run.smithi069.stdout:CentOS Stream 9 - CRB 8.5 MB/s | 6.2 MB 00:00 2024-04-14T13:42:28.269 INFO:teuthology.orchestra.run.smithi069.stdout:CentOS Stream 9 - Extras packages 33 kB/s | 16 kB 00:00 2024-04-14T13:42:30.323 INFO:teuthology.orchestra.run.smithi069.stdout:Extra Packages for Enterprise Linux 11 MB/s | 21 MB 00:01 2024-04-14T13:42:34.389 INFO:teuthology.orchestra.run.smithi122.stdout:CentOS Stream 9 - CRB 793 kB/s | 6.2 MB 00:07 2024-04-14T13:42:35.437 INFO:teuthology.orchestra.run.smithi069.stdout:lab-extras 11 kB/s | 1.7 kB 00:00 2024-04-14T13:42:35.966 INFO:teuthology.orchestra.run.smithi122.stdout:CentOS Stream 9 - Extras packages 52 kB/s | 16 kB 00:00 2024-04-14T13:42:37.703 INFO:teuthology.orchestra.run.smithi069.stderr:Modular dependency problems: 2024-04-14T13:42:37.703 INFO:teuthology.orchestra.run.smithi069.stderr: 2024-04-14T13:42:37.703 INFO:teuthology.orchestra.run.smithi069.stderr: Problem 1: conflicting requests 2024-04-14T13:42:37.703 INFO:teuthology.orchestra.run.smithi069.stderr: - nothing provides module(platform:el8) needed by module container-tools:rhel8:820240321091303:20125149.x86_64 from CentOS-AppStream 2024-04-14T13:42:37.703 INFO:teuthology.orchestra.run.smithi069.stderr: Problem 2: conflicting requests 2024-04-14T13:42:37.703 INFO:teuthology.orchestra.run.smithi069.stderr: - nothing provides module(platform:el8) needed by module eclipse:rhel8:820201023100746:b230ed4e.x86_64 from CentOS-AppStream 2024-04-14T13:42:37.703 INFO:teuthology.orchestra.run.smithi069.stderr: Problem 3: conflicting requests 2024-04-14T13:42:37.703 INFO:teuthology.orchestra.run.smithi069.stderr: - nothing provides module(platform:el8) needed by module freeradius:3.0:8080020230117180605:89170a74.x86_64 from CentOS-AppStream 2024-04-14T13:42:37.703 INFO:teuthology.orchestra.run.smithi069.stderr: Problem 4: conflicting requests 2024-04-14T13:42:37.704 INFO:teuthology.orchestra.run.smithi069.stderr: - nothing provides module(platform:el8) needed by module gimp:2.8:8000020190628145146:4148dfdf.x86_64 from CentOS-AppStream 2024-04-14T13:42:37.704 INFO:teuthology.orchestra.run.smithi069.stderr: Problem 5: conflicting requests 2024-04-14T13:42:37.704 INFO:teuthology.orchestra.run.smithi069.stderr: - nothing provides module(platform:el8) needed by module go-toolset:rhel8:820240401160413:b754926a.x86_64 from CentOS-AppStream 2024-04-14T13:42:37.704 INFO:teuthology.orchestra.run.smithi069.stderr: Problem 6: conflicting requests 2024-04-14T13:42:37.704 INFO:teuthology.orchestra.run.smithi069.stderr: - nothing provides module(platform:el8) needed by module httpd:2.4:820240216084734:9edba152.x86_64 from CentOS-AppStream 2024-04-14T13:42:37.704 INFO:teuthology.orchestra.run.smithi069.stderr: Problem 7: conflicting requests 2024-04-14T13:42:37.704 INFO:teuthology.orchestra.run.smithi069.stderr: - nothing provides module(platform:el8) needed by module idm:client:820240307184541:49cc9d1b.x86_64 from CentOS-AppStream 2024-04-14T13:42:37.704 INFO:teuthology.orchestra.run.smithi069.stderr: Problem 8: conflicting requests 2024-04-14T13:42:37.704 INFO:teuthology.orchestra.run.smithi069.stderr: - nothing provides module(platform:el8) needed by module inkscape:0.92.3:8080020221205124429:3e031279.x86_64 from CentOS-AppStream 2024-04-14T13:42:37.704 INFO:teuthology.orchestra.run.smithi069.stderr: Problem 9: conflicting requests 2024-04-14T13:42:37.704 INFO:teuthology.orchestra.run.smithi069.stderr: - nothing provides module(platform:el8) needed by module jmc:rhel8:8050020211110222101:6392b1f8.x86_64 from CentOS-AppStream 2024-04-14T13:42:37.704 INFO:teuthology.orchestra.run.smithi069.stderr: Problem 10: conflicting requests 2024-04-14T13:42:37.704 INFO:teuthology.orchestra.run.smithi069.stderr: - nothing provides module(platform:el8) needed by module llvm-toolset:rhel8:820240205133135:9edba152.x86_64 from CentOS-AppStream 2024-04-14T13:42:37.704 INFO:teuthology.orchestra.run.smithi069.stderr: Problem 11: conflicting requests 2024-04-14T13:42:37.704 INFO:teuthology.orchestra.run.smithi069.stderr: - nothing provides module(platform:el8) needed by module log4j:2:8080020221205124743:9d367344.x86_64 from CentOS-AppStream 2024-04-14T13:42:37.705 INFO:teuthology.orchestra.run.smithi069.stderr: Problem 12: conflicting requests 2024-04-14T13:42:37.705 INFO:teuthology.orchestra.run.smithi069.stderr: - nothing provides module(platform:el8) needed by module mailman:2.1:820230727085757:77fc8825.x86_64 from CentOS-AppStream 2024-04-14T13:42:37.705 INFO:teuthology.orchestra.run.smithi069.stderr: Problem 13: conflicting requests 2024-04-14T13:42:37.705 INFO:teuthology.orchestra.run.smithi069.stderr: - nothing provides module(platform:el8) needed by module mariadb:10.3:8030020210419150013:30b713e6.x86_64 from CentOS-AppStream 2024-04-14T13:42:37.705 INFO:teuthology.orchestra.run.smithi069.stderr: Problem 14: conflicting requests 2024-04-14T13:42:37.705 INFO:teuthology.orchestra.run.smithi069.stderr: - nothing provides module(platform:el8) needed by module maven:3.5:8060020211203142015:c0229ad2.x86_64 from CentOS-AppStream 2024-04-14T13:42:37.705 INFO:teuthology.orchestra.run.smithi069.stderr: Problem 15: conflicting requests 2024-04-14T13:42:37.705 INFO:teuthology.orchestra.run.smithi069.stderr: - nothing provides module(platform:el8) needed by module mercurial:4.8:8000020190628020724:4148dfdf.x86_64 from CentOS-AppStream 2024-04-14T13:42:37.705 INFO:teuthology.orchestra.run.smithi069.stderr: Problem 16: conflicting requests 2024-04-14T13:42:37.705 INFO:teuthology.orchestra.run.smithi069.stderr: - nothing provides module(platform:el8) needed by module mod_auth_openidc:2.3:820230721123832:9edba152.x86_64 from CentOS-AppStream 2024-04-14T13:42:37.705 INFO:teuthology.orchestra.run.smithi069.stderr: Problem 17: conflicting requests 2024-04-14T13:42:37.705 INFO:teuthology.orchestra.run.smithi069.stderr: - nothing provides module(platform:el8) needed by module mysql:8.0:8040020210901180257:9f9e2e7e.x86_64 from CentOS-AppStream 2024-04-14T13:42:37.705 INFO:teuthology.orchestra.run.smithi069.stderr: Problem 18: conflicting requests 2024-04-14T13:42:37.705 INFO:teuthology.orchestra.run.smithi069.stderr: - nothing provides module(platform:el8) needed by module nginx:1.14:8000020211221191913:55190bc5.x86_64 from CentOS-AppStream 2024-04-14T13:42:37.706 INFO:teuthology.orchestra.run.smithi069.stderr: Problem 19: conflicting requests 2024-04-14T13:42:37.706 INFO:teuthology.orchestra.run.smithi069.stderr: - nothing provides module(platform:el8) needed by module nodejs:10:8040020210120182536:9f9e2e7e.x86_64 from CentOS-AppStream 2024-04-14T13:42:37.706 INFO:teuthology.orchestra.run.smithi069.stderr: Problem 20: conflicting requests 2024-04-14T13:42:37.706 INFO:teuthology.orchestra.run.smithi069.stderr: - nothing provides module(platform:el8) needed by module perl:5.26:8000020190628020724:55190bc5.x86_64 from CentOS-AppStream 2024-04-14T13:42:37.706 INFO:teuthology.orchestra.run.smithi069.stderr: Problem 21: conflicting requests 2024-04-14T13:42:37.706 INFO:teuthology.orchestra.run.smithi069.stderr: - nothing provides module(platform:el8) needed by module php:7.2:8020020200507003613:2c7ca891.x86_64 from CentOS-AppStream 2024-04-14T13:42:37.706 INFO:teuthology.orchestra.run.smithi069.stderr: Problem 22: conflicting requests 2024-04-14T13:42:37.706 INFO:teuthology.orchestra.run.smithi069.stderr: - nothing provides module(platform:el8) needed by module postgresql:10:820230223114800:9edba152.x86_64 from CentOS-AppStream 2024-04-14T13:42:37.706 INFO:teuthology.orchestra.run.smithi069.stderr: Problem 23: conflicting requests 2024-04-14T13:42:37.706 INFO:teuthology.orchestra.run.smithi069.stderr: - nothing provides module(platform:el8) needed by module python27:2.7:820240208011952:182f7c73.x86_64 from CentOS-AppStream 2024-04-14T13:42:37.706 INFO:teuthology.orchestra.run.smithi069.stderr: Problem 24: conflicting requests 2024-04-14T13:42:37.706 INFO:teuthology.orchestra.run.smithi069.stderr: - nothing provides module(platform:el8) needed by module python36:3.6:8050020210825152031:982725ab.x86_64 from CentOS-AppStream 2024-04-14T13:42:37.706 INFO:teuthology.orchestra.run.smithi069.stderr: Problem 25: conflicting requests 2024-04-14T13:42:37.706 INFO:teuthology.orchestra.run.smithi069.stderr: - nothing provides module(platform:el8) needed by module python38:3.8:820230810143931:64c1cd5a.x86_64 from CentOS-AppStream 2024-04-14T13:42:37.706 INFO:teuthology.orchestra.run.smithi069.stderr: Problem 26: conflicting requests 2024-04-14T13:42:37.707 INFO:teuthology.orchestra.run.smithi069.stderr: - nothing provides module(platform:el8) needed by module python39:3.9:820240214182535:17377f89.x86_64 from CentOS-AppStream 2024-04-14T13:42:37.707 INFO:teuthology.orchestra.run.smithi069.stderr: Problem 27: conflicting requests 2024-04-14T13:42:37.707 INFO:teuthology.orchestra.run.smithi069.stderr: - nothing provides module(platform:el8) needed by module redis:5:8040020211019153849:9f9e2e7e.x86_64 from CentOS-AppStream 2024-04-14T13:42:37.707 INFO:teuthology.orchestra.run.smithi069.stderr: Problem 28: conflicting requests 2024-04-14T13:42:37.707 INFO:teuthology.orchestra.run.smithi069.stderr: - nothing provides module(platform:el8) needed by module rhn-tools:1.0:8010020191114034948:f69d1239.x86_64 from CentOS-AppStream 2024-04-14T13:42:37.707 INFO:teuthology.orchestra.run.smithi069.stderr: Problem 29: conflicting requests 2024-04-14T13:42:37.707 INFO:teuthology.orchestra.run.smithi069.stderr: - nothing provides module(platform:el8) needed by module ruby:2.5:820230627084142:9edba152.x86_64 from CentOS-AppStream 2024-04-14T13:42:37.707 INFO:teuthology.orchestra.run.smithi069.stderr: Problem 30: conflicting requests 2024-04-14T13:42:37.707 INFO:teuthology.orchestra.run.smithi069.stderr: - nothing provides module(platform:el8) needed by module rust-toolset:rhel8:820240119204620:b09eea91.x86_64 from CentOS-AppStream 2024-04-14T13:42:37.707 INFO:teuthology.orchestra.run.smithi069.stderr: Problem 31: conflicting requests 2024-04-14T13:42:37.707 INFO:teuthology.orchestra.run.smithi069.stderr: - nothing provides module(platform:el8) needed by module satellite-5-client:1.0:8010020191114035551:cdc1202b.x86_64 from CentOS-AppStream 2024-04-14T13:42:37.707 INFO:teuthology.orchestra.run.smithi069.stderr: Problem 32: conflicting requests 2024-04-14T13:42:37.707 INFO:teuthology.orchestra.run.smithi069.stderr: - nothing provides module(platform:el8) needed by module squid:4:820240319173245:9edba152.x86_64 from CentOS-AppStream 2024-04-14T13:42:37.707 INFO:teuthology.orchestra.run.smithi069.stderr: Problem 33: conflicting requests 2024-04-14T13:42:37.707 INFO:teuthology.orchestra.run.smithi069.stderr: - nothing provides module(platform:el8) needed by module subversion:1.10:8070020220711155714:78111232.x86_64 from CentOS-AppStream 2024-04-14T13:42:37.708 INFO:teuthology.orchestra.run.smithi069.stderr: Problem 34: conflicting requests 2024-04-14T13:42:37.708 INFO:teuthology.orchestra.run.smithi069.stderr: - nothing provides module(platform:el8) needed by module swig:3.0:8030020200706142531:30b713e6.x86_64 from CentOS-AppStream 2024-04-14T13:42:37.708 INFO:teuthology.orchestra.run.smithi069.stderr: Problem 35: conflicting requests 2024-04-14T13:42:37.708 INFO:teuthology.orchestra.run.smithi069.stderr: - nothing provides module(platform:el8) needed by module varnish:6:8050020211109225449:b4937e53.x86_64 from CentOS-AppStream 2024-04-14T13:42:37.708 INFO:teuthology.orchestra.run.smithi069.stderr: Problem 36: conflicting requests 2024-04-14T13:42:37.708 INFO:teuthology.orchestra.run.smithi069.stderr: - nothing provides module(platform:el8) needed by module virt:rhel:820240314161907:9edba152.x86_64 from CentOS-AppStream 2024-04-14T13:42:37.708 INFO:teuthology.orchestra.run.smithi069.stderr: Problem 37: conflicting requests 2024-04-14T13:42:37.708 INFO:teuthology.orchestra.run.smithi069.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:8030020200716155257:7cc0a66d.x86_64 from CentOS-AppStream 2024-04-14T13:42:37.708 INFO:teuthology.orchestra.run.smithi069.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:8030020200716155257:b967a9a2.x86_64 from CentOS-AppStream 2024-04-14T13:42:37.708 INFO:teuthology.orchestra.run.smithi069.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:820230811133638:36f9fae6.x86_64 from CentOS-AppStream 2024-04-14T13:42:37.708 INFO:teuthology.orchestra.run.smithi069.stderr: Problem 38: cannot install the best candidate for the job 2024-04-14T13:42:37.708 INFO:teuthology.orchestra.run.smithi069.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:8010020191114031501:a5949e2e.x86_64 from CentOS-AppStream 2024-04-14T13:42:37.708 INFO:teuthology.orchestra.run.smithi069.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:8010020191114031501:be2e4737.x86_64 from CentOS-AppStream 2024-04-14T13:42:37.708 INFO:teuthology.orchestra.run.smithi069.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:820230223114759:2ab761e1.x86_64 from CentOS-AppStream 2024-04-14T13:42:37.708 INFO:teuthology.orchestra.run.smithi069.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:820230223114759:86e45846.x86_64 from CentOS-AppStream 2024-04-14T13:42:37.709 INFO:teuthology.orchestra.run.smithi069.stderr: Problem 39: conflicting requests 2024-04-14T13:42:37.709 INFO:teuthology.orchestra.run.smithi069.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:8030020200715230104:1e4bbb35.x86_64 from CentOS-AppStream 2024-04-14T13:42:37.709 INFO:teuthology.orchestra.run.smithi069.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:8030020200715230104:ea09926d.x86_64 from CentOS-AppStream 2024-04-14T13:42:37.709 INFO:teuthology.orchestra.run.smithi069.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:082fdf2f.x86_64 from CentOS-AppStream 2024-04-14T13:42:37.709 INFO:teuthology.orchestra.run.smithi069.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-04-14T13:42:37.709 INFO:teuthology.orchestra.run.smithi069.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-04-14T13:42:37.709 INFO:teuthology.orchestra.run.smithi069.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:fbe42456.x86_64 from CentOS-AppStream 2024-04-14T13:42:37.709 INFO:teuthology.orchestra.run.smithi069.stderr: Problem 40: cannot install the best candidate for the job 2024-04-14T13:42:37.709 INFO:teuthology.orchestra.run.smithi069.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:8010020191114031513:16b3ab4d.x86_64 from CentOS-AppStream 2024-04-14T13:42:37.709 INFO:teuthology.orchestra.run.smithi069.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:8010020191114031513:e1f37755.x86_64 from CentOS-AppStream 2024-04-14T13:42:37.709 INFO:teuthology.orchestra.run.smithi069.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-04-14T13:42:37.709 INFO:teuthology.orchestra.run.smithi069.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-04-14T13:42:37.709 INFO:teuthology.orchestra.run.smithi069.stderr: Problem 41: cannot install the best candidate for the job 2024-04-14T13:42:37.709 INFO:teuthology.orchestra.run.smithi069.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:8030020200716150652:1e4bbb35.x86_64 from CentOS-AppStream 2024-04-14T13:42:37.710 INFO:teuthology.orchestra.run.smithi069.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:8030020200716150652:ea09926d.x86_64 from CentOS-AppStream 2024-04-14T13:42:37.710 INFO:teuthology.orchestra.run.smithi069.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-04-14T13:42:37.710 INFO:teuthology.orchestra.run.smithi069.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-04-14T13:42:37.710 INFO:teuthology.orchestra.run.smithi069.stderr: Problem 42: cannot install the best candidate for the job 2024-04-14T13:42:37.710 INFO:teuthology.orchestra.run.smithi069.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:8030020200716174729:3a70019f.x86_64 from CentOS-AppStream 2024-04-14T13:42:37.710 INFO:teuthology.orchestra.run.smithi069.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:8030020200716174729:cccafca5.x86_64 from CentOS-AppStream 2024-04-14T13:42:37.710 INFO:teuthology.orchestra.run.smithi069.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:820230223114759:6fdf7e5d.x86_64 from CentOS-AppStream 2024-04-14T13:42:37.710 INFO:teuthology.orchestra.run.smithi069.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:820230223114759:d4326aba.x86_64 from CentOS-AppStream 2024-04-14T13:42:37.710 INFO:teuthology.orchestra.run.smithi069.stderr: Problem 43: cannot install the best candidate for the job 2024-04-14T13:42:37.710 INFO:teuthology.orchestra.run.smithi069.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:8030020200716213634:56fce90f.x86_64 from CentOS-AppStream 2024-04-14T13:42:37.710 INFO:teuthology.orchestra.run.smithi069.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:8030020200716213634:98a3c9d0.x86_64 from CentOS-AppStream 2024-04-14T13:42:37.710 INFO:teuthology.orchestra.run.smithi069.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:820230223114759:06b7596e.x86_64 from CentOS-AppStream 2024-04-14T13:42:37.710 INFO:teuthology.orchestra.run.smithi069.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:820230223114759:d2fc5d76.x86_64 from CentOS-AppStream 2024-04-14T13:42:37.710 INFO:teuthology.orchestra.run.smithi069.stderr: Problem 44: cannot install the best candidate for the job 2024-04-14T13:42:37.711 INFO:teuthology.orchestra.run.smithi069.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:8030020200716171139:3a70019f.x86_64 from CentOS-AppStream 2024-04-14T13:42:37.711 INFO:teuthology.orchestra.run.smithi069.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:8030020200716171139:cccafca5.x86_64 from CentOS-AppStream 2024-04-14T13:42:37.711 INFO:teuthology.orchestra.run.smithi069.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:820230223114759:6fdf7e5d.x86_64 from CentOS-AppStream 2024-04-14T13:42:37.711 INFO:teuthology.orchestra.run.smithi069.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:820230223114759:d4326aba.x86_64 from CentOS-AppStream 2024-04-14T13:42:37.711 INFO:teuthology.orchestra.run.smithi069.stderr: Problem 45: cannot install the best candidate for the job 2024-04-14T13:42:37.711 INFO:teuthology.orchestra.run.smithi069.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:8010020191120175858:95fce66d.x86_64 from CentOS-AppStream 2024-04-14T13:42:37.711 INFO:teuthology.orchestra.run.smithi069.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:8030020200723233049:09acf126.x86_64 from CentOS-AppStream 2024-04-14T13:42:37.711 INFO:teuthology.orchestra.run.smithi069.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:820230223114759:436af1ae.x86_64 from CentOS-AppStream 2024-04-14T13:42:37.711 INFO:teuthology.orchestra.run.smithi069.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:820230223114759:b12195ea.x86_64 from CentOS-AppStream 2024-04-14T13:42:38.118 INFO:teuthology.orchestra.run.smithi069.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-04-14T13:42:38.119 INFO:teuthology.orchestra.run.smithi069.stdout:Package kernel-5.14.0-361.el9.x86_64 is already installed. 2024-04-14T13:42:38.222 INFO:teuthology.orchestra.run.smithi069.stdout:Dependencies resolved. 2024-04-14T13:42:38.233 INFO:teuthology.orchestra.run.smithi069.stdout:================================================================================ 2024-04-14T13:42:38.234 INFO:teuthology.orchestra.run.smithi069.stdout: Package Architecture Version Repository Size 2024-04-14T13:42:38.234 INFO:teuthology.orchestra.run.smithi069.stdout:================================================================================ 2024-04-14T13:42:38.234 INFO:teuthology.orchestra.run.smithi069.stdout:Installing: 2024-04-14T13:42:38.234 INFO:teuthology.orchestra.run.smithi069.stdout: kernel x86_64 5.14.0-435.el9 baseos 214 k 2024-04-14T13:42:38.234 INFO:teuthology.orchestra.run.smithi069.stdout:Installing dependencies: 2024-04-14T13:42:38.234 INFO:teuthology.orchestra.run.smithi069.stdout: kernel-core x86_64 5.14.0-435.el9 baseos 15 M 2024-04-14T13:42:38.234 INFO:teuthology.orchestra.run.smithi069.stdout: kernel-modules x86_64 5.14.0-435.el9 baseos 34 M 2024-04-14T13:42:38.234 INFO:teuthology.orchestra.run.smithi069.stdout: kernel-modules-core x86_64 5.14.0-435.el9 baseos 28 M 2024-04-14T13:42:38.234 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-14T13:42:38.234 INFO:teuthology.orchestra.run.smithi069.stdout:Transaction Summary 2024-04-14T13:42:38.234 INFO:teuthology.orchestra.run.smithi069.stdout:================================================================================ 2024-04-14T13:42:38.234 INFO:teuthology.orchestra.run.smithi069.stdout:Install 4 Packages 2024-04-14T13:42:38.234 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-14T13:42:38.235 INFO:teuthology.orchestra.run.smithi069.stdout:Total download size: 77 M 2024-04-14T13:42:38.235 INFO:teuthology.orchestra.run.smithi069.stdout:Installed size: 123 M 2024-04-14T13:42:38.235 INFO:teuthology.orchestra.run.smithi069.stdout:Downloading Packages: 2024-04-14T13:42:38.588 INFO:teuthology.orchestra.run.smithi069.stdout:(1/4): kernel-5.14.0-435.el9.x86_64.rpm 883 kB/s | 214 kB 00:00 2024-04-14T13:42:39.105 INFO:teuthology.orchestra.run.smithi069.stdout:(2/4): kernel-core-5.14.0-435.el9.x86_64.rpm 20 MB/s | 15 MB 00:00 2024-04-14T13:42:39.897 INFO:teuthology.orchestra.run.smithi069.stdout:(3/4): kernel-modules-core-5.14.0-435.el9.x86_6 21 MB/s | 28 MB 00:01 2024-04-14T13:42:40.313 INFO:teuthology.orchestra.run.smithi069.stdout:(4/4): kernel-modules-5.14.0-435.el9.x86_64.rpm 17 MB/s | 34 MB 00:01 2024-04-14T13:42:40.316 INFO:teuthology.orchestra.run.smithi069.stdout:-------------------------------------------------------------------------------- 2024-04-14T13:42:40.317 INFO:teuthology.orchestra.run.smithi069.stdout:Total 37 MB/s | 77 MB 00:02 2024-04-14T13:42:40.357 INFO:teuthology.orchestra.run.smithi122.stdout:Extra Packages for Enterprise Linux 4.9 MB/s | 21 MB 00:04 2024-04-14T13:42:40.989 INFO:teuthology.orchestra.run.smithi069.stdout:Running transaction check 2024-04-14T13:42:41.356 INFO:teuthology.orchestra.run.smithi069.stdout:Transaction check succeeded. 2024-04-14T13:42:41.356 INFO:teuthology.orchestra.run.smithi069.stdout:Running transaction test 2024-04-14T13:42:42.346 INFO:teuthology.orchestra.run.smithi069.stdout:Transaction test succeeded. 2024-04-14T13:42:42.346 INFO:teuthology.orchestra.run.smithi069.stdout:Running transaction 2024-04-14T13:42:44.798 INFO:teuthology.orchestra.run.smithi069.stdout: Preparing : 1/1 2024-04-14T13:42:45.379 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : kernel-modules-core-5.14.0-435.el9.x86_64 1/4 2024-04-14T13:42:45.471 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : kernel-core-5.14.0-435.el9.x86_64 2/4 2024-04-14T13:42:45.583 INFO:teuthology.orchestra.run.smithi122.stdout:lab-extras 17 kB/s | 1.7 kB 00:00 2024-04-14T13:42:47.907 INFO:teuthology.orchestra.run.smithi122.stderr:Modular dependency problems: 2024-04-14T13:42:47.907 INFO:teuthology.orchestra.run.smithi122.stderr: 2024-04-14T13:42:47.907 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 1: conflicting requests 2024-04-14T13:42:47.907 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module container-tools:rhel8:820240321091303:20125149.x86_64 from CentOS-AppStream 2024-04-14T13:42:47.907 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 2: conflicting requests 2024-04-14T13:42:47.907 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module eclipse:rhel8:820201023100746:b230ed4e.x86_64 from CentOS-AppStream 2024-04-14T13:42:47.907 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 3: conflicting requests 2024-04-14T13:42:47.907 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module freeradius:3.0:8080020230117180605:89170a74.x86_64 from CentOS-AppStream 2024-04-14T13:42:47.907 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 4: conflicting requests 2024-04-14T13:42:47.907 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module gimp:2.8:8000020190628145146:4148dfdf.x86_64 from CentOS-AppStream 2024-04-14T13:42:47.908 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 5: conflicting requests 2024-04-14T13:42:47.908 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module go-toolset:rhel8:820240401160413:b754926a.x86_64 from CentOS-AppStream 2024-04-14T13:42:47.908 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 6: conflicting requests 2024-04-14T13:42:47.908 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module httpd:2.4:820240216084734:9edba152.x86_64 from CentOS-AppStream 2024-04-14T13:42:47.908 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 7: conflicting requests 2024-04-14T13:42:47.908 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module idm:client:820240307184541:49cc9d1b.x86_64 from CentOS-AppStream 2024-04-14T13:42:47.908 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 8: conflicting requests 2024-04-14T13:42:47.908 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module inkscape:0.92.3:8080020221205124429:3e031279.x86_64 from CentOS-AppStream 2024-04-14T13:42:47.908 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 9: conflicting requests 2024-04-14T13:42:47.908 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module jmc:rhel8:8050020211110222101:6392b1f8.x86_64 from CentOS-AppStream 2024-04-14T13:42:47.908 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 10: conflicting requests 2024-04-14T13:42:47.908 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module llvm-toolset:rhel8:820240205133135:9edba152.x86_64 from CentOS-AppStream 2024-04-14T13:42:47.908 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 11: conflicting requests 2024-04-14T13:42:47.908 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module log4j:2:8080020221205124743:9d367344.x86_64 from CentOS-AppStream 2024-04-14T13:42:47.908 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 12: conflicting requests 2024-04-14T13:42:47.909 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module mailman:2.1:820230727085757:77fc8825.x86_64 from CentOS-AppStream 2024-04-14T13:42:47.909 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 13: conflicting requests 2024-04-14T13:42:47.909 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module mariadb:10.3:8030020210419150013:30b713e6.x86_64 from CentOS-AppStream 2024-04-14T13:42:47.909 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 14: conflicting requests 2024-04-14T13:42:47.909 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module maven:3.5:8060020211203142015:c0229ad2.x86_64 from CentOS-AppStream 2024-04-14T13:42:47.909 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 15: conflicting requests 2024-04-14T13:42:47.909 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module mercurial:4.8:8000020190628020724:4148dfdf.x86_64 from CentOS-AppStream 2024-04-14T13:42:47.909 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 16: conflicting requests 2024-04-14T13:42:47.909 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module mod_auth_openidc:2.3:820230721123832:9edba152.x86_64 from CentOS-AppStream 2024-04-14T13:42:47.909 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 17: conflicting requests 2024-04-14T13:42:47.909 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module mysql:8.0:8040020210901180257:9f9e2e7e.x86_64 from CentOS-AppStream 2024-04-14T13:42:47.909 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 18: conflicting requests 2024-04-14T13:42:47.909 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module nginx:1.14:8000020211221191913:55190bc5.x86_64 from CentOS-AppStream 2024-04-14T13:42:47.909 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 19: conflicting requests 2024-04-14T13:42:47.909 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module nodejs:10:8040020210120182536:9f9e2e7e.x86_64 from CentOS-AppStream 2024-04-14T13:42:47.909 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 20: conflicting requests 2024-04-14T13:42:47.910 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl:5.26:8000020190628020724:55190bc5.x86_64 from CentOS-AppStream 2024-04-14T13:42:47.910 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 21: conflicting requests 2024-04-14T13:42:47.910 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module php:7.2:8020020200507003613:2c7ca891.x86_64 from CentOS-AppStream 2024-04-14T13:42:47.910 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 22: conflicting requests 2024-04-14T13:42:47.910 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module postgresql:10:820230223114800:9edba152.x86_64 from CentOS-AppStream 2024-04-14T13:42:47.910 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 23: conflicting requests 2024-04-14T13:42:47.910 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module python27:2.7:820240208011952:182f7c73.x86_64 from CentOS-AppStream 2024-04-14T13:42:47.910 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 24: conflicting requests 2024-04-14T13:42:47.910 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module python36:3.6:8050020210825152031:982725ab.x86_64 from CentOS-AppStream 2024-04-14T13:42:47.910 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 25: conflicting requests 2024-04-14T13:42:47.910 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module python38:3.8:820230810143931:64c1cd5a.x86_64 from CentOS-AppStream 2024-04-14T13:42:47.910 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 26: conflicting requests 2024-04-14T13:42:47.910 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module python39:3.9:820240214182535:17377f89.x86_64 from CentOS-AppStream 2024-04-14T13:42:47.911 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 27: conflicting requests 2024-04-14T13:42:47.911 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module redis:5:8040020211019153849:9f9e2e7e.x86_64 from CentOS-AppStream 2024-04-14T13:42:47.911 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 28: conflicting requests 2024-04-14T13:42:47.911 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module rhn-tools:1.0:8010020191114034948:f69d1239.x86_64 from CentOS-AppStream 2024-04-14T13:42:47.911 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 29: conflicting requests 2024-04-14T13:42:47.911 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module ruby:2.5:820230627084142:9edba152.x86_64 from CentOS-AppStream 2024-04-14T13:42:47.911 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 30: conflicting requests 2024-04-14T13:42:47.911 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module rust-toolset:rhel8:820240119204620:b09eea91.x86_64 from CentOS-AppStream 2024-04-14T13:42:47.911 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 31: conflicting requests 2024-04-14T13:42:47.911 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module satellite-5-client:1.0:8010020191114035551:cdc1202b.x86_64 from CentOS-AppStream 2024-04-14T13:42:47.911 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 32: conflicting requests 2024-04-14T13:42:47.911 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module squid:4:820240319173245:9edba152.x86_64 from CentOS-AppStream 2024-04-14T13:42:47.911 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 33: conflicting requests 2024-04-14T13:42:47.911 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module subversion:1.10:8070020220711155714:78111232.x86_64 from CentOS-AppStream 2024-04-14T13:42:47.911 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 34: conflicting requests 2024-04-14T13:42:47.911 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module swig:3.0:8030020200706142531:30b713e6.x86_64 from CentOS-AppStream 2024-04-14T13:42:47.912 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 35: conflicting requests 2024-04-14T13:42:47.912 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module varnish:6:8050020211109225449:b4937e53.x86_64 from CentOS-AppStream 2024-04-14T13:42:47.912 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 36: conflicting requests 2024-04-14T13:42:47.912 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module virt:rhel:820240314161907:9edba152.x86_64 from CentOS-AppStream 2024-04-14T13:42:47.912 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 37: conflicting requests 2024-04-14T13:42:47.912 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:8030020200716155257:7cc0a66d.x86_64 from CentOS-AppStream 2024-04-14T13:42:47.912 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:8030020200716155257:b967a9a2.x86_64 from CentOS-AppStream 2024-04-14T13:42:47.912 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:820230811133638:36f9fae6.x86_64 from CentOS-AppStream 2024-04-14T13:42:47.912 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 38: cannot install the best candidate for the job 2024-04-14T13:42:47.912 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:8010020191114031501:a5949e2e.x86_64 from CentOS-AppStream 2024-04-14T13:42:47.912 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:8010020191114031501:be2e4737.x86_64 from CentOS-AppStream 2024-04-14T13:42:47.912 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:820230223114759:2ab761e1.x86_64 from CentOS-AppStream 2024-04-14T13:42:47.912 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:820230223114759:86e45846.x86_64 from CentOS-AppStream 2024-04-14T13:42:47.912 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 39: conflicting requests 2024-04-14T13:42:47.912 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:8030020200715230104:1e4bbb35.x86_64 from CentOS-AppStream 2024-04-14T13:42:47.912 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:8030020200715230104:ea09926d.x86_64 from CentOS-AppStream 2024-04-14T13:42:47.913 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:082fdf2f.x86_64 from CentOS-AppStream 2024-04-14T13:42:47.913 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-04-14T13:42:47.913 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-04-14T13:42:47.913 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:fbe42456.x86_64 from CentOS-AppStream 2024-04-14T13:42:47.913 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 40: cannot install the best candidate for the job 2024-04-14T13:42:47.913 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:8010020191114031513:16b3ab4d.x86_64 from CentOS-AppStream 2024-04-14T13:42:47.913 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:8010020191114031513:e1f37755.x86_64 from CentOS-AppStream 2024-04-14T13:42:47.913 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-04-14T13:42:47.913 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-04-14T13:42:47.913 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 41: cannot install the best candidate for the job 2024-04-14T13:42:47.913 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:8030020200716150652:1e4bbb35.x86_64 from CentOS-AppStream 2024-04-14T13:42:47.913 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:8030020200716150652:ea09926d.x86_64 from CentOS-AppStream 2024-04-14T13:42:47.913 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-04-14T13:42:47.913 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-04-14T13:42:47.913 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 42: cannot install the best candidate for the job 2024-04-14T13:42:47.914 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:8030020200716174729:3a70019f.x86_64 from CentOS-AppStream 2024-04-14T13:42:47.914 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:8030020200716174729:cccafca5.x86_64 from CentOS-AppStream 2024-04-14T13:42:47.914 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:820230223114759:6fdf7e5d.x86_64 from CentOS-AppStream 2024-04-14T13:42:47.914 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:820230223114759:d4326aba.x86_64 from CentOS-AppStream 2024-04-14T13:42:47.914 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 43: cannot install the best candidate for the job 2024-04-14T13:42:47.914 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:8030020200716213634:56fce90f.x86_64 from CentOS-AppStream 2024-04-14T13:42:47.914 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:8030020200716213634:98a3c9d0.x86_64 from CentOS-AppStream 2024-04-14T13:42:47.914 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:820230223114759:06b7596e.x86_64 from CentOS-AppStream 2024-04-14T13:42:47.914 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:820230223114759:d2fc5d76.x86_64 from CentOS-AppStream 2024-04-14T13:42:47.914 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 44: cannot install the best candidate for the job 2024-04-14T13:42:47.914 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:8030020200716171139:3a70019f.x86_64 from CentOS-AppStream 2024-04-14T13:42:47.914 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:8030020200716171139:cccafca5.x86_64 from CentOS-AppStream 2024-04-14T13:42:47.914 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:820230223114759:6fdf7e5d.x86_64 from CentOS-AppStream 2024-04-14T13:42:47.914 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:820230223114759:d4326aba.x86_64 from CentOS-AppStream 2024-04-14T13:42:47.914 INFO:teuthology.orchestra.run.smithi122.stderr: Problem 45: cannot install the best candidate for the job 2024-04-14T13:42:47.915 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:8010020191120175858:95fce66d.x86_64 from CentOS-AppStream 2024-04-14T13:42:47.915 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:8030020200723233049:09acf126.x86_64 from CentOS-AppStream 2024-04-14T13:42:47.915 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:820230223114759:436af1ae.x86_64 from CentOS-AppStream 2024-04-14T13:42:47.915 INFO:teuthology.orchestra.run.smithi122.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:820230223114759:b12195ea.x86_64 from CentOS-AppStream 2024-04-14T13:42:48.220 INFO:teuthology.orchestra.run.smithi069.stdout: Running scriptlet: kernel-core-5.14.0-435.el9.x86_64 2/4 2024-04-14T13:42:48.348 INFO:teuthology.orchestra.run.smithi122.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-04-14T13:42:48.348 INFO:teuthology.orchestra.run.smithi122.stdout:Package kernel-5.14.0-361.el9.x86_64 is already installed. 2024-04-14T13:42:48.356 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : kernel-modules-5.14.0-435.el9.x86_64 3/4 2024-04-14T13:42:48.458 INFO:teuthology.orchestra.run.smithi122.stdout:Dependencies resolved. 2024-04-14T13:42:48.464 INFO:teuthology.orchestra.run.smithi122.stdout:================================================================================ 2024-04-14T13:42:48.464 INFO:teuthology.orchestra.run.smithi122.stdout: Package Architecture Version Repository Size 2024-04-14T13:42:48.464 INFO:teuthology.orchestra.run.smithi122.stdout:================================================================================ 2024-04-14T13:42:48.464 INFO:teuthology.orchestra.run.smithi122.stdout:Installing: 2024-04-14T13:42:48.464 INFO:teuthology.orchestra.run.smithi122.stdout: kernel x86_64 5.14.0-435.el9 baseos 214 k 2024-04-14T13:42:48.464 INFO:teuthology.orchestra.run.smithi122.stdout:Installing dependencies: 2024-04-14T13:42:48.464 INFO:teuthology.orchestra.run.smithi122.stdout: kernel-core x86_64 5.14.0-435.el9 baseos 15 M 2024-04-14T13:42:48.464 INFO:teuthology.orchestra.run.smithi122.stdout: kernel-modules x86_64 5.14.0-435.el9 baseos 34 M 2024-04-14T13:42:48.464 INFO:teuthology.orchestra.run.smithi122.stdout: kernel-modules-core x86_64 5.14.0-435.el9 baseos 28 M 2024-04-14T13:42:48.464 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-04-14T13:42:48.464 INFO:teuthology.orchestra.run.smithi122.stdout:Transaction Summary 2024-04-14T13:42:48.464 INFO:teuthology.orchestra.run.smithi122.stdout:================================================================================ 2024-04-14T13:42:48.465 INFO:teuthology.orchestra.run.smithi122.stdout:Install 4 Packages 2024-04-14T13:42:48.465 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-04-14T13:42:48.465 INFO:teuthology.orchestra.run.smithi122.stdout:Total download size: 77 M 2024-04-14T13:42:48.465 INFO:teuthology.orchestra.run.smithi122.stdout:Installed size: 123 M 2024-04-14T13:42:48.465 INFO:teuthology.orchestra.run.smithi122.stdout:Downloading Packages: 2024-04-14T13:42:48.651 INFO:teuthology.orchestra.run.smithi122.stdout:(1/4): kernel-5.14.0-435.el9.x86_64.rpm 1.7 MB/s | 214 kB 00:00 2024-04-14T13:42:49.443 INFO:teuthology.orchestra.run.smithi122.stdout:(2/4): kernel-core-5.14.0-435.el9.x86_64.rpm 16 MB/s | 15 MB 00:00 2024-04-14T13:42:50.101 INFO:teuthology.orchestra.run.smithi122.stdout:(3/4): kernel-modules-core-5.14.0-435.el9.x86_6 19 MB/s | 28 MB 00:01 2024-04-14T13:42:52.061 INFO:teuthology.orchestra.run.smithi122.stdout:(4/4): kernel-modules-5.14.0-435.el9.x86_64.rpm 9.5 MB/s | 34 MB 00:03 2024-04-14T13:42:52.061 INFO:teuthology.orchestra.run.smithi122.stdout:-------------------------------------------------------------------------------- 2024-04-14T13:42:52.061 INFO:teuthology.orchestra.run.smithi122.stdout:Total 21 MB/s | 77 MB 00:03 2024-04-14T13:42:52.731 INFO:teuthology.orchestra.run.smithi122.stdout:Running transaction check 2024-04-14T13:42:53.040 INFO:teuthology.orchestra.run.smithi069.stdout: Running scriptlet: kernel-modules-5.14.0-435.el9.x86_64 3/4 2024-04-14T13:42:53.109 INFO:teuthology.orchestra.run.smithi122.stdout:Transaction check succeeded. 2024-04-14T13:42:53.110 INFO:teuthology.orchestra.run.smithi122.stdout:Running transaction test 2024-04-14T13:42:53.265 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : kernel-5.14.0-435.el9.x86_64 4/4 2024-04-14T13:42:54.098 INFO:teuthology.orchestra.run.smithi122.stdout:Transaction test succeeded. 2024-04-14T13:42:54.098 INFO:teuthology.orchestra.run.smithi122.stdout:Running transaction 2024-04-14T13:42:56.574 INFO:teuthology.orchestra.run.smithi122.stdout: Preparing : 1/1 2024-04-14T13:42:57.258 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : kernel-modules-core-5.14.0-435.el9.x86_64 1/4 2024-04-14T13:42:57.354 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : kernel-core-5.14.0-435.el9.x86_64 2/4 2024-04-14T13:42:57.983 INFO:teuthology.orchestra.run.smithi069.stdout: Running scriptlet: kernel-modules-core-5.14.0-435.el9.x86_64 4/4 2024-04-14T13:43:00.081 INFO:teuthology.orchestra.run.smithi122.stdout: Running scriptlet: kernel-core-5.14.0-435.el9.x86_64 2/4 2024-04-14T13:43:00.228 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : kernel-modules-5.14.0-435.el9.x86_64 3/4 2024-04-14T13:43:04.948 INFO:teuthology.orchestra.run.smithi122.stdout: Running scriptlet: kernel-modules-5.14.0-435.el9.x86_64 3/4 2024-04-14T13:43:04.988 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : kernel-5.14.0-435.el9.x86_64 4/4 2024-04-14T13:43:09.709 INFO:teuthology.orchestra.run.smithi122.stdout: Running scriptlet: kernel-modules-core-5.14.0-435.el9.x86_64 4/4 2024-04-14T13:43:27.393 INFO:teuthology.orchestra.run.smithi069.stdout: Running scriptlet: kernel-core-5.14.0-435.el9.x86_64 4/4 2024-04-14T13:43:27.393 INFO:teuthology.orchestra.run.smithi069.stdout:dracut: Disabling early microcode, because kernel does not support it. CONFIG_MICROCODE_[AMD|INTEL]!=y 2024-04-14T13:43:27.393 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-14T13:43:27.440 INFO:teuthology.orchestra.run.smithi069.stdout: Running scriptlet: kernel-modules-5.14.0-435.el9.x86_64 4/4 2024-04-14T13:43:27.791 INFO:teuthology.orchestra.run.smithi069.stdout: Running scriptlet: kernel-5.14.0-435.el9.x86_64 4/4 2024-04-14T13:43:27.791 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : kernel-5.14.0-435.el9.x86_64 1/4 2024-04-14T13:43:27.791 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : kernel-core-5.14.0-435.el9.x86_64 2/4 2024-04-14T13:43:27.791 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : kernel-modules-5.14.0-435.el9.x86_64 3/4 2024-04-14T13:43:28.232 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : kernel-modules-core-5.14.0-435.el9.x86_64 4/4 2024-04-14T13:43:28.232 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-14T13:43:28.232 INFO:teuthology.orchestra.run.smithi069.stdout:Installed: 2024-04-14T13:43:28.232 INFO:teuthology.orchestra.run.smithi069.stdout: kernel-5.14.0-435.el9.x86_64 2024-04-14T13:43:28.232 INFO:teuthology.orchestra.run.smithi069.stdout: kernel-core-5.14.0-435.el9.x86_64 2024-04-14T13:43:28.232 INFO:teuthology.orchestra.run.smithi069.stdout: kernel-modules-5.14.0-435.el9.x86_64 2024-04-14T13:43:28.232 INFO:teuthology.orchestra.run.smithi069.stdout: kernel-modules-core-5.14.0-435.el9.x86_64 2024-04-14T13:43:28.232 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-14T13:43:28.232 INFO:teuthology.orchestra.run.smithi069.stdout:Complete! 2024-04-14T13:43:28.598 DEBUG:teuthology.orchestra.run.smithi069:> rpm -q kernel | sort -rV | head -n 1 2024-04-14T13:43:28.692 INFO:teuthology.orchestra.run.smithi069.stdout:kernel-5.14.0-435.el9.x86_64 2024-04-14T13:43:28.693 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-435.el9.x86_64 2024-04-14T13:43:28.693 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-361.el9.x86_64 Expected: 5.14.0-435.el9.x86_64 2024-04-14T13:43:28.693 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-04-14T13:43:28.693 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-04-14T13:43:28.693 INFO:teuthology.task.kernel:Installing distro kernel on mon.a... 2024-04-14T13:43:28.693 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi069.front.sepia.ceph.com, path=None, version=distro) 2024-04-14T13:43:28.693 DEBUG:teuthology.orchestra.run.smithi069:> rpm -q kernel | sort -rV | head -n 1 2024-04-14T13:43:28.774 INFO:teuthology.orchestra.run.smithi069.stdout:kernel-5.14.0-435.el9.x86_64 2024-04-14T13:43:28.775 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-435.el9.x86_64 2024-04-14T13:43:28.775 DEBUG:teuthology.orchestra.run.smithi069:> sudo rpm -qi grub2-tools 2024-04-14T13:43:28.818 INFO:teuthology.orchestra.run.smithi069.stdout:Name : grub2-tools 2024-04-14T13:43:28.818 INFO:teuthology.orchestra.run.smithi069.stdout:Epoch : 1 2024-04-14T13:43:28.818 INFO:teuthology.orchestra.run.smithi069.stdout:Version : 2.06 2024-04-14T13:43:28.819 INFO:teuthology.orchestra.run.smithi069.stdout:Release : 68.el9 2024-04-14T13:43:28.819 INFO:teuthology.orchestra.run.smithi069.stdout:Architecture: x86_64 2024-04-14T13:43:28.819 INFO:teuthology.orchestra.run.smithi069.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2024-04-14T13:43:28.819 INFO:teuthology.orchestra.run.smithi069.stdout:Group : Unspecified 2024-04-14T13:43:28.819 INFO:teuthology.orchestra.run.smithi069.stdout:Size : 8264254 2024-04-14T13:43:28.819 INFO:teuthology.orchestra.run.smithi069.stdout:License : GPLv3+ 2024-04-14T13:43:28.819 INFO:teuthology.orchestra.run.smithi069.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-04-14T13:43:28.819 INFO:teuthology.orchestra.run.smithi069.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-04-14T13:43:28.819 INFO:teuthology.orchestra.run.smithi069.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-04-14T13:43:28.819 INFO:teuthology.orchestra.run.smithi069.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-04-14T13:43:28.819 INFO:teuthology.orchestra.run.smithi069.stdout:Packager : builder@centos.org 2024-04-14T13:43:28.819 INFO:teuthology.orchestra.run.smithi069.stdout:Vendor : CentOS 2024-04-14T13:43:28.819 INFO:teuthology.orchestra.run.smithi069.stdout:URL : http://www.gnu.org/software/grub/ 2024-04-14T13:43:28.819 INFO:teuthology.orchestra.run.smithi069.stdout:Summary : Support tools for GRUB. 2024-04-14T13:43:28.820 INFO:teuthology.orchestra.run.smithi069.stdout:Description : 2024-04-14T13:43:28.820 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-14T13:43:28.820 INFO:teuthology.orchestra.run.smithi069.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-04-14T13:43:28.820 INFO:teuthology.orchestra.run.smithi069.stdout:customizable bootloader with modular architecture. It supports a rich 2024-04-14T13:43:28.820 INFO:teuthology.orchestra.run.smithi069.stdout:variety of kernel formats, file systems, computer architectures and 2024-04-14T13:43:28.820 INFO:teuthology.orchestra.run.smithi069.stdout:hardware devices. 2024-04-14T13:43:28.820 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-14T13:43:28.820 INFO:teuthology.orchestra.run.smithi069.stdout:This subpackage provides tools for support of all platforms. 2024-04-14T13:43:28.821 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-04-14T13:43:28.821 INFO:teuthology.task.kernel:Updating grub on smithi069 to boot 5.14.0-435.el9.x86_64 2024-04-14T13:43:28.821 DEBUG:teuthology.orchestra.run.smithi069:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-04-14T13:43:29.322 INFO:teuthology.orchestra.run.smithi069.stderr:Generating grub configuration file ... 2024-04-14T13:43:30.645 INFO:teuthology.orchestra.run.smithi069.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-04-14T13:43:30.679 INFO:teuthology.orchestra.run.smithi069.stderr:done 2024-04-14T13:43:30.682 DEBUG:teuthology.orchestra.run.smithi069:> mktemp 2024-04-14T13:43:30.697 INFO:teuthology.orchestra.run.smithi069.stdout:/tmp/tmp.guIQ9nRIl4 2024-04-14T13:43:30.697 DEBUG:teuthology.orchestra.run.smithi069:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.guIQ9nRIl4 2024-04-14T13:43:30.766 DEBUG:teuthology.orchestra.run.smithi069:> sudo chmod 0666 /tmp/tmp.guIQ9nRIl4 2024-04-14T13:43:30.907 DEBUG:teuthology.orchestra.remote:smithi069:/tmp/tmp.guIQ9nRIl4 is 6KB 2024-04-14T13:43:30.955 DEBUG:teuthology.orchestra.run.smithi069:> rm -fr /tmp/tmp.guIQ9nRIl4 2024-04-14T13:43:30.969 DEBUG:teuthology.orchestra.run.smithi069:> sudo /bin/ls /boot/loader/entries || true 2024-04-14T13:43:31.041 INFO:teuthology.orchestra.run.smithi069.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2024-04-14T13:43:31.041 INFO:teuthology.orchestra.run.smithi069.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2024-04-14T13:43:31.041 INFO:teuthology.orchestra.run.smithi069.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2024-04-14T13:43:31.041 INFO:teuthology.orchestra.run.smithi069.stdout:15949e3c88704300882482eddfd7baae-5.14.0-435.el9.x86_64.conf 2024-04-14T13:43:31.043 DEBUG:teuthology.orchestra.run.smithi069:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-5.14.0-435.el9.x86_64 2024-04-14T13:43:31.213 DEBUG:teuthology.orchestra.run.smithi069:> sudo shutdown -r now 2024-04-14T13:43:38.963 INFO:teuthology.orchestra.run.smithi122.stdout: Running scriptlet: kernel-core-5.14.0-435.el9.x86_64 4/4 2024-04-14T13:43:38.963 INFO:teuthology.orchestra.run.smithi122.stdout:dracut: Disabling early microcode, because kernel does not support it. CONFIG_MICROCODE_[AMD|INTEL]!=y 2024-04-14T13:43:38.963 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-04-14T13:43:39.014 INFO:teuthology.orchestra.run.smithi122.stdout: Running scriptlet: kernel-modules-5.14.0-435.el9.x86_64 4/4 2024-04-14T13:43:39.416 INFO:teuthology.orchestra.run.smithi122.stdout: Running scriptlet: kernel-5.14.0-435.el9.x86_64 4/4 2024-04-14T13:43:39.416 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : kernel-5.14.0-435.el9.x86_64 1/4 2024-04-14T13:43:39.417 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : kernel-core-5.14.0-435.el9.x86_64 2/4 2024-04-14T13:43:39.417 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : kernel-modules-5.14.0-435.el9.x86_64 3/4 2024-04-14T13:43:39.831 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : kernel-modules-core-5.14.0-435.el9.x86_64 4/4 2024-04-14T13:43:39.831 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-04-14T13:43:39.831 INFO:teuthology.orchestra.run.smithi122.stdout:Installed: 2024-04-14T13:43:39.831 INFO:teuthology.orchestra.run.smithi122.stdout: kernel-5.14.0-435.el9.x86_64 2024-04-14T13:43:39.831 INFO:teuthology.orchestra.run.smithi122.stdout: kernel-core-5.14.0-435.el9.x86_64 2024-04-14T13:43:39.831 INFO:teuthology.orchestra.run.smithi122.stdout: kernel-modules-5.14.0-435.el9.x86_64 2024-04-14T13:43:39.831 INFO:teuthology.orchestra.run.smithi122.stdout: kernel-modules-core-5.14.0-435.el9.x86_64 2024-04-14T13:43:39.831 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-04-14T13:43:39.831 INFO:teuthology.orchestra.run.smithi122.stdout:Complete! 2024-04-14T13:43:40.188 DEBUG:teuthology.orchestra.run.smithi122:> rpm -q kernel | sort -rV | head -n 1 2024-04-14T13:43:40.287 INFO:teuthology.orchestra.run.smithi122.stdout:kernel-5.14.0-435.el9.x86_64 2024-04-14T13:43:40.287 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-435.el9.x86_64 2024-04-14T13:43:40.287 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-361.el9.x86_64 Expected: 5.14.0-435.el9.x86_64 2024-04-14T13:43:40.287 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-04-14T13:43:40.287 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-04-14T13:43:40.287 INFO:teuthology.task.kernel:Installing distro kernel on mon.b... 2024-04-14T13:43:40.288 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi122.front.sepia.ceph.com, path=None, version=distro) 2024-04-14T13:43:40.288 DEBUG:teuthology.orchestra.run.smithi122:> rpm -q kernel | sort -rV | head -n 1 2024-04-14T13:43:40.367 INFO:teuthology.orchestra.run.smithi122.stdout:kernel-5.14.0-435.el9.x86_64 2024-04-14T13:43:40.367 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-435.el9.x86_64 2024-04-14T13:43:40.368 DEBUG:teuthology.orchestra.run.smithi122:> sudo rpm -qi grub2-tools 2024-04-14T13:43:40.404 INFO:teuthology.orchestra.run.smithi122.stdout:Name : grub2-tools 2024-04-14T13:43:40.404 INFO:teuthology.orchestra.run.smithi122.stdout:Epoch : 1 2024-04-14T13:43:40.405 INFO:teuthology.orchestra.run.smithi122.stdout:Version : 2.06 2024-04-14T13:43:40.405 INFO:teuthology.orchestra.run.smithi122.stdout:Release : 68.el9 2024-04-14T13:43:40.405 INFO:teuthology.orchestra.run.smithi122.stdout:Architecture: x86_64 2024-04-14T13:43:40.405 INFO:teuthology.orchestra.run.smithi122.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2024-04-14T13:43:40.405 INFO:teuthology.orchestra.run.smithi122.stdout:Group : Unspecified 2024-04-14T13:43:40.405 INFO:teuthology.orchestra.run.smithi122.stdout:Size : 8264254 2024-04-14T13:43:40.405 INFO:teuthology.orchestra.run.smithi122.stdout:License : GPLv3+ 2024-04-14T13:43:40.405 INFO:teuthology.orchestra.run.smithi122.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-04-14T13:43:40.405 INFO:teuthology.orchestra.run.smithi122.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-04-14T13:43:40.405 INFO:teuthology.orchestra.run.smithi122.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-04-14T13:43:40.405 INFO:teuthology.orchestra.run.smithi122.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-04-14T13:43:40.405 INFO:teuthology.orchestra.run.smithi122.stdout:Packager : builder@centos.org 2024-04-14T13:43:40.406 INFO:teuthology.orchestra.run.smithi122.stdout:Vendor : CentOS 2024-04-14T13:43:40.406 INFO:teuthology.orchestra.run.smithi122.stdout:URL : http://www.gnu.org/software/grub/ 2024-04-14T13:43:40.406 INFO:teuthology.orchestra.run.smithi122.stdout:Summary : Support tools for GRUB. 2024-04-14T13:43:40.406 INFO:teuthology.orchestra.run.smithi122.stdout:Description : 2024-04-14T13:43:40.406 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-04-14T13:43:40.406 INFO:teuthology.orchestra.run.smithi122.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-04-14T13:43:40.406 INFO:teuthology.orchestra.run.smithi122.stdout:customizable bootloader with modular architecture. It supports a rich 2024-04-14T13:43:40.406 INFO:teuthology.orchestra.run.smithi122.stdout:variety of kernel formats, file systems, computer architectures and 2024-04-14T13:43:40.406 INFO:teuthology.orchestra.run.smithi122.stdout:hardware devices. 2024-04-14T13:43:40.406 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-04-14T13:43:40.406 INFO:teuthology.orchestra.run.smithi122.stdout:This subpackage provides tools for support of all platforms. 2024-04-14T13:43:40.407 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-04-14T13:43:40.407 INFO:teuthology.task.kernel:Updating grub on smithi122 to boot 5.14.0-435.el9.x86_64 2024-04-14T13:43:40.408 DEBUG:teuthology.orchestra.run.smithi122:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-04-14T13:43:40.957 INFO:teuthology.orchestra.run.smithi122.stderr:Generating grub configuration file ... 2024-04-14T13:43:42.377 INFO:teuthology.orchestra.run.smithi122.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-04-14T13:43:42.403 INFO:teuthology.orchestra.run.smithi122.stderr:done 2024-04-14T13:43:42.405 DEBUG:teuthology.orchestra.run.smithi122:> mktemp 2024-04-14T13:43:42.423 INFO:teuthology.orchestra.run.smithi122.stdout:/tmp/tmp.Dc11YA9Svb 2024-04-14T13:43:42.423 DEBUG:teuthology.orchestra.run.smithi122:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.Dc11YA9Svb 2024-04-14T13:43:42.492 DEBUG:teuthology.orchestra.run.smithi122:> sudo chmod 0666 /tmp/tmp.Dc11YA9Svb 2024-04-14T13:43:42.636 DEBUG:teuthology.orchestra.remote:smithi122:/tmp/tmp.Dc11YA9Svb is 6KB 2024-04-14T13:43:42.685 DEBUG:teuthology.orchestra.run.smithi122:> rm -fr /tmp/tmp.Dc11YA9Svb 2024-04-14T13:43:42.699 DEBUG:teuthology.orchestra.run.smithi122:> sudo /bin/ls /boot/loader/entries || true 2024-04-14T13:43:42.764 INFO:teuthology.orchestra.run.smithi122.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2024-04-14T13:43:42.764 INFO:teuthology.orchestra.run.smithi122.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2024-04-14T13:43:42.764 INFO:teuthology.orchestra.run.smithi122.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2024-04-14T13:43:42.764 INFO:teuthology.orchestra.run.smithi122.stdout:15949e3c88704300882482eddfd7baae-5.14.0-435.el9.x86_64.conf 2024-04-14T13:43:42.765 DEBUG:teuthology.orchestra.run.smithi122:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-5.14.0-435.el9.x86_64 2024-04-14T13:43:42.915 DEBUG:teuthology.orchestra.run.smithi122:> sudo shutdown -r now 2024-04-14T13:44:01.233 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2024-04-14T13:44:01.234 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi069.front.sepia.ceph.com' 2024-04-14T13:44:01.235 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi069.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-14T13:44:12.926 INFO:teuthology.task.kernel:Checking client mon.b for new kernel version... 2024-04-14T13:44:12.927 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi122.front.sepia.ceph.com' 2024-04-14T13:44:12.927 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi122.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-14T13:44:31.264 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.122 2024-04-14T13:44:40.270 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi122.front.sepia.ceph.com' 2024-04-14T13:44:40.271 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi122.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-14T13:44:43.326 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.122 2024-04-14T13:44:55.339 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi122.front.sepia.ceph.com' 2024-04-14T13:44:55.340 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi122.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-14T13:45:01.243 DEBUG:teuthology.orchestra.remote:timed out 2024-04-14T13:45:10.250 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi069.front.sepia.ceph.com' 2024-04-14T13:45:10.251 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi069.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-14T13:45:10.688 DEBUG:teuthology.orchestra.run.smithi069:> true 2024-04-14T13:45:11.133 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi069.front.sepia.ceph.com' 2024-04-14T13:45:11.133 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "5.14.0-435.el9.x86_64"... 2024-04-14T13:45:11.133 DEBUG:teuthology.orchestra.run.smithi069:> uname -r 2024-04-14T13:45:11.147 INFO:teuthology.orchestra.run.smithi069.stdout:5.14.0-435.el9.x86_64 2024-04-14T13:45:11.147 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-435.el9.x86_64 vs 5.14.0-435.el9.x86_64 2024-04-14T13:45:11.147 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-04-14T13:45:11.147 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-04-14T13:45:12.149 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2024-04-14T13:45:12.149 DEBUG:teuthology.orchestra.run.smithi069:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-04-14T13:45:12.203 INFO:teuthology.orchestra.run.smithi069.stdout:ttyS1 2024-04-14T13:45:12.257 DEBUG:teuthology.parallel:result is None 2024-04-14T13:45:27.162 DEBUG:teuthology.orchestra.run.smithi122:> true 2024-04-14T13:45:27.767 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi122.front.sepia.ceph.com' 2024-04-14T13:45:27.767 INFO:teuthology.task.kernel:Checking kernel version of mon.b, want "5.14.0-435.el9.x86_64"... 2024-04-14T13:45:27.767 DEBUG:teuthology.orchestra.run.smithi122:> uname -r 2024-04-14T13:45:27.793 INFO:teuthology.orchestra.run.smithi122.stdout:5.14.0-435.el9.x86_64 2024-04-14T13:45:27.793 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-435.el9.x86_64 vs 5.14.0-435.el9.x86_64 2024-04-14T13:45:27.793 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-04-14T13:45:27.793 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-04-14T13:45:28.794 INFO:teuthology.task.kernel:Enabling kdb on mon.b... 2024-04-14T13:45:28.794 DEBUG:teuthology.orchestra.run.smithi122:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-04-14T13:45:29.151 INFO:teuthology.orchestra.run.smithi122.stdout:ttyS1 2024-04-14T13:45:29.195 DEBUG:teuthology.parallel:result is None 2024-04-14T13:45:29.196 INFO:teuthology.run_tasks:Running task internal.base... 2024-04-14T13:45:29.241 INFO:teuthology.task.internal:Creating test directory... 2024-04-14T13:45:29.241 DEBUG:teuthology.orchestra.run.smithi069:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-04-14T13:45:29.245 DEBUG:teuthology.orchestra.run.smithi122:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-04-14T13:45:29.261 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-04-14T13:45:29.290 INFO:teuthology.run_tasks:Running task internal.archive... 2024-04-14T13:45:29.357 INFO:teuthology.task.internal:Creating archive directory... 2024-04-14T13:45:29.357 DEBUG:teuthology.orchestra.run.smithi069:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-04-14T13:45:29.362 DEBUG:teuthology.orchestra.run.smithi122:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-04-14T13:45:29.541 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-04-14T13:45:29.565 INFO:teuthology.task.internal:Enabling coredump saving... 2024-04-14T13:45:29.565 DEBUG:teuthology.orchestra.run.smithi069:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-04-14T13:45:29.572 DEBUG:teuthology.orchestra.run.smithi122:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-04-14T13:45:29.607 INFO:teuthology.orchestra.run.smithi122.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-14T13:45:29.615 INFO:teuthology.orchestra.run.smithi069.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-14T13:45:29.619 INFO:teuthology.orchestra.run.smithi122.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-14T13:45:29.629 INFO:teuthology.orchestra.run.smithi069.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-14T13:45:29.631 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-04-14T13:45:29.664 INFO:teuthology.task.internal:Configuring sudo... 2024-04-14T13:45:29.665 DEBUG:teuthology.orchestra.run.smithi069:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-04-14T13:45:29.670 DEBUG:teuthology.orchestra.run.smithi122:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-04-14T13:45:29.699 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-04-14T13:45:29.769 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-04-14T13:45:29.770 DEBUG:teuthology.orchestra.run.smithi069:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-04-14T13:45:29.774 DEBUG:teuthology.orchestra.run.smithi122:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-04-14T13:45:29.792 DEBUG:teuthology.orchestra.run.smithi069:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-14T13:45:29.870 DEBUG:teuthology.orchestra.run.smithi069:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-14T13:45:29.923 DEBUG:teuthology.orchestra.run.smithi069:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-14T13:45:30.006 DEBUG:teuthology.orchestra.run.smithi069:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-14T13:45:30.055 DEBUG:teuthology.orchestra.run.smithi069:> set -ex 2024-04-14T13:45:30.055 DEBUG:teuthology.orchestra.run.smithi069:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-04-14T13:45:30.125 DEBUG:teuthology.orchestra.run.smithi122:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-14T13:45:30.166 DEBUG:teuthology.orchestra.run.smithi122:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-14T13:45:30.214 DEBUG:teuthology.orchestra.run.smithi122:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-14T13:45:30.293 DEBUG:teuthology.orchestra.run.smithi122:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-14T13:45:30.336 DEBUG:teuthology.orchestra.run.smithi122:> set -ex 2024-04-14T13:45:30.337 DEBUG:teuthology.orchestra.run.smithi122:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-04-14T13:45:30.404 DEBUG:teuthology.orchestra.run.smithi069:> sudo service rsyslog restart 2024-04-14T13:45:30.407 DEBUG:teuthology.orchestra.run.smithi122:> sudo service rsyslog restart 2024-04-14T13:45:30.464 INFO:teuthology.orchestra.run.smithi069.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-14T13:45:30.491 INFO:teuthology.orchestra.run.smithi122.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-14T13:45:30.881 INFO:teuthology.run_tasks:Running task internal.timer... 2024-04-14T13:45:30.889 INFO:teuthology.task.internal:Starting timer... 2024-04-14T13:45:30.889 INFO:teuthology.run_tasks:Running task pcp... 2024-04-14T13:45:30.919 INFO:teuthology.run_tasks:Running task selinux... 2024-04-14T13:45:30.952 DEBUG:teuthology.orchestra.run.smithi069:> sudo service auditd rotate 2024-04-14T13:45:31.019 INFO:teuthology.orchestra.run.smithi069.stdout:Rotating logs: 2024-04-14T13:45:31.021 DEBUG:teuthology.orchestra.run.smithi122:> sudo service auditd rotate 2024-04-14T13:45:31.069 INFO:teuthology.orchestra.run.smithi122.stdout:Rotating logs: 2024-04-14T13:45:31.071 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-04-14T13:45:31.071 DEBUG:teuthology.orchestra.run.smithi069:> /usr/sbin/getenforce 2024-04-14T13:45:31.098 INFO:teuthology.orchestra.run.smithi069.stdout:Permissive 2024-04-14T13:45:31.098 DEBUG:teuthology.orchestra.run.smithi122:> /usr/sbin/getenforce 2024-04-14T13:45:31.122 INFO:teuthology.orchestra.run.smithi122.stdout:Permissive 2024-04-14T13:45:31.122 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi069.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi122.front.sepia.ceph.com': 'permissive'} 2024-04-14T13:45:31.123 DEBUG:teuthology.orchestra.run.smithi069:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-04-14T13:45:31.167 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T13:45:31.168 DEBUG:teuthology.orchestra.run.smithi122:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-04-14T13:45:31.195 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T13:45:31.195 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-04-14T13:45:31.195 DEBUG:teuthology.orchestra.run.smithi069:> sudo /usr/sbin/setenforce permissive 2024-04-14T13:45:31.243 DEBUG:teuthology.orchestra.run.smithi122:> sudo /usr/sbin/setenforce permissive 2024-04-14T13:45:31.274 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-04-14T13:45:31.292 DEBUG:teuthology.repo_utils:Setting repo remote to https://git.ceph.com/ceph-cm-ansible.git 2024-04-14T13:45:31.318 INFO:teuthology.repo_utils:Fetching git.ceph.com_ceph-cm-ansible_main from origin 2024-04-14T13:45:31.469 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-04-14T13:45:31.514 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-04-14T13:45:31.516 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi069.front.sepia.ceph.com,smithi122.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-04-14T13:49:48.157 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi069.front.sepia.ceph.com'), Remote(name='ubuntu@smithi122.front.sepia.ceph.com')] 2024-04-14T13:49:48.158 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi069.front.sepia.ceph.com' 2024-04-14T13:49:48.160 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi069.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-14T13:49:48.252 DEBUG:teuthology.orchestra.run.smithi069:> true 2024-04-14T13:49:48.324 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi069.front.sepia.ceph.com' 2024-04-14T13:49:48.325 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi122.front.sepia.ceph.com' 2024-04-14T13:49:48.325 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi122.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-14T13:49:48.412 DEBUG:teuthology.orchestra.run.smithi122:> true 2024-04-14T13:49:48.485 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi122.front.sepia.ceph.com' 2024-04-14T13:49:48.485 INFO:teuthology.run_tasks:Running task clock... 2024-04-14T13:49:48.498 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-04-14T13:49:48.498 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-04-14T13:49:48.499 DEBUG:teuthology.orchestra.run.smithi069:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-14T13:49:48.501 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-04-14T13:49:48.501 DEBUG:teuthology.orchestra.run.smithi122:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-14T13:49:48.530 INFO:teuthology.orchestra.run.smithi069.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-04-14T13:49:48.546 INFO:teuthology.orchestra.run.smithi069.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-04-14T13:49:48.555 INFO:teuthology.orchestra.run.smithi122.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-04-14T13:49:48.572 INFO:teuthology.orchestra.run.smithi122.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-04-14T13:49:48.576 INFO:teuthology.orchestra.run.smithi069.stderr:sudo: ntpd: command not found 2024-04-14T13:49:48.589 INFO:teuthology.orchestra.run.smithi069.stdout:506 Cannot talk to daemon 2024-04-14T13:49:48.601 INFO:teuthology.orchestra.run.smithi122.stderr:sudo: ntpd: command not found 2024-04-14T13:49:48.605 INFO:teuthology.orchestra.run.smithi069.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-04-14T13:49:48.614 INFO:teuthology.orchestra.run.smithi122.stdout:506 Cannot talk to daemon 2024-04-14T13:49:48.622 INFO:teuthology.orchestra.run.smithi069.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-04-14T13:49:48.631 INFO:teuthology.orchestra.run.smithi122.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-04-14T13:49:48.647 INFO:teuthology.orchestra.run.smithi122.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-04-14T13:49:48.685 INFO:teuthology.orchestra.run.smithi069.stderr:bash: line 1: ntpq: command not found 2024-04-14T13:49:48.688 INFO:teuthology.orchestra.run.smithi069.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-04-14T13:49:48.688 INFO:teuthology.orchestra.run.smithi069.stdout:=============================================================================== 2024-04-14T13:49:48.688 INFO:teuthology.orchestra.run.smithi069.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-14T13:49:48.688 INFO:teuthology.orchestra.run.smithi069.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-14T13:49:48.688 INFO:teuthology.orchestra.run.smithi069.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-14T13:49:48.688 INFO:teuthology.orchestra.run.smithi069.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-14T13:49:48.699 INFO:teuthology.orchestra.run.smithi122.stderr:bash: line 1: ntpq: command not found 2024-04-14T13:49:48.703 INFO:teuthology.orchestra.run.smithi122.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-04-14T13:49:48.703 INFO:teuthology.orchestra.run.smithi122.stdout:=============================================================================== 2024-04-14T13:49:48.703 INFO:teuthology.orchestra.run.smithi122.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-14T13:49:48.703 INFO:teuthology.orchestra.run.smithi122.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-14T13:49:48.703 INFO:teuthology.orchestra.run.smithi122.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-14T13:49:48.703 INFO:teuthology.orchestra.run.smithi122.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-14T13:49:48.704 INFO:teuthology.run_tasks:Running task cephadm... 2024-04-14T13:49:48.798 INFO:tasks.cephadm:Config: {'cephadm_branch': 'v17.2.0', 'cephadm_git_url': 'https://github.com/ceph/ceph', 'image': 'quay.io/ceph/ceph:v17.2.0', 'conf': {'global': {'mon election default strategy': 3}, 'mgr': {'debug mgr': 20, 'debug ms': 1, 'mgr/cephadm/use_agent': False}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'debug ms': 1, 'debug osd': 20}}, 'flavor': 'default', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)'], 'sha1': 'a971fd5f5719038c70b817bafacd6d5400b1d888'} 2024-04-14T13:49:48.799 INFO:tasks.cephadm:Cluster image is quay.io/ceph/ceph:v17.2.0 2024-04-14T13:49:48.799 INFO:tasks.cephadm:Cluster fsid is dc326fc2-fa65-11ee-bc8e-c7b262605968 2024-04-14T13:49:48.800 INFO:tasks.cephadm:Choosing monitor IPs and ports... 2024-04-14T13:49:48.800 INFO:tasks.cephadm:Monitor IPs: {'mon.a': '172.21.15.69', 'mon.c': '[v2:172.21.15.69:3301,v1:172.21.15.69:6790]', 'mon.b': '172.21.15.122'} 2024-04-14T13:49:48.800 INFO:tasks.cephadm:First mon is mon.a on smithi069 2024-04-14T13:49:48.800 INFO:tasks.cephadm:First mgr is y 2024-04-14T13:49:48.800 INFO:tasks.cephadm:Normalizing hostnames... 2024-04-14T13:49:48.800 DEBUG:teuthology.orchestra.run.smithi069:> sudo hostname $(hostname -s) 2024-04-14T13:49:48.828 DEBUG:teuthology.orchestra.run.smithi122:> sudo hostname $(hostname -s) 2024-04-14T13:49:48.856 INFO:tasks.cephadm:Downloading cephadm (repo https://github.com/ceph/ceph ref v17.2.0)... 2024-04-14T13:49:48.856 DEBUG:teuthology.orchestra.run.smithi069:> curl --silent https://raw.githubusercontent.com/ceph/ceph/v17.2.0/src/cephadm/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-04-14T13:49:49.096 INFO:teuthology.orchestra.run.smithi069.stdout:-rw-r--r--. 1 ubuntu ubuntu 320521 Apr 14 13:49 /home/ubuntu/cephtest/cephadm 2024-04-14T13:49:49.097 DEBUG:teuthology.orchestra.run.smithi122:> curl --silent https://raw.githubusercontent.com/ceph/ceph/v17.2.0/src/cephadm/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-04-14T13:49:49.267 INFO:teuthology.orchestra.run.smithi122.stdout:-rw-r--r--. 1 ubuntu ubuntu 320521 Apr 14 13:49 /home/ubuntu/cephtest/cephadm 2024-04-14T13:49:49.267 DEBUG:teuthology.orchestra.run.smithi069:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-04-14T13:49:49.282 DEBUG:teuthology.orchestra.run.smithi122:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-04-14T13:49:49.308 INFO:tasks.cephadm:Pulling image quay.io/ceph/ceph:v17.2.0 on all hosts... 2024-04-14T13:49:49.308 DEBUG:teuthology.orchestra.run.smithi069:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 pull 2024-04-14T13:49:49.324 DEBUG:teuthology.orchestra.run.smithi122:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 pull 2024-04-14T13:49:50.189 INFO:teuthology.orchestra.run.smithi069.stderr:Pulling container image quay.io/ceph/ceph:v17.2.0... 2024-04-14T13:49:50.246 INFO:teuthology.orchestra.run.smithi122.stderr:Pulling container image quay.io/ceph/ceph:v17.2.0... 2024-04-14T13:50:10.322 INFO:teuthology.orchestra.run.smithi069.stdout:{ 2024-04-14T13:50:10.322 INFO:teuthology.orchestra.run.smithi069.stdout: "ceph_version": "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)", 2024-04-14T13:50:10.323 INFO:teuthology.orchestra.run.smithi069.stdout: "image_id": "e1d6a67b021eb077ee22bf650f1a9fb1980a2cf5c36bdb9cba9eac6de8f702d9", 2024-04-14T13:50:10.323 INFO:teuthology.orchestra.run.smithi069.stdout: "repo_digests": [ 2024-04-14T13:50:10.323 INFO:teuthology.orchestra.run.smithi069.stdout: "quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a", 2024-04-14T13:50:10.323 INFO:teuthology.orchestra.run.smithi069.stdout: "quay.io/ceph/ceph@sha256:cb4d698cb769b6aba05bf6ef04f41a7fe694160140347576e13bd9348514b667" 2024-04-14T13:50:10.323 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2024-04-14T13:50:10.323 INFO:teuthology.orchestra.run.smithi069.stdout:} 2024-04-14T13:50:11.981 INFO:teuthology.orchestra.run.smithi122.stdout:{ 2024-04-14T13:50:11.981 INFO:teuthology.orchestra.run.smithi122.stdout: "ceph_version": "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)", 2024-04-14T13:50:11.981 INFO:teuthology.orchestra.run.smithi122.stdout: "image_id": "e1d6a67b021eb077ee22bf650f1a9fb1980a2cf5c36bdb9cba9eac6de8f702d9", 2024-04-14T13:50:11.981 INFO:teuthology.orchestra.run.smithi122.stdout: "repo_digests": [ 2024-04-14T13:50:11.981 INFO:teuthology.orchestra.run.smithi122.stdout: "quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a", 2024-04-14T13:50:11.981 INFO:teuthology.orchestra.run.smithi122.stdout: "quay.io/ceph/ceph@sha256:cb4d698cb769b6aba05bf6ef04f41a7fe694160140347576e13bd9348514b667" 2024-04-14T13:50:11.981 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-04-14T13:50:11.981 INFO:teuthology.orchestra.run.smithi122.stdout:} 2024-04-14T13:50:11.999 DEBUG:teuthology.orchestra.run.smithi069:> sudo mkdir -p /etc/ceph 2024-04-14T13:50:12.029 DEBUG:teuthology.orchestra.run.smithi122:> sudo mkdir -p /etc/ceph 2024-04-14T13:50:12.058 DEBUG:teuthology.orchestra.run.smithi069:> sudo chmod 777 /etc/ceph 2024-04-14T13:50:12.096 DEBUG:teuthology.orchestra.run.smithi122:> sudo chmod 777 /etc/ceph 2024-04-14T13:50:12.126 INFO:tasks.cephadm:Writing seed config... 2024-04-14T13:50:12.127 INFO:tasks.cephadm: override: [global] mon election default strategy = 3 2024-04-14T13:50:12.127 INFO:tasks.cephadm: override: [mgr] debug mgr = 20 2024-04-14T13:50:12.128 INFO:tasks.cephadm: override: [mgr] debug ms = 1 2024-04-14T13:50:12.128 INFO:tasks.cephadm: override: [mgr] mgr/cephadm/use_agent = False 2024-04-14T13:50:12.128 INFO:tasks.cephadm: override: [mon] debug mon = 20 2024-04-14T13:50:12.128 INFO:tasks.cephadm: override: [mon] debug ms = 1 2024-04-14T13:50:12.128 INFO:tasks.cephadm: override: [mon] debug paxos = 20 2024-04-14T13:50:12.128 INFO:tasks.cephadm: override: [osd] debug ms = 1 2024-04-14T13:50:12.128 INFO:tasks.cephadm: override: [osd] debug osd = 20 2024-04-14T13:50:12.128 DEBUG:teuthology.orchestra.run.smithi069:> set -ex 2024-04-14T13:50:12.129 DEBUG:teuthology.orchestra.run.smithi069:> dd of=/home/ubuntu/cephtest/seed.ceph.conf 2024-04-14T13:50:12.151 DEBUG:tasks.cephadm:Final config: [global] # make logging friendly to teuthology log_to_file = true log_to_stderr = false log to journald = false mon cluster log to file = true mon cluster log file level = debug mon clock drift allowed = 1.000 # replicate across OSDs, not hosts osd crush chooseleaf type = 0 #osd pool default size = 2 osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd # enable some debugging auth debug = true ms die on old message = true ms die on bug = true debug asserts on shutdown = true # adjust warnings mon max pg per osd = 10000# >= luminous mon pg warn max object skew = 0 mon osd allow primary affinity = true mon osd allow pg remap = true mon warn on legacy crush tunables = false mon warn on crush straw calc version zero = false mon warn on no sortbitwise = false mon warn on osd down out interval zero = false mon warn on too few osds = false mon_warn_on_pool_pg_num_not_power_of_two = false # disable pg_autoscaler by default for new pools osd_pool_default_pg_autoscale_mode = off # tests delete pools mon allow pool delete = true fsid = dc326fc2-fa65-11ee-bc8e-c7b262605968 mon election default strategy = 3 [osd] osd scrub load threshold = 5.0 osd scrub max interval = 600 osd mclock profile = high_recovery_ops osd recover clone overlap = true osd recovery max chunk = 1048576 osd deep scrub update digest min age = 30 osd map max advance = 10 osd memory target autotune = true # debugging osd debug shutdown = true osd debug op order = true osd debug verify stray on activate = true osd debug pg log writeout = true osd debug verify cached snaps = true osd debug verify missing on start = true osd debug misdirected ops = true osd op queue = debug_random osd op queue cut off = debug_random osd shutdown pgref assert = true bdev debug aio = true osd sloppy crc = true debug ms = 1 debug osd = 20 [mgr] mon reweight min pgs per osd = 4 mon reweight min bytes per osd = 10 mgr/telemetry/nag = false debug mgr = 20 debug ms = 1 mgr/cephadm/use_agent = False [mon] mon data avail warn = 5 mon mgr mkfs grace = 240 mon reweight min pgs per osd = 4 mon osd reporter subtree level = osd mon osd prime pg temp = true mon reweight min bytes per osd = 10 # rotate auth tickets quickly to exercise renewal paths auth mon ticket ttl = 660# 11m auth service ticket ttl = 240# 4m # don't complain about global id reclaim mon_warn_on_insecure_global_id_reclaim = false mon_warn_on_insecure_global_id_reclaim_allowed = false debug mon = 20 debug ms = 1 debug paxos = 20 [client.rgw] rgw cache enabled = true rgw enable ops log = true rgw enable usage log = true 2024-04-14T13:50:12.151 DEBUG:teuthology.orchestra.run.smithi069:mon.a> sudo journalctl -f -n 0 -u ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968@mon.a.service 2024-04-14T13:50:12.194 DEBUG:teuthology.orchestra.run.smithi069:mgr.y> sudo journalctl -f -n 0 -u ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968@mgr.y.service 2024-04-14T13:50:12.237 INFO:tasks.cephadm:Bootstrapping... 2024-04-14T13:50:12.237 DEBUG:teuthology.orchestra.run.smithi069:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 -v bootstrap --fsid dc326fc2-fa65-11ee-bc8e-c7b262605968 --config /home/ubuntu/cephtest/seed.ceph.conf --output-config /etc/ceph/ceph.conf --output-keyring /etc/ceph/ceph.client.admin.keyring --output-pub-ssh-key /home/ubuntu/cephtest/ceph.pub --mon-id a --mgr-id y --orphan-initial-daemons --skip-monitoring-stack --mon-ip 172.21.15.69 --skip-admin-label && sudo chmod +r /etc/ceph/ceph.client.admin.keyring 2024-04-14T13:50:12.436 INFO:teuthology.orchestra.run.smithi069.stderr:-------------------------------------------------------------------------------- 2024-04-14T13:50:12.436 INFO:teuthology.orchestra.run.smithi069.stderr:cephadm ['--image', 'quay.io/ceph/ceph:v17.2.0', '-v', 'bootstrap', '--fsid', 'dc326fc2-fa65-11ee-bc8e-c7b262605968', '--config', '/home/ubuntu/cephtest/seed.ceph.conf', '--output-config', '/etc/ceph/ceph.conf', '--output-keyring', '/etc/ceph/ceph.client.admin.keyring', '--output-pub-ssh-key', '/home/ubuntu/cephtest/ceph.pub', '--mon-id', 'a', '--mgr-id', 'y', '--orphan-initial-daemons', '--skip-monitoring-stack', '--mon-ip', '172.21.15.69', '--skip-admin-label'] 2024-04-14T13:50:12.460 INFO:teuthology.orchestra.run.smithi069.stderr:/bin/podman: 4.9.4-dev 2024-04-14T13:50:12.462 INFO:teuthology.orchestra.run.smithi069.stderr:Verifying podman|docker is present... 2024-04-14T13:50:12.485 INFO:teuthology.orchestra.run.smithi069.stderr:/bin/podman: 4.9.4-dev 2024-04-14T13:50:12.488 INFO:teuthology.orchestra.run.smithi069.stderr:Verifying lvm2 is present... 2024-04-14T13:50:12.488 INFO:teuthology.orchestra.run.smithi069.stderr:Verifying time synchronization is in place... 2024-04-14T13:50:12.494 INFO:teuthology.orchestra.run.smithi069.stderr:systemctl: Failed to get unit file state for chrony.service: No such file or directory 2024-04-14T13:50:12.501 INFO:teuthology.orchestra.run.smithi069.stderr:systemctl: inactive 2024-04-14T13:50:12.507 INFO:teuthology.orchestra.run.smithi069.stderr:systemctl: enabled 2024-04-14T13:50:12.513 INFO:teuthology.orchestra.run.smithi069.stderr:systemctl: active 2024-04-14T13:50:12.514 INFO:teuthology.orchestra.run.smithi069.stderr:Unit chronyd.service is enabled and running 2024-04-14T13:50:12.514 INFO:teuthology.orchestra.run.smithi069.stderr:Repeating the final host check... 2024-04-14T13:50:12.535 INFO:teuthology.orchestra.run.smithi069.stderr:/bin/podman: 4.9.4-dev 2024-04-14T13:50:12.537 INFO:teuthology.orchestra.run.smithi069.stderr:podman (/bin/podman) version 4.9.4 is present 2024-04-14T13:50:12.537 INFO:teuthology.orchestra.run.smithi069.stderr:systemctl is present 2024-04-14T13:50:12.537 INFO:teuthology.orchestra.run.smithi069.stderr:lvcreate is present 2024-04-14T13:50:12.543 INFO:teuthology.orchestra.run.smithi069.stderr:systemctl: Failed to get unit file state for chrony.service: No such file or directory 2024-04-14T13:50:12.549 INFO:teuthology.orchestra.run.smithi069.stderr:systemctl: inactive 2024-04-14T13:50:12.556 INFO:teuthology.orchestra.run.smithi069.stderr:systemctl: enabled 2024-04-14T13:50:12.561 INFO:teuthology.orchestra.run.smithi069.stderr:systemctl: active 2024-04-14T13:50:12.562 INFO:teuthology.orchestra.run.smithi069.stderr:Unit chronyd.service is enabled and running 2024-04-14T13:50:12.562 INFO:teuthology.orchestra.run.smithi069.stderr:Host looks OK 2024-04-14T13:50:12.562 INFO:teuthology.orchestra.run.smithi069.stderr:Cluster fsid: dc326fc2-fa65-11ee-bc8e-c7b262605968 2024-04-14T13:50:12.563 INFO:teuthology.orchestra.run.smithi069.stderr:Acquiring lock 140718401802144 on /run/cephadm/dc326fc2-fa65-11ee-bc8e-c7b262605968.lock 2024-04-14T13:50:12.563 INFO:teuthology.orchestra.run.smithi069.stderr:Lock 140718401802144 acquired on /run/cephadm/dc326fc2-fa65-11ee-bc8e-c7b262605968.lock 2024-04-14T13:50:12.563 INFO:teuthology.orchestra.run.smithi069.stderr:Verifying IP 172.21.15.69 port 3300 ... 2024-04-14T13:50:12.563 INFO:teuthology.orchestra.run.smithi069.stderr:Verifying IP 172.21.15.69 port 6789 ... 2024-04-14T13:50:12.563 INFO:teuthology.orchestra.run.smithi069.stderr:Base mon IP is 172.21.15.69, final addrv is [v2:172.21.15.69:3300,v1:172.21.15.69:6789] 2024-04-14T13:50:12.565 INFO:teuthology.orchestra.run.smithi069.stderr:/sbin/ip: default via 172.21.15.254 dev enp3s0f1 proto dhcp src 172.21.15.69 metric 100 2024-04-14T13:50:12.566 INFO:teuthology.orchestra.run.smithi069.stderr:/sbin/ip: 172.21.0.0/20 dev enp3s0f1 proto kernel scope link src 172.21.15.69 metric 100 2024-04-14T13:50:12.568 INFO:teuthology.orchestra.run.smithi069.stderr:/sbin/ip: ::1 dev lo proto kernel metric 256 pref medium 2024-04-14T13:50:12.568 INFO:teuthology.orchestra.run.smithi069.stderr:/sbin/ip: fe80::/64 dev enp3s0f1 proto kernel metric 256 pref medium 2024-04-14T13:50:12.568 INFO:teuthology.orchestra.run.smithi069.stderr:/sbin/ip: default via fe80::327c:5e00:6487:71e0 dev enp3s0f1 proto ra metric 1024 expires 1796sec hoplimit 64 pref medium 2024-04-14T13:50:12.571 INFO:teuthology.orchestra.run.smithi069.stderr:/sbin/ip: 1: lo: mtu 65536 state UNKNOWN qlen 1000 2024-04-14T13:50:12.571 INFO:teuthology.orchestra.run.smithi069.stderr:/sbin/ip: inet6 ::1/128 scope host 2024-04-14T13:50:12.571 INFO:teuthology.orchestra.run.smithi069.stderr:/sbin/ip: valid_lft forever preferred_lft forever 2024-04-14T13:50:12.571 INFO:teuthology.orchestra.run.smithi069.stderr:/sbin/ip: 5: enp3s0f1: mtu 1500 state UP qlen 1000 2024-04-14T13:50:12.571 INFO:teuthology.orchestra.run.smithi069.stderr:/sbin/ip: inet6 fe80::ec4:7aff:fe8f:d737/64 scope link 2024-04-14T13:50:12.571 INFO:teuthology.orchestra.run.smithi069.stderr:/sbin/ip: valid_lft forever preferred_lft forever 2024-04-14T13:50:12.572 INFO:teuthology.orchestra.run.smithi069.stderr:Mon IP `172.21.15.69` is in CIDR network `172.21.0.0/20` 2024-04-14T13:50:12.572 INFO:teuthology.orchestra.run.smithi069.stderr:- internal network (--cluster-network) has not been provided, OSD replication will default to the public_network 2024-04-14T13:50:12.572 INFO:teuthology.orchestra.run.smithi069.stderr:Pulling container image quay.io/ceph/ceph:v17.2.0... 2024-04-14T13:50:12.707 INFO:teuthology.orchestra.run.smithi069.stderr:/bin/podman: Trying to pull quay.io/ceph/ceph:v17.2.0... 2024-04-14T13:50:13.124 INFO:teuthology.orchestra.run.smithi069.stderr:/bin/podman: Getting image source signatures 2024-04-14T13:50:13.125 INFO:teuthology.orchestra.run.smithi069.stderr:/bin/podman: Copying blob sha256:c32ab78b488d0c72f64eded765c0cf6b5bf2c75dab66cb62a9d367fa6ec42513 2024-04-14T13:50:13.125 INFO:teuthology.orchestra.run.smithi069.stderr:/bin/podman: Copying blob sha256:a70843738bb77e1ab9c1f85969ebdfa55f178e746be081d1cb4f94011f69eb7c 2024-04-14T13:50:13.125 INFO:teuthology.orchestra.run.smithi069.stderr:/bin/podman: Copying blob sha256:89b4a75bc2d8500f15463747507c9623df43886c134463e7f0527e70900e7a7b 2024-04-14T13:50:13.125 INFO:teuthology.orchestra.run.smithi069.stderr:/bin/podman: Copying blob sha256:599d07cb321ff0a3c82224e1138fc685793fa69b93ed5780415751a5f7e4b8c2 2024-04-14T13:50:13.125 INFO:teuthology.orchestra.run.smithi069.stderr:/bin/podman: Copying blob sha256:33ca8fff7868c4dc0c11e09bca97c720eb9cfbab7221216754367dd8de70388a 2024-04-14T13:50:13.125 INFO:teuthology.orchestra.run.smithi069.stderr:/bin/podman: Copying config sha256:e1d6a67b021eb077ee22bf650f1a9fb1980a2cf5c36bdb9cba9eac6de8f702d9 2024-04-14T13:50:13.265 INFO:teuthology.orchestra.run.smithi069.stderr:/bin/podman: Writing manifest to image destination 2024-04-14T13:50:13.418 INFO:teuthology.orchestra.run.smithi069.stderr:/bin/podman: e1d6a67b021eb077ee22bf650f1a9fb1980a2cf5c36bdb9cba9eac6de8f702d9 2024-04-14T13:50:13.800 INFO:teuthology.orchestra.run.smithi069.stderr:ceph: ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable) 2024-04-14T13:50:14.342 INFO:teuthology.orchestra.run.smithi069.stderr:Ceph version: ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable) 2024-04-14T13:50:14.342 INFO:teuthology.orchestra.run.smithi069.stderr:Extracting ceph user uid/gid from container image... 2024-04-14T13:50:14.700 INFO:teuthology.orchestra.run.smithi069.stderr:stat: 167 167 2024-04-14T13:50:15.247 INFO:teuthology.orchestra.run.smithi069.stderr:Creating initial keys... 2024-04-14T13:50:15.622 INFO:teuthology.orchestra.run.smithi069.stderr:/usr/bin/ceph-authtool: AQAX3xtmxO3uHxAA9ijzjpNVI+dF+5Jwi+82vQ== 2024-04-14T13:50:16.544 INFO:teuthology.orchestra.run.smithi069.stderr:/usr/bin/ceph-authtool: AQAY3xtm7h9ZGxAAOBQZWSMidjn53WCK9wQdhw== 2024-04-14T13:50:17.408 INFO:teuthology.orchestra.run.smithi069.stderr:/usr/bin/ceph-authtool: AQAZ3xtm5auuExAAAWfsGOxkEkIWTEIRYVyxEw== 2024-04-14T13:50:17.849 INFO:teuthology.orchestra.run.smithi069.stderr:Creating initial monmap... 2024-04-14T13:50:18.255 INFO:teuthology.orchestra.run.smithi069.stderr:/usr/bin/monmaptool: /usr/bin/monmaptool: monmap file /tmp/monmap 2024-04-14T13:50:18.256 INFO:teuthology.orchestra.run.smithi069.stderr:/usr/bin/monmaptool: setting min_mon_release = octopus 2024-04-14T13:50:18.256 INFO:teuthology.orchestra.run.smithi069.stderr:/usr/bin/monmaptool: /usr/bin/monmaptool: set fsid to dc326fc2-fa65-11ee-bc8e-c7b262605968 2024-04-14T13:50:18.256 INFO:teuthology.orchestra.run.smithi069.stderr:/usr/bin/monmaptool: /usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-04-14T13:50:18.812 INFO:teuthology.orchestra.run.smithi069.stderr:monmaptool for a [v2:172.21.15.69:3300,v1:172.21.15.69:6789] on /usr/bin/monmaptool: monmap file /tmp/monmap 2024-04-14T13:50:18.812 INFO:teuthology.orchestra.run.smithi069.stderr:setting min_mon_release = octopus 2024-04-14T13:50:18.813 INFO:teuthology.orchestra.run.smithi069.stderr:/usr/bin/monmaptool: set fsid to dc326fc2-fa65-11ee-bc8e-c7b262605968 2024-04-14T13:50:18.813 INFO:teuthology.orchestra.run.smithi069.stderr:/usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-04-14T13:50:18.813 INFO:teuthology.orchestra.run.smithi069.stderr: 2024-04-14T13:50:18.813 INFO:teuthology.orchestra.run.smithi069.stderr:Creating mon... 2024-04-14T13:50:19.776 INFO:teuthology.orchestra.run.smithi069.stderr:create mon.a on 2024-04-14T13:50:19.973 INFO:teuthology.orchestra.run.smithi069.stderr:systemctl: Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /etc/systemd/system/ceph.target. 2024-04-14T13:50:20.122 INFO:teuthology.orchestra.run.smithi069.stderr:systemctl: Created symlink /etc/systemd/system/multi-user.target.wants/ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968.target → /etc/systemd/system/ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968.target. 2024-04-14T13:50:20.122 INFO:teuthology.orchestra.run.smithi069.stderr:systemctl: Created symlink /etc/systemd/system/ceph.target.wants/ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968.target → /etc/systemd/system/ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968.target. 2024-04-14T13:50:20.429 INFO:teuthology.orchestra.run.smithi069.stderr:systemctl: Failed to reset failed state of unit ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968@mon.a.service: Unit ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968@mon.a.service not loaded. 2024-04-14T13:50:20.437 INFO:teuthology.orchestra.run.smithi069.stderr:systemctl: Created symlink /etc/systemd/system/ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968.target.wants/ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968@mon.a.service → /etc/systemd/system/ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968@.service. 2024-04-14T13:50:21.060 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:50:21 smithi069 ceph-mon[24480]: mon.a is new leader, mons a in quorum (ranks 0) 2024-04-14T13:50:21.072 INFO:teuthology.orchestra.run.smithi069.stderr:systemctl: disabled 2024-04-14T13:50:21.078 INFO:teuthology.orchestra.run.smithi069.stderr:systemctl: inactive 2024-04-14T13:50:21.078 INFO:teuthology.orchestra.run.smithi069.stderr:firewalld.service is not enabled 2024-04-14T13:50:21.078 INFO:teuthology.orchestra.run.smithi069.stderr:Not possible to enable service . firewalld.service is not available 2024-04-14T13:50:21.079 INFO:teuthology.orchestra.run.smithi069.stderr:Waiting for mon to start... 2024-04-14T13:50:21.079 INFO:teuthology.orchestra.run.smithi069.stderr:Waiting for mon... 2024-04-14T13:50:21.316 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:50:21 smithi069 bash[24461]: 6c55a449156ee942708672390ce29036b5edf51d92be1f3bf41bfd1d374f2ff4 2024-04-14T13:50:21.316 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:50:21 smithi069 systemd[1]: Started Ceph mon.a for dc326fc2-fa65-11ee-bc8e-c7b262605968. 2024-04-14T13:50:21.567 INFO:teuthology.orchestra.run.smithi069.stderr:/usr/bin/ceph: cluster: 2024-04-14T13:50:21.568 INFO:teuthology.orchestra.run.smithi069.stderr:/usr/bin/ceph: id: dc326fc2-fa65-11ee-bc8e-c7b262605968 2024-04-14T13:50:21.568 INFO:teuthology.orchestra.run.smithi069.stderr:/usr/bin/ceph: health: HEALTH_OK 2024-04-14T13:50:21.568 INFO:teuthology.orchestra.run.smithi069.stderr:/usr/bin/ceph: 2024-04-14T13:50:21.568 INFO:teuthology.orchestra.run.smithi069.stderr:/usr/bin/ceph: services: 2024-04-14T13:50:21.568 INFO:teuthology.orchestra.run.smithi069.stderr:/usr/bin/ceph: mon: 1 daemons, quorum a (age 0.540319s) 2024-04-14T13:50:21.568 INFO:teuthology.orchestra.run.smithi069.stderr:/usr/bin/ceph: mgr: no daemons active 2024-04-14T13:50:21.568 INFO:teuthology.orchestra.run.smithi069.stderr:/usr/bin/ceph: osd: 0 osds: 0 up, 0 in 2024-04-14T13:50:21.568 INFO:teuthology.orchestra.run.smithi069.stderr:/usr/bin/ceph: 2024-04-14T13:50:21.568 INFO:teuthology.orchestra.run.smithi069.stderr:/usr/bin/ceph: data: 2024-04-14T13:50:21.568 INFO:teuthology.orchestra.run.smithi069.stderr:/usr/bin/ceph: pools: 0 pools, 0 pgs 2024-04-14T13:50:21.568 INFO:teuthology.orchestra.run.smithi069.stderr:/usr/bin/ceph: objects: 0 objects, 0 B 2024-04-14T13:50:21.568 INFO:teuthology.orchestra.run.smithi069.stderr:/usr/bin/ceph: usage: 0 B used, 0 B / 0 B avail 2024-04-14T13:50:21.568 INFO:teuthology.orchestra.run.smithi069.stderr:/usr/bin/ceph: pgs: 2024-04-14T13:50:21.568 INFO:teuthology.orchestra.run.smithi069.stderr:/usr/bin/ceph: 2024-04-14T13:50:22.115 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:50:22 smithi069 ceph-mon[24480]: mon.a is new leader, mons a in quorum (ranks 0) 2024-04-14T13:50:22.116 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:50:22 smithi069 ceph-mon[24480]: monmap e1: 1 mons at {a=[v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0]} 2024-04-14T13:50:22.116 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:50:22 smithi069 ceph-mon[24480]: fsmap 2024-04-14T13:50:22.116 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:50:22 smithi069 ceph-mon[24480]: osdmap e1: 0 total, 0 up, 0 in 2024-04-14T13:50:22.116 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:50:22 smithi069 ceph-mon[24480]: mgrmap e1: no daemons active 2024-04-14T13:50:22.116 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:50:22 smithi069 ceph-mon[24480]: from='client.? 172.21.15.69:0/133785438' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch 2024-04-14T13:50:22.118 INFO:teuthology.orchestra.run.smithi069.stderr:mon is available 2024-04-14T13:50:22.118 INFO:teuthology.orchestra.run.smithi069.stderr:Assimilating anything we can from ceph.conf... 2024-04-14T13:50:22.580 INFO:teuthology.orchestra.run.smithi069.stderr:/usr/bin/ceph: 2024-04-14T13:50:22.580 INFO:teuthology.orchestra.run.smithi069.stderr:/usr/bin/ceph: [global] 2024-04-14T13:50:22.580 INFO:teuthology.orchestra.run.smithi069.stderr:/usr/bin/ceph: fsid = dc326fc2-fa65-11ee-bc8e-c7b262605968 2024-04-14T13:50:22.580 INFO:teuthology.orchestra.run.smithi069.stderr:/usr/bin/ceph: mon_host = [v2:172.21.15.69:3300,v1:172.21.15.69:6789] 2024-04-14T13:50:22.580 INFO:teuthology.orchestra.run.smithi069.stderr:/usr/bin/ceph: mon_osd_allow_pg_remap = true 2024-04-14T13:50:22.580 INFO:teuthology.orchestra.run.smithi069.stderr:/usr/bin/ceph: mon_osd_allow_primary_affinity = true 2024-04-14T13:50:22.580 INFO:teuthology.orchestra.run.smithi069.stderr:/usr/bin/ceph: mon_warn_on_no_sortbitwise = false 2024-04-14T13:50:22.580 INFO:teuthology.orchestra.run.smithi069.stderr:/usr/bin/ceph: osd_crush_chooseleaf_type = 0 2024-04-14T13:50:22.580 INFO:teuthology.orchestra.run.smithi069.stderr:/usr/bin/ceph: 2024-04-14T13:50:22.580 INFO:teuthology.orchestra.run.smithi069.stderr:/usr/bin/ceph: [mgr] 2024-04-14T13:50:22.580 INFO:teuthology.orchestra.run.smithi069.stderr:/usr/bin/ceph: mgr/cephadm/use_agent = False 2024-04-14T13:50:22.580 INFO:teuthology.orchestra.run.smithi069.stderr:/usr/bin/ceph: mgr/telemetry/nag = false 2024-04-14T13:50:22.580 INFO:teuthology.orchestra.run.smithi069.stderr:/usr/bin/ceph: 2024-04-14T13:50:22.581 INFO:teuthology.orchestra.run.smithi069.stderr:/usr/bin/ceph: [osd] 2024-04-14T13:50:22.581 INFO:teuthology.orchestra.run.smithi069.stderr:/usr/bin/ceph: osd_map_max_advance = 10 2024-04-14T13:50:22.581 INFO:teuthology.orchestra.run.smithi069.stderr:/usr/bin/ceph: osd_sloppy_crc = true 2024-04-14T13:50:23.050 INFO:teuthology.orchestra.run.smithi069.stderr:Generating new minimal ceph.conf... 2024-04-14T13:50:23.155 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:50:23 smithi069 ceph-mon[24480]: from='client.? 172.21.15.69:0/2110148153' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-04-14T13:50:23.155 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:50:23 smithi069 ceph-mon[24480]: from='client.? 172.21.15.69:0/2110148153' entity='client.admin' cmd='[{"prefix": "config assimilate-conf"}]': finished 2024-04-14T13:50:24.022 INFO:teuthology.orchestra.run.smithi069.stderr:Restarting the monitor... 2024-04-14T13:50:24.318 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:50:24 smithi069 ceph-mon[24480]: from='client.? 172.21.15.69:0/2168344016' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T13:50:24.318 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:50:24 smithi069 systemd[1]: Stopping Ceph mon.a for dc326fc2-fa65-11ee-bc8e-c7b262605968... 2024-04-14T13:50:24.318 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:50:24 smithi069 bash[24697]: Error: no container with name or ID "ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mon.a" found: no such container 2024-04-14T13:50:24.318 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:50:24 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mon-a[24476]: 2024-04-14T13:50:24.169+0000 7fcebbb28700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-04-14T13:50:24.318 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:50:24 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mon-a[24476]: 2024-04-14T13:50:24.169+0000 7fcebbb28700 -1 mon.a@0(leader) e1 *** Got Signal Terminated *** 2024-04-14T13:50:24.318 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:50:24 smithi069 podman[24708]: 2024-04-14 13:50:24.210806204 +0000 UTC m=+0.136194270 container died 6c55a449156ee942708672390ce29036b5edf51d92be1f3bf41bfd1d374f2ff4 (image=quay.io/ceph/ceph:v17.2.0, name=ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mon-a, io.buildah.version=1.19.8, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, RELEASE=HEAD, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, version=8, build-date=2022-05-03T08:36:31.336870, ceph=True, distribution-scope=public, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., release=754, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, maintainer=Guillaume Abrioux , com.redhat.component=centos-stream-container, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, CEPH_POINT_RELEASE=-17.2.0, io.openshift.expose-services=, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, name=centos-stream, vendor=Red Hat, Inc., architecture=x86_64, GIT_CLEAN=True, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, GIT_BRANCH=HEAD, GIT_REPO=https://github.com/ceph/ceph-container.git, io.openshift.tags=base centos centos-stream, io.k8s.display-name=CentOS Stream 8, vcs-type=git) 2024-04-14T13:50:24.818 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:50:24 smithi069 podman[24708]: 2024-04-14 13:50:24.536188607 +0000 UTC m=+0.461576662 container cleanup 6c55a449156ee942708672390ce29036b5edf51d92be1f3bf41bfd1d374f2ff4 (image=quay.io/ceph/ceph:v17.2.0, name=ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mon-a, ceph=True, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, com.redhat.component=centos-stream-container, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, io.openshift.expose-services=, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, maintainer=Guillaume Abrioux , name=centos-stream, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., vendor=Red Hat, Inc., io.k8s.display-name=CentOS Stream 8, version=8, architecture=x86_64, release=754, GIT_BRANCH=HEAD, GIT_CLEAN=True, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, vcs-type=git, RELEASE=HEAD, io.openshift.tags=base centos centos-stream, build-date=2022-05-03T08:36:31.336870, distribution-scope=public, CEPH_POINT_RELEASE=-17.2.0, io.buildah.version=1.19.8, GIT_REPO=https://github.com/ceph/ceph-container.git, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream) 2024-04-14T13:50:24.818 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:50:24 smithi069 bash[24708]: ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mon-a 2024-04-14T13:50:24.818 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:50:24 smithi069 bash[24733]: Error: no container with name or ID "ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mon.a" found: no such container 2024-04-14T13:50:24.818 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:50:24 smithi069 podman[24722]: 2024-04-14 13:50:24.695384438 +0000 UTC m=+0.487232303 container remove 6c55a449156ee942708672390ce29036b5edf51d92be1f3bf41bfd1d374f2ff4 (image=quay.io/ceph/ceph:v17.2.0, name=ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mon-a, release=754, name=centos-stream, io.buildah.version=1.19.8, distribution-scope=public, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., io.openshift.tags=base centos centos-stream, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, com.redhat.component=centos-stream-container, io.k8s.display-name=CentOS Stream 8, vendor=Red Hat, Inc., GIT_BRANCH=HEAD, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, architecture=x86_64, vcs-type=git, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, version=8, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, CEPH_POINT_RELEASE=-17.2.0, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, maintainer=Guillaume Abrioux , com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, GIT_REPO=https://github.com/ceph/ceph-container.git, io.openshift.expose-services=, RELEASE=HEAD, ceph=True, build-date=2022-05-03T08:36:31.336870, GIT_CLEAN=True) 2024-04-14T13:50:24.818 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:50:24 smithi069 systemd[1]: ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968@mon.a.service: Deactivated successfully. 2024-04-14T13:50:24.818 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:50:24 smithi069 systemd[1]: Stopped Ceph mon.a for dc326fc2-fa65-11ee-bc8e-c7b262605968. 2024-04-14T13:50:24.818 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:50:24 smithi069 systemd[1]: Starting Ceph mon.a for dc326fc2-fa65-11ee-bc8e-c7b262605968... 2024-04-14T13:50:25.145 INFO:teuthology.orchestra.run.smithi069.stderr:Setting mon public_network to 172.21.0.0/20 2024-04-14T13:50:25.176 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:50:24 smithi069 podman[24800]: 2024-04-14 13:50:24.921104306 +0000 UTC m=+0.108800787 container create d3c553f919132d9c83653799e05d39a186cec421066a3f156339d49d1a0966b4 (image=quay.io/ceph/ceph:v17.2.0, name=ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mon-a, com.redhat.component=centos-stream-container, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, CEPH_POINT_RELEASE=-17.2.0, name=centos-stream, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, vcs-type=git, maintainer=Guillaume Abrioux , io.k8s.display-name=CentOS Stream 8, io.openshift.tags=base centos centos-stream, release=754, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, io.buildah.version=1.19.8, GIT_BRANCH=HEAD, ceph=True, version=8, io.openshift.expose-services=, RELEASE=HEAD, vendor=Red Hat, Inc., distribution-scope=public, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, GIT_REPO=https://github.com/ceph/ceph-container.git, architecture=x86_64, GIT_CLEAN=True, build-date=2022-05-03T08:36:31.336870) 2024-04-14T13:50:25.176 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:50:24 smithi069 podman[24800]: 2024-04-14 13:50:24.83012342 +0000 UTC m=+0.017819909 image pull e1d6a67b021eb077ee22bf650f1a9fb1980a2cf5c36bdb9cba9eac6de8f702d9 quay.io/ceph/ceph:v17.2.0 2024-04-14T13:50:25.176 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:50:25 smithi069 podman[24800]: 2024-04-14 13:50:25.073420692 +0000 UTC m=+0.261117173 container init d3c553f919132d9c83653799e05d39a186cec421066a3f156339d49d1a0966b4 (image=quay.io/ceph/ceph:v17.2.0, name=ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mon-a, RELEASE=HEAD, GIT_REPO=https://github.com/ceph/ceph-container.git, vcs-type=git, io.openshift.tags=base centos centos-stream, distribution-scope=public, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, io.k8s.display-name=CentOS Stream 8, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., io.openshift.expose-services=, CEPH_POINT_RELEASE=-17.2.0, vendor=Red Hat, Inc., name=centos-stream, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, ceph=True, build-date=2022-05-03T08:36:31.336870, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, GIT_BRANCH=HEAD, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, architecture=x86_64, GIT_CLEAN=True, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, version=8, com.redhat.component=centos-stream-container, release=754, maintainer=Guillaume Abrioux , io.buildah.version=1.19.8, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image) 2024-04-14T13:50:25.176 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:50:25 smithi069 podman[24800]: 2024-04-14 13:50:25.085677629 +0000 UTC m=+0.273374110 container start d3c553f919132d9c83653799e05d39a186cec421066a3f156339d49d1a0966b4 (image=quay.io/ceph/ceph:v17.2.0, name=ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mon-a, com.redhat.component=centos-stream-container, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, build-date=2022-05-03T08:36:31.336870, name=centos-stream, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., maintainer=Guillaume Abrioux , io.k8s.display-name=CentOS Stream 8, io.openshift.tags=base centos centos-stream, vendor=Red Hat, Inc., io.buildah.version=1.19.8, ceph=True, vcs-type=git, io.openshift.expose-services=, CEPH_POINT_RELEASE=-17.2.0, release=754, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, GIT_BRANCH=HEAD, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, architecture=x86_64, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, distribution-scope=public, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, RELEASE=HEAD, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, GIT_CLEAN=True, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, version=8, GIT_REPO=https://github.com/ceph/ceph-container.git) 2024-04-14T13:50:25.176 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:50:25 smithi069 ceph-mon[24818]: set uid:gid to 167:167 (ceph:ceph) 2024-04-14T13:50:25.176 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:50:25 smithi069 ceph-mon[24818]: ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable), process ceph-mon, pid 2 2024-04-14T13:50:25.176 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:50:25 smithi069 ceph-mon[24818]: pidfile_write: ignore empty --pid-file 2024-04-14T13:50:25.176 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:50:25 smithi069 ceph-mon[24818]: load: jerasure load: lrc 2024-04-14T13:50:25.176 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:50:25 smithi069 ceph-mon[24818]: rocksdb: RocksDB version: 6.15.5 2024-04-14T13:50:25.176 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:50:25 smithi069 ceph-mon[24818]: rocksdb: Git sha rocksdb_build_git_sha:@0@ 2024-04-14T13:50:25.176 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:50:25 smithi069 ceph-mon[24818]: rocksdb: Compile date Apr 18 2022 2024-04-14T13:50:25.177 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:50:25 smithi069 ceph-mon[24818]: rocksdb: DB SUMMARY 2024-04-14T13:50:25.177 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:50:25 smithi069 ceph-mon[24818]: rocksdb: DB Session ID: T6TCESRN5QAZWAXNK9YW 2024-04-14T13:50:25.177 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:50:25 smithi069 ceph-mon[24818]: rocksdb: CURRENT file: CURRENT 2024-04-14T13:50:25.177 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:50:25 smithi069 ceph-mon[24818]: rocksdb: IDENTITY file: IDENTITY 2024-04-14T13:50:25.177 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:50:25 smithi069 ceph-mon[24818]: rocksdb: MANIFEST file: MANIFEST-000009 size: 131 Bytes 2024-04-14T13:50:25.177 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:50:25 smithi069 ceph-mon[24818]: rocksdb: SST files in /var/lib/ceph/mon/ceph-a/store.db dir, Total Num: 1, files: 000008.sst 2024-04-14T13:50:25.177 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:50:25 smithi069 ceph-mon[24818]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-a/store.db: 000010.log size: 82633 ; 2024-04-14T13:50:25.177 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:50:25 smithi069 ceph-mon[24818]: rocksdb: Options.error_if_exists: 0 2024-04-14T13:50:25.177 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:50:25 smithi069 ceph-mon[24818]: rocksdb: Options.create_if_missing: 0 2024-04-14T13:50:25.177 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:50:25 smithi069 ceph-mon[24818]: rocksdb: Options.paranoid_checks: 1 2024-04-14T13:50:25.177 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:50:25 smithi069 ceph-mon[24818]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-04-14T13:50:25.177 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:50:25 smithi069 ceph-mon[24818]: rocksdb: Options.env: 0x5581050c6860 2024-04-14T13:50:25.177 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:50:25 smithi069 ceph-mon[24818]: rocksdb: Options.fs: Posix File System 2024-04-14T13:50:25.177 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:50:25 smithi069 ceph-mon[24818]: rocksdb: Options.info_log: 0x558107093dc0 2024-04-14T13:50:25.177 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:50:25 smithi069 ceph-mon[24818]: rocksdb: Options.max_file_opening_threads: 16 2024-04-14T13:50:25.178 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:50:25 smithi069 ceph-mon[24818]: rocksdb: Options.statistics: (nil) 2024-04-14T13:50:25.178 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:50:25 smithi069 ceph-mon[24818]: rocksdb: Options.use_fsync: 0 2024-04-14T13:50:25.178 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:50:25 smithi069 ceph-mon[24818]: rocksdb: Options.max_log_file_size: 0 2024-04-14T13:50:25.178 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:50:25 smithi069 ceph-mon[24818]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-04-14T13:50:25.179 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:50:25 smithi069 ceph-mon[24818]: rocksdb: Options.log_file_time_to_roll: 0 2024-04-14T13:50:25.179 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:50:25 smithi069 ceph-mon[24818]: rocksdb: Options.keep_log_file_num: 1000 2024-04-14T13:50:25.179 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:50:25 smithi069 ceph-mon[24818]: rocksdb: Options.recycle_log_file_num: 0 2024-04-14T13:50:25.179 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:50:25 smithi069 ceph-mon[24818]: rocksdb: Options.allow_fallocate: 1 2024-04-14T13:50:25.179 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:50:25 smithi069 ceph-mon[24818]: rocksdb: Options.allow_mmap_reads: 0 2024-04-14T13:50:25.179 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:50:25 smithi069 ceph-mon[24818]: rocksdb: Options.allow_mmap_writes: 0 2024-04-14T13:50:25.179 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:50:25 smithi069 ceph-mon[24818]: rocksdb: Options.use_direct_reads: 0 2024-04-14T13:50:25.180 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:50:25 smithi069 ceph-mon[24818]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-04-14T13:50:25.180 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:50:25 smithi069 ceph-mon[24818]: rocksdb: Options.create_missing_column_families: 0 2024-04-14T13:50:25.180 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:50:25 smithi069 ceph-mon[24818]: rocksdb: Options.db_log_dir: 2024-04-14T13:50:25.180 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:50:25 smithi069 ceph-mon[24818]: rocksdb: Options.wal_dir: /var/lib/ceph/mon/ceph-a/store.db 2024-04-14T13:50:25.180 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:50:25 smithi069 ceph-mon[24818]: rocksdb: Options.table_cache_numshardbits: 6 2024-04-14T13:50:25.180 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:50:25 smithi069 ceph-mon[24818]: rocksdb: Options.WAL_ttl_seconds: 0 2024-04-14T13:50:25.180 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:50:25 smithi069 ceph-mon[24818]: rocksdb: Options.WAL_size_limit_MB: 0 2024-04-14T13:50:25.180 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:50:25 smithi069 ceph-mon[24818]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-04-14T13:50:25.180 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:50:25 smithi069 ceph-mon[24818]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-04-14T13:50:25.180 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:50:25 smithi069 ceph-mon[24818]: rocksdb: Options.is_fd_close_on_exec: 1 2024-04-14T13:50:25.180 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:50:25 smithi069 ceph-mon[24818]: rocksdb: Options.advise_random_on_open: 1 2024-04-14T13:50:25.180 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:50:25 smithi069 ceph-mon[24818]: rocksdb: Options.db_write_buffer_size: 0 2024-04-14T13:50:25.180 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:50:25 smithi069 ceph-mon[24818]: rocksdb: Options.write_buffer_manager: 0x5581071842a0 2024-04-14T13:50:25.181 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:50:25 smithi069 ceph-mon[24818]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-04-14T13:50:25.181 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:50:25 smithi069 ceph-mon[24818]: rocksdb: Options.new_table_reader_for_compaction_inputs: 0 2024-04-14T13:50:25.181 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:50:25 smithi069 ceph-mon[24818]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-04-14T13:50:25.181 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:50:25 smithi069 ceph-mon[24818]: rocksdb: Options.use_adaptive_mutex: 0 2024-04-14T13:50:25.181 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:50:25 smithi069 ceph-mon[24818]: rocksdb: Options.rate_limiter: (nil) 2024-04-14T13:50:25.181 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:50:25 smithi069 ceph-mon[24818]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-04-14T13:50:25.181 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:50:25 smithi069 ceph-mon[24818]: rocksdb: Options.wal_recovery_mode: 2 2024-04-14T13:50:25.181 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:50:25 smithi069 ceph-mon[24818]: rocksdb: Options.enable_thread_tracking: 0 2024-04-14T13:50:25.181 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:50:25 smithi069 ceph-mon[24818]: rocksdb: Options.enable_pipelined_write: 0 2024-04-14T13:50:25.181 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:50:25 smithi069 ceph-mon[24818]: rocksdb: Options.unordered_write: 0 2024-04-14T13:50:25.181 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:50:25 smithi069 ceph-mon[24818]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-04-14T13:50:25.181 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:50:25 smithi069 ceph-mon[24818]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-04-14T13:50:25.181 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:50:25 smithi069 ceph-mon[24818]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-04-14T13:50:25.181 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:50:25 smithi069 ceph-mon[24818]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-04-14T13:50:25.182 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:50:25 smithi069 ceph-mon[24818]: rocksdb: Options.row_cache: None 2024-04-14T13:50:25.182 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:50:25 smithi069 ceph-mon[24818]: rocksdb: Options.wal_filter: None 2024-04-14T13:50:25.182 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:50:25 smithi069 ceph-mon[24818]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-04-14T13:50:25.182 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:50:25 smithi069 ceph-mon[24818]: rocksdb: Options.allow_ingest_behind: 0 2024-04-14T13:50:25.182 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:50:25 smithi069 ceph-mon[24818]: rocksdb: Options.preserve_deletes: 0 2024-04-14T13:50:25.182 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:50:25 smithi069 ceph-mon[24818]: rocksdb: Options.two_write_queues: 0 2024-04-14T13:50:25.182 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:50:25 smithi069 ceph-mon[24818]: rocksdb: Options.manual_wal_flush: 0 2024-04-14T13:50:25.182 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:50:25 smithi069 ceph-mon[24818]: rocksdb: Options.atomic_flush: 0 2024-04-14T13:50:25.182 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:50:25 smithi069 ceph-mon[24818]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-04-14T13:50:25.182 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:50:25 smithi069 ceph-mon[24818]: rocksdb: Options.persist_stats_to_disk: 0 2024-04-14T13:50:25.182 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:50:25 smithi069 ceph-mon[24818]: rocksdb: Options.write_dbid_to_manifest: 0 2024-04-14T13:50:25.182 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:50:25 smithi069 ceph-mon[24818]: rocksdb: Options.log_readahead_size: 0 2024-04-14T13:50:25.182 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:50:25 smithi069 ceph-mon[24818]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-04-14T13:50:25.182 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:50:25 smithi069 ceph-mon[24818]: rocksdb: Options.best_efforts_recovery: 0 2024-04-14T13:50:25.183 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:50:25 smithi069 ceph-mon[24818]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-04-14T13:50:25.183 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:50:25 smithi069 ceph-mon[24818]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-04-14T13:50:25.183 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:50:25 smithi069 ceph-mon[24818]: rocksdb: Options.allow_data_in_errors: 0 2024-04-14T13:50:25.183 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:50:25 smithi069 ceph-mon[24818]: rocksdb: Options.db_host_id: __hostname__ 2024-04-14T13:50:25.183 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:50:25 smithi069 ceph-mon[24818]: rocksdb: Options.max_background_jobs: 2 2024-04-14T13:50:25.183 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:50:25 smithi069 ceph-mon[24818]: rocksdb: Options.max_background_compactions: -1 2024-04-14T13:50:25.183 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:50:25 smithi069 ceph-mon[24818]: rocksdb: Options.max_subcompactions: 1 2024-04-14T13:50:25.183 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:50:25 smithi069 ceph-mon[24818]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-04-14T13:50:25.183 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:50:25 smithi069 ceph-mon[24818]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-04-14T13:50:25.183 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:50:25 smithi069 ceph-mon[24818]: rocksdb: Options.delayed_write_rate : 16777216 2024-04-14T13:50:25.183 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:50:25 smithi069 ceph-mon[24818]: rocksdb: Options.max_total_wal_size: 0 2024-04-14T13:50:25.183 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:50:25 smithi069 ceph-mon[24818]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-04-14T13:50:25.183 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:50:25 smithi069 ceph-mon[24818]: rocksdb: Options.stats_dump_period_sec: 600 2024-04-14T13:50:25.183 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:50:25 smithi069 ceph-mon[24818]: rocksdb: Options.stats_persist_period_sec: 600 2024-04-14T13:50:25.183 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:50:25 smithi069 ceph-mon[24818]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-04-14T13:50:25.184 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:50:25 smithi069 ceph-mon[24818]: rocksdb: Options.max_open_files: -1 2024-04-14T13:50:25.184 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:50:25 smithi069 ceph-mon[24818]: rocksdb: Options.bytes_per_sync: 0 2024-04-14T13:50:25.184 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:50:25 smithi069 ceph-mon[24818]: rocksdb: Options.wal_bytes_per_sync: 0 2024-04-14T13:50:25.184 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:50:25 smithi069 ceph-mon[24818]: rocksdb: Options.strict_bytes_per_sync: 0 2024-04-14T13:50:25.184 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:50:25 smithi069 ceph-mon[24818]: rocksdb: Options.compaction_readahead_size: 0 2024-04-14T13:50:25.184 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:50:25 smithi069 ceph-mon[24818]: rocksdb: Options.max_background_flushes: -1 2024-04-14T13:50:25.184 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:50:25 smithi069 ceph-mon[24818]: rocksdb: Compression algorithms supported: 2024-04-14T13:50:25.184 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:50:25 smithi069 ceph-mon[24818]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-04-14T13:50:25.184 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:50:25 smithi069 ceph-mon[24818]: rocksdb: kZSTD supported: 0 2024-04-14T13:50:25.184 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:50:25 smithi069 ceph-mon[24818]: rocksdb: kXpressCompression supported: 0 2024-04-14T13:50:25.184 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:50:25 smithi069 ceph-mon[24818]: rocksdb: kLZ4HCCompression supported: 1 2024-04-14T13:50:25.184 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:50:25 smithi069 ceph-mon[24818]: rocksdb: kLZ4Compression supported: 1 2024-04-14T13:50:25.184 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:50:25 smithi069 ceph-mon[24818]: rocksdb: kBZip2Compression supported: 0 2024-04-14T13:50:25.184 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:50:25 smithi069 ceph-mon[24818]: rocksdb: kZlibCompression supported: 1 2024-04-14T13:50:25.184 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:50:25 smithi069 ceph-mon[24818]: rocksdb: kSnappyCompression supported: 1 2024-04-14T13:50:25.185 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:50:25 smithi069 ceph-mon[24818]: rocksdb: Fast CRC32 supported: Supported on x86 2024-04-14T13:50:25.185 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:50:25 smithi069 ceph-mon[24818]: rocksdb: [db/version_set.cc:4725] Recovering from manifest file: /var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000009 2024-04-14T13:50:25.185 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:50:25 smithi069 ceph-mon[24818]: rocksdb: [db/column_family.cc:597] --------------- Options for column family [default]: 2024-04-14T13:50:25.185 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:50:25 smithi069 ceph-mon[24818]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-04-14T13:50:25.185 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:50:25 smithi069 ceph-mon[24818]: rocksdb: Options.merge_operator: 2024-04-14T13:50:25.185 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:50:25 smithi069 ceph-mon[24818]: rocksdb: Options.compaction_filter: None 2024-04-14T13:50:25.185 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:50:25 smithi069 ceph-mon[24818]: rocksdb: Options.compaction_filter_factory: None 2024-04-14T13:50:25.185 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:50:25 smithi069 ceph-mon[24818]: rocksdb: Options.sst_partitioner_factory: None 2024-04-14T13:50:25.185 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:50:25 smithi069 ceph-mon[24818]: rocksdb: Options.memtable_factory: SkipListFactory 2024-04-14T13:50:25.185 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:50:25 smithi069 ceph-mon[24818]: rocksdb: Options.table_factory: BlockBasedTable 2024-04-14T13:50:25.185 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:50:25 smithi069 ceph-mon[24818]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x55810705fd00) 2024-04-14T13:50:25.185 INFO:journalctl@ceph.mon.a.smithi069.stdout: cache_index_and_filter_blocks: 1 2024-04-14T13:50:25.185 INFO:journalctl@ceph.mon.a.smithi069.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-04-14T13:50:25.185 INFO:journalctl@ceph.mon.a.smithi069.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-04-14T13:50:25.186 INFO:journalctl@ceph.mon.a.smithi069.stdout: pin_top_level_index_and_filter: 1 2024-04-14T13:50:25.186 INFO:journalctl@ceph.mon.a.smithi069.stdout: index_type: 0 2024-04-14T13:50:25.186 INFO:journalctl@ceph.mon.a.smithi069.stdout: data_block_index_type: 0 2024-04-14T13:50:25.186 INFO:journalctl@ceph.mon.a.smithi069.stdout: index_shortening: 1 2024-04-14T13:50:25.186 INFO:journalctl@ceph.mon.a.smithi069.stdout: data_block_hash_table_util_ratio: 0.750000 2024-04-14T13:50:25.186 INFO:journalctl@ceph.mon.a.smithi069.stdout: hash_index_allow_collision: 1 2024-04-14T13:50:25.186 INFO:journalctl@ceph.mon.a.smithi069.stdout: checksum: 1 2024-04-14T13:50:25.186 INFO:journalctl@ceph.mon.a.smithi069.stdout: no_block_cache: 0 2024-04-14T13:50:25.186 INFO:journalctl@ceph.mon.a.smithi069.stdout: block_cache: 0x5581070ca170 2024-04-14T13:50:25.186 INFO:journalctl@ceph.mon.a.smithi069.stdout: block_cache_name: BinnedLRUCache 2024-04-14T13:50:25.186 INFO:journalctl@ceph.mon.a.smithi069.stdout: block_cache_options: 2024-04-14T13:50:25.186 INFO:journalctl@ceph.mon.a.smithi069.stdout: capacity : 536870912 2024-04-14T13:50:25.186 INFO:journalctl@ceph.mon.a.smithi069.stdout: num_shard_bits : 4 2024-04-14T13:50:25.186 INFO:journalctl@ceph.mon.a.smithi069.stdout: strict_capacity_limit : 0 2024-04-14T13:50:25.187 INFO:journalctl@ceph.mon.a.smithi069.stdout: high_pri_pool_ratio: 0.000 2024-04-14T13:50:25.187 INFO:journalctl@ceph.mon.a.smithi069.stdout: block_cache_compressed: (nil) 2024-04-14T13:50:25.187 INFO:journalctl@ceph.mon.a.smithi069.stdout: persistent_cache: (nil) 2024-04-14T13:50:25.187 INFO:journalctl@ceph.mon.a.smithi069.stdout: block_size: 4096 2024-04-14T13:50:25.187 INFO:journalctl@ceph.mon.a.smithi069.stdout: block_size_deviation: 10 2024-04-14T13:50:25.187 INFO:journalctl@ceph.mon.a.smithi069.stdout: block_restart_interval: 16 2024-04-14T13:50:25.187 INFO:journalctl@ceph.mon.a.smithi069.stdout: index_block_restart_interval: 1 2024-04-14T13:50:25.187 INFO:journalctl@ceph.mon.a.smithi069.stdout: metadata_block_size: 4096 2024-04-14T13:50:25.187 INFO:journalctl@ceph.mon.a.smithi069.stdout: partition_filters: 0 2024-04-14T13:50:25.187 INFO:journalctl@ceph.mon.a.smithi069.stdout: use_delta_encoding: 1 2024-04-14T13:50:25.187 INFO:journalctl@ceph.mon.a.smithi069.stdout: filter_policy: rocksdb.BuiltinBloomFilter 2024-04-14T13:50:25.187 INFO:journalctl@ceph.mon.a.smithi069.stdout: whole_key_filtering: 1 2024-04-14T13:50:25.187 INFO:journalctl@ceph.mon.a.smithi069.stdout: verify_compression: 0 2024-04-14T13:50:25.187 INFO:journalctl@ceph.mon.a.smithi069.stdout: read_amp_bytes_per_bit: 0 2024-04-14T13:50:25.187 INFO:journalctl@ceph.mon.a.smithi069.stdout: format_version: 4 2024-04-14T13:50:25.187 INFO:journalctl@ceph.mon.a.smithi069.stdout: enable_index_compression: 1 2024-04-14T13:50:25.188 INFO:journalctl@ceph.mon.a.smithi069.stdout: block_align: 0 2024-04-14T13:50:25.188 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:50:25 smithi069 ceph-mon[24818]: rocksdb: Options.write_buffer_size: 33554432 2024-04-14T13:50:25.188 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:50:25 smithi069 ceph-mon[24818]: rocksdb: Options.max_write_buffer_number: 2 2024-04-14T13:50:25.188 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:50:25 smithi069 ceph-mon[24818]: rocksdb: Options.compression: NoCompression 2024-04-14T13:50:25.188 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:50:25 smithi069 ceph-mon[24818]: rocksdb: Options.bottommost_compression: Disabled 2024-04-14T13:50:25.188 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:50:25 smithi069 ceph-mon[24818]: rocksdb: Options.prefix_extractor: nullptr 2024-04-14T13:50:25.188 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:50:25 smithi069 ceph-mon[24818]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-04-14T13:50:25.188 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:50:25 smithi069 ceph-mon[24818]: rocksdb: Options.num_levels: 7 2024-04-14T13:50:25.188 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:50:25 smithi069 ceph-mon[24818]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-04-14T13:50:25.188 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:50:25 smithi069 ceph-mon[24818]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-04-14T13:50:25.188 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:50:25 smithi069 ceph-mon[24818]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-04-14T13:50:25.188 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:50:25 smithi069 ceph-mon[24818]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-04-14T13:50:25.188 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:50:25 smithi069 ceph-mon[24818]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-04-14T13:50:25.188 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:50:25 smithi069 ceph-mon[24818]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-04-14T13:50:25.188 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:50:25 smithi069 ceph-mon[24818]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-04-14T13:50:25.189 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:50:25 smithi069 ceph-mon[24818]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-04-14T13:50:25.189 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:50:25 smithi069 ceph-mon[24818]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-04-14T13:50:25.189 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:50:25 smithi069 ceph-mon[24818]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-04-14T13:50:25.189 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:50:25 smithi069 ceph-mon[24818]: rocksdb: Options.compression_opts.window_bits: -14 2024-04-14T13:50:25.189 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:50:25 smithi069 ceph-mon[24818]: rocksdb: Options.compression_opts.level: 32767 2024-04-14T13:50:25.189 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:50:25 smithi069 ceph-mon[24818]: rocksdb: Options.compression_opts.strategy: 0 2024-04-14T13:50:25.189 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:50:25 smithi069 ceph-mon[24818]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-04-14T13:50:25.189 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:50:25 smithi069 ceph-mon[24818]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-04-14T13:50:25.189 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:50:25 smithi069 ceph-mon[24818]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-04-14T13:50:25.189 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:50:25 smithi069 ceph-mon[24818]: rocksdb: Options.compression_opts.enabled: false 2024-04-14T13:50:25.189 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:50:25 smithi069 ceph-mon[24818]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-04-14T13:50:25.189 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:50:25 smithi069 ceph-mon[24818]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-04-14T13:50:25.189 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:50:25 smithi069 ceph-mon[24818]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-04-14T13:50:25.189 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:50:25 smithi069 ceph-mon[24818]: rocksdb: Options.target_file_size_base: 67108864 2024-04-14T13:50:25.190 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:50:25 smithi069 ceph-mon[24818]: rocksdb: Options.target_file_size_multiplier: 1 2024-04-14T13:50:25.190 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:50:25 smithi069 ceph-mon[24818]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-04-14T13:50:25.190 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:50:25 smithi069 ceph-mon[24818]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-04-14T13:50:25.190 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:50:25 smithi069 ceph-mon[24818]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-04-14T13:50:25.190 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:50:25 smithi069 ceph-mon[24818]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-04-14T13:50:25.190 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:50:25 smithi069 ceph-mon[24818]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-04-14T13:50:25.190 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:50:25 smithi069 ceph-mon[24818]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-04-14T13:50:25.190 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:50:25 smithi069 ceph-mon[24818]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-04-14T13:50:25.190 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:50:25 smithi069 ceph-mon[24818]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-04-14T13:50:25.190 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:50:25 smithi069 ceph-mon[24818]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-04-14T13:50:25.190 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:50:25 smithi069 ceph-mon[24818]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-04-14T13:50:25.190 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:50:25 smithi069 ceph-mon[24818]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-04-14T13:50:25.190 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:50:25 smithi069 ceph-mon[24818]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-04-14T13:50:25.190 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:50:25 smithi069 ceph-mon[24818]: rocksdb: Options.arena_block_size: 4194304 2024-04-14T13:50:25.191 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:50:25 smithi069 ceph-mon[24818]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-04-14T13:50:25.191 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:50:25 smithi069 ceph-mon[24818]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-04-14T13:50:25.191 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:50:25 smithi069 ceph-mon[24818]: rocksdb: Options.rate_limit_delay_max_milliseconds: 100 2024-04-14T13:50:25.191 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:50:25 smithi069 ceph-mon[24818]: rocksdb: Options.disable_auto_compactions: 0 2024-04-14T13:50:25.191 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:50:25 smithi069 ceph-mon[24818]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-04-14T13:50:25.191 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:50:25 smithi069 ceph-mon[24818]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-04-14T13:50:25.191 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:50:25 smithi069 ceph-mon[24818]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-04-14T13:50:25.191 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:50:25 smithi069 ceph-mon[24818]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-04-14T13:50:25.191 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:50:25 smithi069 ceph-mon[24818]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-04-14T13:50:25.191 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:50:25 smithi069 ceph-mon[24818]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-04-14T13:50:25.191 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:50:25 smithi069 ceph-mon[24818]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-04-14T13:50:25.191 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:50:25 smithi069 ceph-mon[24818]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-04-14T13:50:25.191 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:50:25 smithi069 ceph-mon[24818]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-04-14T13:50:25.191 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:50:25 smithi069 ceph-mon[24818]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-04-14T13:50:25.191 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:50:25 smithi069 ceph-mon[24818]: rocksdb: Options.table_properties_collectors: 2024-04-14T13:50:25.192 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:50:25 smithi069 ceph-mon[24818]: rocksdb: Options.inplace_update_support: 0 2024-04-14T13:50:25.192 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:50:25 smithi069 ceph-mon[24818]: rocksdb: Options.inplace_update_num_locks: 10000 2024-04-14T13:50:25.192 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:50:25 smithi069 ceph-mon[24818]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-04-14T13:50:25.192 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:50:25 smithi069 ceph-mon[24818]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-04-14T13:50:25.192 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:50:25 smithi069 ceph-mon[24818]: rocksdb: Options.memtable_huge_page_size: 0 2024-04-14T13:50:25.192 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:50:25 smithi069 ceph-mon[24818]: rocksdb: Options.bloom_locality: 0 2024-04-14T13:50:25.192 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:50:25 smithi069 ceph-mon[24818]: rocksdb: Options.max_successive_merges: 0 2024-04-14T13:50:25.192 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:50:25 smithi069 ceph-mon[24818]: rocksdb: Options.optimize_filters_for_hits: 0 2024-04-14T13:50:25.192 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:50:25 smithi069 ceph-mon[24818]: rocksdb: Options.paranoid_file_checks: 0 2024-04-14T13:50:25.192 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:50:25 smithi069 ceph-mon[24818]: rocksdb: Options.force_consistency_checks: 1 2024-04-14T13:50:25.192 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:50:25 smithi069 ceph-mon[24818]: rocksdb: Options.report_bg_io_stats: 0 2024-04-14T13:50:25.192 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:50:25 smithi069 ceph-mon[24818]: rocksdb: Options.ttl: 2592000 2024-04-14T13:50:25.192 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:50:25 smithi069 ceph-mon[24818]: rocksdb: Options.periodic_compaction_seconds: 0 2024-04-14T13:50:25.192 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:50:25 smithi069 ceph-mon[24818]: rocksdb: Options.enable_blob_files: false 2024-04-14T13:50:25.193 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:50:25 smithi069 ceph-mon[24818]: rocksdb: Options.min_blob_size: 0 2024-04-14T13:50:25.193 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:50:25 smithi069 ceph-mon[24818]: rocksdb: Options.blob_file_size: 268435456 2024-04-14T13:50:25.193 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:50:25 smithi069 ceph-mon[24818]: rocksdb: Options.blob_compression_type: NoCompression 2024-04-14T13:50:25.193 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:50:25 smithi069 ceph-mon[24818]: rocksdb: Options.enable_blob_garbage_collection: false 2024-04-14T13:50:25.193 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:50:25 smithi069 ceph-mon[24818]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-04-14T13:50:25.193 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:50:25 smithi069 ceph-mon[24818]: rocksdb: [db/version_set.cc:4773] Recovered from manifest file:/var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000009 succeeded,manifest_file_number is 9, next_file_number is 11, last_sequence is 5, log_number is 5,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2024-04-14T13:50:25.193 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:50:25 smithi069 ceph-mon[24818]: rocksdb: [db/version_set.cc:4782] Column family [default] (ID 0), log number is 5 2024-04-14T13:50:25.193 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:50:25 smithi069 ceph-mon[24818]: rocksdb: [db/version_set.cc:4083] Creating manifest 13 2024-04-14T13:50:25.193 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:50:25 smithi069 ceph-mon[24818]: rocksdb: EVENT_LOG_v1 {"time_micros": 1713102625111364, "job": 1, "event": "recovery_started", "wal_files": [10]} 2024-04-14T13:50:25.193 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:50:25 smithi069 ceph-mon[24818]: rocksdb: [db/db_impl/db_impl_open.cc:847] Recovering log #10 mode 2 2024-04-14T13:50:25.193 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:50:25 smithi069 ceph-mon[24818]: rocksdb: [table/block_based/filter_policy.cc:996] Using legacy Bloom filter with high (20) bits/key. Dramatic filter space and/or accuracy improvement is available with format_version>=5. 2024-04-14T13:50:25.193 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:50:25 smithi069 ceph-mon[24818]: rocksdb: EVENT_LOG_v1 {"time_micros": 1713102625112064, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 14, "file_size": 79183, "file_checksum": "", "file_checksum_func_name": "Unknown", "table_properties": {"data_size": 77446, "index_size": 225, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 581, "raw_key_size": 10027, "raw_average_key_size": 47, "raw_value_size": 71755, "raw_average_value_size": 340, "num_data_blocks": 10, "num_entries": 211, "num_deletions": 3, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "rocksdb.BuiltinBloomFilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; ", "creation_time": 1713102625, "oldest_key_time": 0, "file_creation_time": 0, "db_id": "68dacaa5-8ecc-4064-9fea-cbab7d38d01a", "db_session_id": "T6TCESRN5QAZWAXNK9YW"}} 2024-04-14T13:50:25.194 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:50:25 smithi069 ceph-mon[24818]: rocksdb: [db/version_set.cc:4083] Creating manifest 15 2024-04-14T13:50:25.194 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:50:25 smithi069 ceph-mon[24818]: rocksdb: EVENT_LOG_v1 {"time_micros": 1713102625112406, "job": 1, "event": "recovery_finished"} 2024-04-14T13:50:25.194 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:50:25 smithi069 ceph-mon[24818]: rocksdb: [file/delete_scheduler.cc:73] Deleted file /var/lib/ceph/mon/ceph-a/store.db/000010.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-04-14T13:50:25.194 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:50:25 smithi069 ceph-mon[24818]: rocksdb: [db/db_impl/db_impl_open.cc:1701] SstFileManager instance 0x5581070b0a80 2024-04-14T13:50:25.194 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:50:25 smithi069 ceph-mon[24818]: rocksdb: DB pointer 0x5581070c0000 2024-04-14T13:50:25.194 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:50:25 smithi069 ceph-mon[24818]: rocksdb: [db/db_impl/db_impl.cc:902] ------- DUMPING STATS ------- 2024-04-14T13:50:25.194 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:50:25 smithi069 ceph-mon[24818]: rocksdb: [db/db_impl/db_impl.cc:903] 2024-04-14T13:50:25.194 INFO:journalctl@ceph.mon.a.smithi069.stdout: ** DB Stats ** 2024-04-14T13:50:25.194 INFO:journalctl@ceph.mon.a.smithi069.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-04-14T13:50:25.194 INFO:journalctl@ceph.mon.a.smithi069.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-04-14T13:50:25.194 INFO:journalctl@ceph.mon.a.smithi069.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-04-14T13:50:25.194 INFO:journalctl@ceph.mon.a.smithi069.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-04-14T13:50:25.194 INFO:journalctl@ceph.mon.a.smithi069.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-04-14T13:50:25.194 INFO:journalctl@ceph.mon.a.smithi069.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 MB, 0.00 MB/s 2024-04-14T13:50:25.195 INFO:journalctl@ceph.mon.a.smithi069.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-04-14T13:50:25.195 INFO:journalctl@ceph.mon.a.smithi069.stdout: 2024-04-14T13:50:25.195 INFO:journalctl@ceph.mon.a.smithi069.stdout: ** Compaction Stats [default] ** 2024-04-14T13:50:25.195 INFO:journalctl@ceph.mon.a.smithi069.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-04-14T13:50:25.195 INFO:journalctl@ceph.mon.a.smithi069.stdout: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-04-14T13:50:25.195 INFO:journalctl@ceph.mon.a.smithi069.stdout: L0 2/0 79.09 KB 0.5 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 162.4 0.00 0.00 1 0.000 0 0 2024-04-14T13:50:25.195 INFO:journalctl@ceph.mon.a.smithi069.stdout: Sum 2/0 79.09 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 162.4 0.00 0.00 1 0.000 0 0 2024-04-14T13:50:25.195 INFO:journalctl@ceph.mon.a.smithi069.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 162.4 0.00 0.00 1 0.000 0 0 2024-04-14T13:50:25.195 INFO:journalctl@ceph.mon.a.smithi069.stdout: 2024-04-14T13:50:25.195 INFO:journalctl@ceph.mon.a.smithi069.stdout: ** Compaction Stats [default] ** 2024-04-14T13:50:25.195 INFO:journalctl@ceph.mon.a.smithi069.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-04-14T13:50:25.195 INFO:journalctl@ceph.mon.a.smithi069.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-04-14T13:50:25.195 INFO:journalctl@ceph.mon.a.smithi069.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 162.4 0.00 0.00 1 0.000 0 0 2024-04-14T13:50:25.195 INFO:journalctl@ceph.mon.a.smithi069.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-04-14T13:50:25.195 INFO:journalctl@ceph.mon.a.smithi069.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-04-14T13:50:25.196 INFO:journalctl@ceph.mon.a.smithi069.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-04-14T13:50:25.196 INFO:journalctl@ceph.mon.a.smithi069.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-04-14T13:50:25.196 INFO:journalctl@ceph.mon.a.smithi069.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-04-14T13:50:25.196 INFO:journalctl@ceph.mon.a.smithi069.stdout: AddFile(Keys): cumulative 0, interval 0 2024-04-14T13:50:25.196 INFO:journalctl@ceph.mon.a.smithi069.stdout: Cumulative compaction: 0.00 GB write, 25.77 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-04-14T13:50:25.196 INFO:journalctl@ceph.mon.a.smithi069.stdout: Interval compaction: 0.00 GB write, 25.77 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-04-14T13:50:25.196 INFO:journalctl@ceph.mon.a.smithi069.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-04-14T13:50:25.196 INFO:journalctl@ceph.mon.a.smithi069.stdout: 2024-04-14T13:50:25.196 INFO:journalctl@ceph.mon.a.smithi069.stdout: ** File Read Latency Histogram By Level [default] ** 2024-04-14T13:50:25.196 INFO:journalctl@ceph.mon.a.smithi069.stdout: 2024-04-14T13:50:25.196 INFO:journalctl@ceph.mon.a.smithi069.stdout: ** Compaction Stats [default] ** 2024-04-14T13:50:25.196 INFO:journalctl@ceph.mon.a.smithi069.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-04-14T13:50:25.196 INFO:journalctl@ceph.mon.a.smithi069.stdout: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-04-14T13:50:25.196 INFO:journalctl@ceph.mon.a.smithi069.stdout: L0 2/0 79.09 KB 0.5 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 162.4 0.00 0.00 1 0.000 0 0 2024-04-14T13:50:25.196 INFO:journalctl@ceph.mon.a.smithi069.stdout: Sum 2/0 79.09 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 162.4 0.00 0.00 1 0.000 0 0 2024-04-14T13:50:25.197 INFO:journalctl@ceph.mon.a.smithi069.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2024-04-14T13:50:25.197 INFO:journalctl@ceph.mon.a.smithi069.stdout: 2024-04-14T13:50:25.197 INFO:journalctl@ceph.mon.a.smithi069.stdout: ** Compaction Stats [default] ** 2024-04-14T13:50:25.197 INFO:journalctl@ceph.mon.a.smithi069.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-04-14T13:50:25.197 INFO:journalctl@ceph.mon.a.smithi069.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-04-14T13:50:25.197 INFO:journalctl@ceph.mon.a.smithi069.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 162.4 0.00 0.00 1 0.000 0 0 2024-04-14T13:50:25.197 INFO:journalctl@ceph.mon.a.smithi069.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-04-14T13:50:25.197 INFO:journalctl@ceph.mon.a.smithi069.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-04-14T13:50:25.197 INFO:journalctl@ceph.mon.a.smithi069.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-04-14T13:50:25.197 INFO:journalctl@ceph.mon.a.smithi069.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-04-14T13:50:25.197 INFO:journalctl@ceph.mon.a.smithi069.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-04-14T13:50:25.197 INFO:journalctl@ceph.mon.a.smithi069.stdout: AddFile(Keys): cumulative 0, interval 0 2024-04-14T13:50:25.197 INFO:journalctl@ceph.mon.a.smithi069.stdout: Cumulative compaction: 0.00 GB write, 25.50 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-04-14T13:50:25.197 INFO:journalctl@ceph.mon.a.smithi069.stdout: Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-04-14T13:50:25.198 INFO:journalctl@ceph.mon.a.smithi069.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-04-14T13:50:25.198 INFO:journalctl@ceph.mon.a.smithi069.stdout: 2024-04-14T13:50:25.198 INFO:journalctl@ceph.mon.a.smithi069.stdout: ** File Read Latency Histogram By Level [default] ** 2024-04-14T13:50:25.198 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:50:25 smithi069 ceph-mon[24818]: starting mon.a rank 0 at public addrs [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] at bind addrs [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon_data /var/lib/ceph/mon/ceph-a fsid dc326fc2-fa65-11ee-bc8e-c7b262605968 2024-04-14T13:50:25.198 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:50:25 smithi069 ceph-mon[24818]: mon.a@-1(???) e1 preinit fsid dc326fc2-fa65-11ee-bc8e-c7b262605968 2024-04-14T13:50:25.198 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:50:25 smithi069 ceph-mon[24818]: mon.a@-1(???).mds e1 new map 2024-04-14T13:50:25.198 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:50:25 smithi069 ceph-mon[24818]: mon.a@-1(???).mds e1 print_map 2024-04-14T13:50:25.198 INFO:journalctl@ceph.mon.a.smithi069.stdout: e1 2024-04-14T13:50:25.198 INFO:journalctl@ceph.mon.a.smithi069.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-04-14T13:50:25.198 INFO:journalctl@ceph.mon.a.smithi069.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-14T13:50:25.198 INFO:journalctl@ceph.mon.a.smithi069.stdout: legacy client fscid: -1 2024-04-14T13:50:25.198 INFO:journalctl@ceph.mon.a.smithi069.stdout: 2024-04-14T13:50:25.198 INFO:journalctl@ceph.mon.a.smithi069.stdout: No filesystems configured 2024-04-14T13:50:25.198 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:50:25 smithi069 ceph-mon[24818]: mon.a@-1(???).osd e1 crush map has features 3314932999778484224, adjusting msgr requires 2024-04-14T13:50:25.199 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:50:25 smithi069 ceph-mon[24818]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-04-14T13:50:25.199 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:50:25 smithi069 ceph-mon[24818]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-04-14T13:50:25.199 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:50:25 smithi069 ceph-mon[24818]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-04-14T13:50:25.199 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:50:25 smithi069 ceph-mon[24818]: mon.a@-1(???).paxosservice(auth 1..2) refresh upgraded, format 0 -> 3 2024-04-14T13:50:25.199 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:50:25 smithi069 bash[24800]: d3c553f919132d9c83653799e05d39a186cec421066a3f156339d49d1a0966b4 2024-04-14T13:50:25.199 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:50:25 smithi069 systemd[1]: Started Ceph mon.a for dc326fc2-fa65-11ee-bc8e-c7b262605968. 2024-04-14T13:50:25.199 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:50:25 smithi069 ceph-mon[24818]: mon.a is new leader, mons a in quorum (ranks 0) 2024-04-14T13:50:25.199 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:50:25 smithi069 ceph-mon[24818]: monmap e1: 1 mons at {a=[v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0]} 2024-04-14T13:50:25.199 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:50:25 smithi069 ceph-mon[24818]: fsmap 2024-04-14T13:50:25.199 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:50:25 smithi069 ceph-mon[24818]: osdmap e1: 0 total, 0 up, 0 in 2024-04-14T13:50:25.199 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:50:25 smithi069 ceph-mon[24818]: mgrmap e1: no daemons active 2024-04-14T13:50:26.101 INFO:teuthology.orchestra.run.smithi069.stderr:Wrote config to /etc/ceph/ceph.conf 2024-04-14T13:50:26.102 INFO:teuthology.orchestra.run.smithi069.stderr:Wrote keyring to /etc/ceph/ceph.client.admin.keyring 2024-04-14T13:50:26.102 INFO:teuthology.orchestra.run.smithi069.stderr:Creating mgr... 2024-04-14T13:50:26.102 INFO:teuthology.orchestra.run.smithi069.stderr:Verifying port 9283 ... 2024-04-14T13:50:26.272 INFO:teuthology.orchestra.run.smithi069.stderr:systemctl: Failed to reset failed state of unit ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968@mgr.y.service: Unit ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968@mgr.y.service not loaded. 2024-04-14T13:50:26.279 INFO:teuthology.orchestra.run.smithi069.stderr:systemctl: Created symlink /etc/systemd/system/ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968.target.wants/ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968@mgr.y.service → /etc/systemd/system/ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968@.service. 2024-04-14T13:50:26.568 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 14 13:50:26 smithi069 systemd[1]: Starting Ceph mgr.y for dc326fc2-fa65-11ee-bc8e-c7b262605968... 2024-04-14T13:50:27.000 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:50:26 smithi069 ceph-mon[24818]: from='client.? 172.21.15.69:0/1424749570' entity='client.admin' 2024-04-14T13:50:27.001 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 14 13:50:26 smithi069 podman[25039]: 2024-04-14 13:50:26.655238001 +0000 UTC m=+0.111122021 container create 52e1363a8f98693029d56b4c0560dff745fb1b1bb8ed2d0f2dec9a4a49e3d02e (image=quay.io/ceph/ceph:v17.2.0, name=ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-y, name=centos-stream, io.k8s.display-name=CentOS Stream 8, release=754, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, CEPH_POINT_RELEASE=-17.2.0, GIT_REPO=https://github.com/ceph/ceph-container.git, GIT_BRANCH=HEAD, build-date=2022-05-03T08:36:31.336870, GIT_CLEAN=True, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, architecture=x86_64, com.redhat.component=centos-stream-container, io.openshift.expose-services=, io.buildah.version=1.19.8, version=8, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., distribution-scope=public, vcs-type=git, RELEASE=HEAD, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, io.openshift.tags=base centos centos-stream, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, vendor=Red Hat, Inc., description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, maintainer=Guillaume Abrioux , GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, ceph=True) 2024-04-14T13:50:27.001 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 14 13:50:26 smithi069 podman[25039]: 2024-04-14 13:50:26.561985279 +0000 UTC m=+0.017869326 image pull e1d6a67b021eb077ee22bf650f1a9fb1980a2cf5c36bdb9cba9eac6de8f702d9 quay.io/ceph/ceph:v17.2.0 2024-04-14T13:50:28.068 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 14 13:50:27 smithi069 podman[25039]: 2024-04-14 13:50:27.74820603 +0000 UTC m=+1.204090049 container init 52e1363a8f98693029d56b4c0560dff745fb1b1bb8ed2d0f2dec9a4a49e3d02e (image=quay.io/ceph/ceph:v17.2.0, name=ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-y, GIT_CLEAN=True, CEPH_POINT_RELEASE=-17.2.0, RELEASE=HEAD, io.openshift.expose-services=, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, build-date=2022-05-03T08:36:31.336870, vcs-type=git, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, ceph=True, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, io.buildah.version=1.19.8, io.openshift.tags=base centos centos-stream, version=8, architecture=x86_64, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, io.k8s.display-name=CentOS Stream 8, com.redhat.component=centos-stream-container, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., name=centos-stream, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, maintainer=Guillaume Abrioux , GIT_REPO=https://github.com/ceph/ceph-container.git, GIT_BRANCH=HEAD, distribution-scope=public, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, release=754, vendor=Red Hat, Inc.) 2024-04-14T13:50:28.068 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 14 13:50:27 smithi069 podman[25039]: 2024-04-14 13:50:27.751545039 +0000 UTC m=+1.207429060 container start 52e1363a8f98693029d56b4c0560dff745fb1b1bb8ed2d0f2dec9a4a49e3d02e (image=quay.io/ceph/ceph:v17.2.0, name=ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-y, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, distribution-scope=public, GIT_BRANCH=HEAD, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, io.openshift.tags=base centos centos-stream, GIT_REPO=https://github.com/ceph/ceph-container.git, io.k8s.display-name=CentOS Stream 8, release=754, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., vendor=Red Hat, Inc., description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, maintainer=Guillaume Abrioux , io.openshift.expose-services=, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, vcs-type=git, RELEASE=HEAD, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, GIT_CLEAN=True, io.buildah.version=1.19.8, name=centos-stream, build-date=2022-05-03T08:36:31.336870, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, CEPH_POINT_RELEASE=-17.2.0, version=8, ceph=True, architecture=x86_64, com.redhat.component=centos-stream-container, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac) 2024-04-14T13:50:28.523 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 14 13:50:28 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-y[25054]: 2024-04-14T13:50:28.244+0000 7fa115b43000 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-14T13:50:28.523 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 14 13:50:28 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-y[25054]: 2024-04-14T13:50:28.334+0000 7fa115b43000 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-14T13:50:28.536 INFO:teuthology.orchestra.run.smithi069.stderr:systemctl: disabled 2024-04-14T13:50:28.545 INFO:teuthology.orchestra.run.smithi069.stderr:systemctl: inactive 2024-04-14T13:50:28.546 INFO:teuthology.orchestra.run.smithi069.stderr:firewalld.service is not enabled 2024-04-14T13:50:28.546 INFO:teuthology.orchestra.run.smithi069.stderr:Not possible to enable service . firewalld.service is not available 2024-04-14T13:50:28.554 INFO:teuthology.orchestra.run.smithi069.stderr:systemctl: disabled 2024-04-14T13:50:28.561 INFO:teuthology.orchestra.run.smithi069.stderr:systemctl: inactive 2024-04-14T13:50:28.561 INFO:teuthology.orchestra.run.smithi069.stderr:firewalld.service is not enabled 2024-04-14T13:50:28.562 INFO:teuthology.orchestra.run.smithi069.stderr:Not possible to open ports <[9283]>. firewalld.service is not available 2024-04-14T13:50:28.562 INFO:teuthology.orchestra.run.smithi069.stderr:Waiting for mgr to start... 2024-04-14T13:50:28.562 INFO:teuthology.orchestra.run.smithi069.stderr:Waiting for mgr... 2024-04-14T13:50:28.817 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 14 13:50:28 smithi069 bash[25039]: 52e1363a8f98693029d56b4c0560dff745fb1b1bb8ed2d0f2dec9a4a49e3d02e 2024-04-14T13:50:28.817 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 14 13:50:28 smithi069 systemd[1]: Started Ceph mgr.y for dc326fc2-fa65-11ee-bc8e-c7b262605968. 2024-04-14T13:50:29.286 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 14 13:50:28 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-y[25054]: 2024-04-14T13:50:28.936+0000 7fa115b43000 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-14T13:50:29.286 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 14 13:50:29 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-y[25054]: 2024-04-14T13:50:29.024+0000 7fa115b43000 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-14T13:50:29.286 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 14 13:50:29 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-y[25054]: 2024-04-14T13:50:29.107+0000 7fa115b43000 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-14T13:50:29.567 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 14 13:50:29 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-y[25054]: 2024-04-14T13:50:29.253+0000 7fa115b43000 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-14T13:50:29.568 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 14 13:50:29 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-y[25054]: 2024-04-14T13:50:29.333+0000 7fa115b43000 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-14T13:50:29.568 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 14 13:50:29 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-y[25054]: 2024-04-14T13:50:29.439+0000 7fa115b43000 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-14T13:50:29.568 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 14 13:50:29 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-y[25054]: 2024-04-14T13:50:29.521+0000 7fa115b43000 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-14T13:50:30.067 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 14 13:50:29 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-y[25054]: 2024-04-14T13:50:29.681+0000 7fa115b43000 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-14T13:50:30.068 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 14 13:50:29 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-y[25054]: 2024-04-14T13:50:29.766+0000 7fa115b43000 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-14T13:50:30.924 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 14 13:50:30 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-y[25054]: 2024-04-14T13:50:30.794+0000 7fa115b43000 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-14T13:50:30.924 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 14 13:50:30 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-y[25054]: 2024-04-14T13:50:30.921+0000 7fa115b43000 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-14T13:50:31.205 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 14 13:50:31 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-y[25054]: 2024-04-14T13:50:31.107+0000 7fa115b43000 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-14T13:50:31.206 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 14 13:50:31 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-y[25054]: 2024-04-14T13:50:31.191+0000 7fa115b43000 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-14T13:50:31.567 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 14 13:50:31 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-y[25054]: 2024-04-14T13:50:31.324+0000 7fa115b43000 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-14T13:50:31.884 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:50:31 smithi069 ceph-mon[24818]: from='client.? 172.21.15.69:0/1478143617' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-04-14T13:50:31.885 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 14 13:50:31 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-y[25054]: 2024-04-14T13:50:31.638+0000 7fa115b43000 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-14T13:50:32.154 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 14 13:50:31 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-y[25054]: 2024-04-14T13:50:31.864+0000 7fa115b43000 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-14T13:50:32.154 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 14 13:50:31 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-y[25054]: 2024-04-14T13:50:31.948+0000 7fa115b43000 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-14T13:50:32.154 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 14 13:50:32 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-y[25054]: 2024-04-14T13:50:32.114+0000 7fa115b43000 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-14T13:50:32.567 INFO:teuthology.orchestra.run.smithi069.stderr:/usr/bin/ceph: 2024-04-14T13:50:32.567 INFO:teuthology.orchestra.run.smithi069.stderr:/usr/bin/ceph: { 2024-04-14T13:50:32.567 INFO:teuthology.orchestra.run.smithi069.stderr:/usr/bin/ceph: "fsid": "dc326fc2-fa65-11ee-bc8e-c7b262605968", 2024-04-14T13:50:32.567 INFO:teuthology.orchestra.run.smithi069.stderr:/usr/bin/ceph: "health": { 2024-04-14T13:50:32.567 INFO:teuthology.orchestra.run.smithi069.stderr:/usr/bin/ceph: "status": "HEALTH_OK", 2024-04-14T13:50:32.568 INFO:teuthology.orchestra.run.smithi069.stderr:/usr/bin/ceph: "checks": {}, 2024-04-14T13:50:32.568 INFO:teuthology.orchestra.run.smithi069.stderr:/usr/bin/ceph: "mutes": [] 2024-04-14T13:50:32.568 INFO:teuthology.orchestra.run.smithi069.stderr:/usr/bin/ceph: }, 2024-04-14T13:50:32.568 INFO:teuthology.orchestra.run.smithi069.stderr:/usr/bin/ceph: "election_epoch": 5, 2024-04-14T13:50:32.568 INFO:teuthology.orchestra.run.smithi069.stderr:/usr/bin/ceph: "quorum": [ 2024-04-14T13:50:32.568 INFO:teuthology.orchestra.run.smithi069.stderr:/usr/bin/ceph: 0 2024-04-14T13:50:32.568 INFO:teuthology.orchestra.run.smithi069.stderr:/usr/bin/ceph: ], 2024-04-14T13:50:32.568 INFO:teuthology.orchestra.run.smithi069.stderr:/usr/bin/ceph: "quorum_names": [ 2024-04-14T13:50:32.568 INFO:teuthology.orchestra.run.smithi069.stderr:/usr/bin/ceph: "a" 2024-04-14T13:50:32.568 INFO:teuthology.orchestra.run.smithi069.stderr:/usr/bin/ceph: ], 2024-04-14T13:50:32.568 INFO:teuthology.orchestra.run.smithi069.stderr:/usr/bin/ceph: "quorum_age": 6, 2024-04-14T13:50:32.568 INFO:teuthology.orchestra.run.smithi069.stderr:/usr/bin/ceph: "monmap": { 2024-04-14T13:50:32.568 INFO:teuthology.orchestra.run.smithi069.stderr:/usr/bin/ceph: "epoch": 1, 2024-04-14T13:50:32.568 INFO:teuthology.orchestra.run.smithi069.stderr:/usr/bin/ceph: "min_mon_release_name": "quincy", 2024-04-14T13:50:32.569 INFO:teuthology.orchestra.run.smithi069.stderr:/usr/bin/ceph: "num_mons": 1 2024-04-14T13:50:32.569 INFO:teuthology.orchestra.run.smithi069.stderr:/usr/bin/ceph: }, 2024-04-14T13:50:32.569 INFO:teuthology.orchestra.run.smithi069.stderr:/usr/bin/ceph: "osdmap": { 2024-04-14T13:50:32.569 INFO:teuthology.orchestra.run.smithi069.stderr:/usr/bin/ceph: "epoch": 1, 2024-04-14T13:50:32.569 INFO:teuthology.orchestra.run.smithi069.stderr:/usr/bin/ceph: "num_osds": 0, 2024-04-14T13:50:32.569 INFO:teuthology.orchestra.run.smithi069.stderr:/usr/bin/ceph: "num_up_osds": 0, 2024-04-14T13:50:32.569 INFO:teuthology.orchestra.run.smithi069.stderr:/usr/bin/ceph: "osd_up_since": 0, 2024-04-14T13:50:32.569 INFO:teuthology.orchestra.run.smithi069.stderr:/usr/bin/ceph: "num_in_osds": 0, 2024-04-14T13:50:32.569 INFO:teuthology.orchestra.run.smithi069.stderr:/usr/bin/ceph: "osd_in_since": 0, 2024-04-14T13:50:32.569 INFO:teuthology.orchestra.run.smithi069.stderr:/usr/bin/ceph: "num_remapped_pgs": 0 2024-04-14T13:50:32.569 INFO:teuthology.orchestra.run.smithi069.stderr:/usr/bin/ceph: }, 2024-04-14T13:50:32.569 INFO:teuthology.orchestra.run.smithi069.stderr:/usr/bin/ceph: "pgmap": { 2024-04-14T13:50:32.569 INFO:teuthology.orchestra.run.smithi069.stderr:/usr/bin/ceph: "pgs_by_state": [], 2024-04-14T13:50:32.569 INFO:teuthology.orchestra.run.smithi069.stderr:/usr/bin/ceph: "num_pgs": 0, 2024-04-14T13:50:32.569 INFO:teuthology.orchestra.run.smithi069.stderr:/usr/bin/ceph: "num_pools": 0, 2024-04-14T13:50:32.569 INFO:teuthology.orchestra.run.smithi069.stderr:/usr/bin/ceph: "num_objects": 0, 2024-04-14T13:50:32.570 INFO:teuthology.orchestra.run.smithi069.stderr:/usr/bin/ceph: "data_bytes": 0, 2024-04-14T13:50:32.570 INFO:teuthology.orchestra.run.smithi069.stderr:/usr/bin/ceph: "bytes_used": 0, 2024-04-14T13:50:32.570 INFO:teuthology.orchestra.run.smithi069.stderr:/usr/bin/ceph: "bytes_avail": 0, 2024-04-14T13:50:32.570 INFO:teuthology.orchestra.run.smithi069.stderr:/usr/bin/ceph: "bytes_total": 0 2024-04-14T13:50:32.570 INFO:teuthology.orchestra.run.smithi069.stderr:/usr/bin/ceph: }, 2024-04-14T13:50:32.570 INFO:teuthology.orchestra.run.smithi069.stderr:/usr/bin/ceph: "fsmap": { 2024-04-14T13:50:32.570 INFO:teuthology.orchestra.run.smithi069.stderr:/usr/bin/ceph: "epoch": 1, 2024-04-14T13:50:32.570 INFO:teuthology.orchestra.run.smithi069.stderr:/usr/bin/ceph: "by_rank": [], 2024-04-14T13:50:32.570 INFO:teuthology.orchestra.run.smithi069.stderr:/usr/bin/ceph: "up:standby": 0 2024-04-14T13:50:32.570 INFO:teuthology.orchestra.run.smithi069.stderr:/usr/bin/ceph: }, 2024-04-14T13:50:32.570 INFO:teuthology.orchestra.run.smithi069.stderr:/usr/bin/ceph: "mgrmap": { 2024-04-14T13:50:32.570 INFO:teuthology.orchestra.run.smithi069.stderr:/usr/bin/ceph: "available": false, 2024-04-14T13:50:32.570 INFO:teuthology.orchestra.run.smithi069.stderr:/usr/bin/ceph: "num_standbys": 0, 2024-04-14T13:50:32.570 INFO:teuthology.orchestra.run.smithi069.stderr:/usr/bin/ceph: "modules": [ 2024-04-14T13:50:32.570 INFO:teuthology.orchestra.run.smithi069.stderr:/usr/bin/ceph: "iostat", 2024-04-14T13:50:32.571 INFO:teuthology.orchestra.run.smithi069.stderr:/usr/bin/ceph: "nfs", 2024-04-14T13:50:32.571 INFO:teuthology.orchestra.run.smithi069.stderr:/usr/bin/ceph: "restful" 2024-04-14T13:50:32.571 INFO:teuthology.orchestra.run.smithi069.stderr:/usr/bin/ceph: ], 2024-04-14T13:50:32.571 INFO:teuthology.orchestra.run.smithi069.stderr:/usr/bin/ceph: "services": {} 2024-04-14T13:50:32.571 INFO:teuthology.orchestra.run.smithi069.stderr:/usr/bin/ceph: }, 2024-04-14T13:50:32.571 INFO:teuthology.orchestra.run.smithi069.stderr:/usr/bin/ceph: "servicemap": { 2024-04-14T13:50:32.571 INFO:teuthology.orchestra.run.smithi069.stderr:/usr/bin/ceph: "epoch": 1, 2024-04-14T13:50:32.571 INFO:teuthology.orchestra.run.smithi069.stderr:/usr/bin/ceph: "modified": "2024-04-14T13:50:21.025195+0000", 2024-04-14T13:50:32.571 INFO:teuthology.orchestra.run.smithi069.stderr:/usr/bin/ceph: "services": {} 2024-04-14T13:50:32.571 INFO:teuthology.orchestra.run.smithi069.stderr:/usr/bin/ceph: }, 2024-04-14T13:50:32.571 INFO:teuthology.orchestra.run.smithi069.stderr:/usr/bin/ceph: "progress_events": {} 2024-04-14T13:50:32.571 INFO:teuthology.orchestra.run.smithi069.stderr:/usr/bin/ceph: } 2024-04-14T13:50:33.316 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 14 13:50:33 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-y[25054]: 2024-04-14T13:50:33.020+0000 7fa115b43000 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-14T13:50:33.567 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 14 13:50:33 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-y[25054]: 2024-04-14T13:50:33.313+0000 7fa115b43000 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-14T13:50:33.568 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 14 13:50:33 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-y[25054]: 2024-04-14T13:50:33.407+0000 7fa115b43000 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-14T13:50:33.568 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 14 13:50:33 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-y[25054]: 2024-04-14T13:50:33.556+0000 7fa115b43000 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-14T13:50:34.068 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:50:33 smithi069 ceph-mon[24818]: Activating manager daemon y 2024-04-14T13:50:34.068 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:50:33 smithi069 ceph-mon[24818]: mgrmap e2: y(active, starting, since 0.0027925s) 2024-04-14T13:50:34.068 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:50:33 smithi069 ceph-mon[24818]: from='mgr.14100 172.21.15.69:0/1990457113' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-04-14T13:50:34.068 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:50:33 smithi069 ceph-mon[24818]: from='mgr.14100 172.21.15.69:0/1990457113' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-04-14T13:50:34.068 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:50:33 smithi069 ceph-mon[24818]: from='mgr.14100 172.21.15.69:0/1990457113' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-04-14T13:50:34.068 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:50:33 smithi069 ceph-mon[24818]: from='mgr.14100 172.21.15.69:0/1990457113' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-14T13:50:34.068 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:50:33 smithi069 ceph-mon[24818]: from='mgr.14100 172.21.15.69:0/1990457113' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-04-14T13:50:34.068 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:50:33 smithi069 ceph-mon[24818]: Manager daemon y is now available 2024-04-14T13:50:34.069 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:50:33 smithi069 ceph-mon[24818]: from='mgr.14100 172.21.15.69:0/1990457113' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-04-14T13:50:34.069 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:50:33 smithi069 ceph-mon[24818]: from='mgr.14100 172.21.15.69:0/1990457113' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-04-14T13:50:34.069 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:50:33 smithi069 ceph-mon[24818]: from='mgr.14100 172.21.15.69:0/1990457113' entity='mgr.y' 2024-04-14T13:50:34.069 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:50:33 smithi069 ceph-mon[24818]: from='mgr.14100 172.21.15.69:0/1990457113' entity='mgr.y' 2024-04-14T13:50:34.069 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:50:33 smithi069 ceph-mon[24818]: from='mgr.14100 172.21.15.69:0/1990457113' entity='mgr.y' 2024-04-14T13:50:35.818 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:50:35 smithi069 ceph-mon[24818]: mgrmap e3: y(active, since 1.00683s) 2024-04-14T13:50:37.567 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:50:37 smithi069 ceph-mon[24818]: mgrmap e4: y(active, since 2s) 2024-04-14T13:50:38.039 INFO:teuthology.orchestra.run.smithi069.stderr:mgr not available, waiting (1/15)... 2024-04-14T13:50:40.715 INFO:teuthology.orchestra.run.smithi069.stderr:/usr/bin/ceph: 2024-04-14T13:50:40.715 INFO:teuthology.orchestra.run.smithi069.stderr:/usr/bin/ceph: { 2024-04-14T13:50:40.715 INFO:teuthology.orchestra.run.smithi069.stderr:/usr/bin/ceph: "fsid": "dc326fc2-fa65-11ee-bc8e-c7b262605968", 2024-04-14T13:50:40.715 INFO:teuthology.orchestra.run.smithi069.stderr:/usr/bin/ceph: "health": { 2024-04-14T13:50:40.715 INFO:teuthology.orchestra.run.smithi069.stderr:/usr/bin/ceph: "status": "HEALTH_OK", 2024-04-14T13:50:40.715 INFO:teuthology.orchestra.run.smithi069.stderr:/usr/bin/ceph: "checks": {}, 2024-04-14T13:50:40.715 INFO:teuthology.orchestra.run.smithi069.stderr:/usr/bin/ceph: "mutes": [] 2024-04-14T13:50:40.715 INFO:teuthology.orchestra.run.smithi069.stderr:/usr/bin/ceph: }, 2024-04-14T13:50:40.715 INFO:teuthology.orchestra.run.smithi069.stderr:/usr/bin/ceph: "election_epoch": 5, 2024-04-14T13:50:40.715 INFO:teuthology.orchestra.run.smithi069.stderr:/usr/bin/ceph: "quorum": [ 2024-04-14T13:50:40.716 INFO:teuthology.orchestra.run.smithi069.stderr:/usr/bin/ceph: 0 2024-04-14T13:50:40.716 INFO:teuthology.orchestra.run.smithi069.stderr:/usr/bin/ceph: ], 2024-04-14T13:50:40.716 INFO:teuthology.orchestra.run.smithi069.stderr:/usr/bin/ceph: "quorum_names": [ 2024-04-14T13:50:40.716 INFO:teuthology.orchestra.run.smithi069.stderr:/usr/bin/ceph: "a" 2024-04-14T13:50:40.716 INFO:teuthology.orchestra.run.smithi069.stderr:/usr/bin/ceph: ], 2024-04-14T13:50:40.716 INFO:teuthology.orchestra.run.smithi069.stderr:/usr/bin/ceph: "quorum_age": 15, 2024-04-14T13:50:40.716 INFO:teuthology.orchestra.run.smithi069.stderr:/usr/bin/ceph: "monmap": { 2024-04-14T13:50:40.716 INFO:teuthology.orchestra.run.smithi069.stderr:/usr/bin/ceph: "epoch": 1, 2024-04-14T13:50:40.716 INFO:teuthology.orchestra.run.smithi069.stderr:/usr/bin/ceph: "min_mon_release_name": "quincy", 2024-04-14T13:50:40.716 INFO:teuthology.orchestra.run.smithi069.stderr:/usr/bin/ceph: "num_mons": 1 2024-04-14T13:50:40.716 INFO:teuthology.orchestra.run.smithi069.stderr:/usr/bin/ceph: }, 2024-04-14T13:50:40.716 INFO:teuthology.orchestra.run.smithi069.stderr:/usr/bin/ceph: "osdmap": { 2024-04-14T13:50:40.716 INFO:teuthology.orchestra.run.smithi069.stderr:/usr/bin/ceph: "epoch": 1, 2024-04-14T13:50:40.716 INFO:teuthology.orchestra.run.smithi069.stderr:/usr/bin/ceph: "num_osds": 0, 2024-04-14T13:50:40.716 INFO:teuthology.orchestra.run.smithi069.stderr:/usr/bin/ceph: "num_up_osds": 0, 2024-04-14T13:50:40.717 INFO:teuthology.orchestra.run.smithi069.stderr:/usr/bin/ceph: "osd_up_since": 0, 2024-04-14T13:50:40.720 INFO:teuthology.orchestra.run.smithi069.stderr:/usr/bin/ceph: "num_in_osds": 0, 2024-04-14T13:50:40.720 INFO:teuthology.orchestra.run.smithi069.stderr:/usr/bin/ceph: "osd_in_since": 0, 2024-04-14T13:50:40.720 INFO:teuthology.orchestra.run.smithi069.stderr:/usr/bin/ceph: "num_remapped_pgs": 0 2024-04-14T13:50:40.720 INFO:teuthology.orchestra.run.smithi069.stderr:/usr/bin/ceph: }, 2024-04-14T13:50:40.720 INFO:teuthology.orchestra.run.smithi069.stderr:/usr/bin/ceph: "pgmap": { 2024-04-14T13:50:40.720 INFO:teuthology.orchestra.run.smithi069.stderr:/usr/bin/ceph: "pgs_by_state": [], 2024-04-14T13:50:40.720 INFO:teuthology.orchestra.run.smithi069.stderr:/usr/bin/ceph: "num_pgs": 0, 2024-04-14T13:50:40.720 INFO:teuthology.orchestra.run.smithi069.stderr:/usr/bin/ceph: "num_pools": 0, 2024-04-14T13:50:40.720 INFO:teuthology.orchestra.run.smithi069.stderr:/usr/bin/ceph: "num_objects": 0, 2024-04-14T13:50:40.721 INFO:teuthology.orchestra.run.smithi069.stderr:/usr/bin/ceph: "data_bytes": 0, 2024-04-14T13:50:40.721 INFO:teuthology.orchestra.run.smithi069.stderr:/usr/bin/ceph: "bytes_used": 0, 2024-04-14T13:50:40.721 INFO:teuthology.orchestra.run.smithi069.stderr:/usr/bin/ceph: "bytes_avail": 0, 2024-04-14T13:50:40.721 INFO:teuthology.orchestra.run.smithi069.stderr:/usr/bin/ceph: "bytes_total": 0 2024-04-14T13:50:40.721 INFO:teuthology.orchestra.run.smithi069.stderr:/usr/bin/ceph: }, 2024-04-14T13:50:40.721 INFO:teuthology.orchestra.run.smithi069.stderr:/usr/bin/ceph: "fsmap": { 2024-04-14T13:50:40.721 INFO:teuthology.orchestra.run.smithi069.stderr:/usr/bin/ceph: "epoch": 1, 2024-04-14T13:50:40.721 INFO:teuthology.orchestra.run.smithi069.stderr:/usr/bin/ceph: "by_rank": [], 2024-04-14T13:50:40.721 INFO:teuthology.orchestra.run.smithi069.stderr:/usr/bin/ceph: "up:standby": 0 2024-04-14T13:50:40.721 INFO:teuthology.orchestra.run.smithi069.stderr:/usr/bin/ceph: }, 2024-04-14T13:50:40.721 INFO:teuthology.orchestra.run.smithi069.stderr:/usr/bin/ceph: "mgrmap": { 2024-04-14T13:50:40.721 INFO:teuthology.orchestra.run.smithi069.stderr:/usr/bin/ceph: "available": true, 2024-04-14T13:50:40.721 INFO:teuthology.orchestra.run.smithi069.stderr:/usr/bin/ceph: "num_standbys": 0, 2024-04-14T13:50:40.721 INFO:teuthology.orchestra.run.smithi069.stderr:/usr/bin/ceph: "modules": [ 2024-04-14T13:50:40.721 INFO:teuthology.orchestra.run.smithi069.stderr:/usr/bin/ceph: "iostat", 2024-04-14T13:50:40.721 INFO:teuthology.orchestra.run.smithi069.stderr:/usr/bin/ceph: "nfs", 2024-04-14T13:50:40.722 INFO:teuthology.orchestra.run.smithi069.stderr:/usr/bin/ceph: "restful" 2024-04-14T13:50:40.722 INFO:teuthology.orchestra.run.smithi069.stderr:/usr/bin/ceph: ], 2024-04-14T13:50:40.722 INFO:teuthology.orchestra.run.smithi069.stderr:/usr/bin/ceph: "services": {} 2024-04-14T13:50:40.722 INFO:teuthology.orchestra.run.smithi069.stderr:/usr/bin/ceph: }, 2024-04-14T13:50:40.722 INFO:teuthology.orchestra.run.smithi069.stderr:/usr/bin/ceph: "servicemap": { 2024-04-14T13:50:40.722 INFO:teuthology.orchestra.run.smithi069.stderr:/usr/bin/ceph: "epoch": 1, 2024-04-14T13:50:40.722 INFO:teuthology.orchestra.run.smithi069.stderr:/usr/bin/ceph: "modified": "2024-04-14T13:50:21.025195+0000", 2024-04-14T13:50:40.722 INFO:teuthology.orchestra.run.smithi069.stderr:/usr/bin/ceph: "services": {} 2024-04-14T13:50:40.722 INFO:teuthology.orchestra.run.smithi069.stderr:/usr/bin/ceph: }, 2024-04-14T13:50:40.722 INFO:teuthology.orchestra.run.smithi069.stderr:/usr/bin/ceph: "progress_events": {} 2024-04-14T13:50:40.722 INFO:teuthology.orchestra.run.smithi069.stderr:/usr/bin/ceph: } 2024-04-14T13:50:41.235 INFO:teuthology.orchestra.run.smithi069.stderr:mgr is available 2024-04-14T13:50:41.317 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:50:41 smithi069 ceph-mon[24818]: from='client.? 172.21.15.69:0/1165687134' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-04-14T13:50:41.804 INFO:teuthology.orchestra.run.smithi069.stderr:/usr/bin/ceph: 2024-04-14T13:50:41.804 INFO:teuthology.orchestra.run.smithi069.stderr:/usr/bin/ceph: [global] 2024-04-14T13:50:41.804 INFO:teuthology.orchestra.run.smithi069.stderr:/usr/bin/ceph: fsid = dc326fc2-fa65-11ee-bc8e-c7b262605968 2024-04-14T13:50:41.804 INFO:teuthology.orchestra.run.smithi069.stderr:/usr/bin/ceph: mon_osd_allow_pg_remap = true 2024-04-14T13:50:41.804 INFO:teuthology.orchestra.run.smithi069.stderr:/usr/bin/ceph: mon_osd_allow_primary_affinity = true 2024-04-14T13:50:41.804 INFO:teuthology.orchestra.run.smithi069.stderr:/usr/bin/ceph: mon_warn_on_no_sortbitwise = false 2024-04-14T13:50:41.804 INFO:teuthology.orchestra.run.smithi069.stderr:/usr/bin/ceph: osd_crush_chooseleaf_type = 0 2024-04-14T13:50:41.804 INFO:teuthology.orchestra.run.smithi069.stderr:/usr/bin/ceph: 2024-04-14T13:50:41.804 INFO:teuthology.orchestra.run.smithi069.stderr:/usr/bin/ceph: [mgr] 2024-04-14T13:50:41.804 INFO:teuthology.orchestra.run.smithi069.stderr:/usr/bin/ceph: mgr/telemetry/nag = false 2024-04-14T13:50:41.804 INFO:teuthology.orchestra.run.smithi069.stderr:/usr/bin/ceph: 2024-04-14T13:50:41.804 INFO:teuthology.orchestra.run.smithi069.stderr:/usr/bin/ceph: [osd] 2024-04-14T13:50:41.805 INFO:teuthology.orchestra.run.smithi069.stderr:/usr/bin/ceph: osd_map_max_advance = 10 2024-04-14T13:50:41.805 INFO:teuthology.orchestra.run.smithi069.stderr:/usr/bin/ceph: osd_sloppy_crc = true 2024-04-14T13:50:42.375 INFO:teuthology.orchestra.run.smithi069.stderr:Enabling cephadm module... 2024-04-14T13:50:42.497 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:50:42 smithi069 ceph-mon[24818]: from='client.? 172.21.15.69:0/1239533345' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-04-14T13:50:42.497 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:50:42 smithi069 ceph-mon[24818]: from='client.? 172.21.15.69:0/1239533345' entity='client.admin' cmd='[{"prefix": "config assimilate-conf"}]': finished 2024-04-14T13:50:43.767 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:50:43 smithi069 ceph-mon[24818]: from='client.? 172.21.15.69:0/3853089842' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "cephadm"}]: dispatch 2024-04-14T13:50:43.767 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 14 13:50:43 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-y[25054]: ignoring --setuser ceph since I am not root 2024-04-14T13:50:43.767 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 14 13:50:43 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-y[25054]: ignoring --setgroup ceph since I am not root 2024-04-14T13:50:44.067 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 14 13:50:43 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-y[25054]: 2024-04-14T13:50:43.976+0000 7fd5bff82000 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-14T13:50:44.324 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 14 13:50:44 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-y[25054]: 2024-04-14T13:50:44.067+0000 7fd5bff82000 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-14T13:50:44.689 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:50:44 smithi069 ceph-mon[24818]: from='client.? 172.21.15.69:0/3853089842' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "cephadm"}]': finished 2024-04-14T13:50:44.690 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:50:44 smithi069 ceph-mon[24818]: mgrmap e5: y(active, since 9s) 2024-04-14T13:50:44.767 INFO:teuthology.orchestra.run.smithi069.stderr:/usr/bin/ceph: { 2024-04-14T13:50:44.768 INFO:teuthology.orchestra.run.smithi069.stderr:/usr/bin/ceph: "epoch": 5, 2024-04-14T13:50:44.768 INFO:teuthology.orchestra.run.smithi069.stderr:/usr/bin/ceph: "available": true, 2024-04-14T13:50:44.768 INFO:teuthology.orchestra.run.smithi069.stderr:/usr/bin/ceph: "active_name": "y", 2024-04-14T13:50:44.768 INFO:teuthology.orchestra.run.smithi069.stderr:/usr/bin/ceph: "num_standby": 0 2024-04-14T13:50:44.768 INFO:teuthology.orchestra.run.smithi069.stderr:/usr/bin/ceph: } 2024-04-14T13:50:45.021 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 14 13:50:44 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-y[25054]: 2024-04-14T13:50:44.686+0000 7fd5bff82000 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-14T13:50:45.022 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 14 13:50:44 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-y[25054]: 2024-04-14T13:50:44.778+0000 7fd5bff82000 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-14T13:50:45.022 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 14 13:50:44 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-y[25054]: 2024-04-14T13:50:44.867+0000 7fd5bff82000 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-14T13:50:45.022 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 14 13:50:45 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-y[25054]: 2024-04-14T13:50:45.019+0000 7fd5bff82000 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-14T13:50:45.297 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 14 13:50:45 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-y[25054]: 2024-04-14T13:50:45.100+0000 7fd5bff82000 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-14T13:50:45.297 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 14 13:50:45 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-y[25054]: 2024-04-14T13:50:45.210+0000 7fd5bff82000 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-14T13:50:45.298 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 14 13:50:45 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-y[25054]: 2024-04-14T13:50:45.294+0000 7fd5bff82000 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-14T13:50:45.333 INFO:teuthology.orchestra.run.smithi069.stderr:Waiting for the mgr to restart... 2024-04-14T13:50:45.333 INFO:teuthology.orchestra.run.smithi069.stderr:Waiting for mgr epoch 5... 2024-04-14T13:50:45.552 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:50:45 smithi069 ceph-mon[24818]: from='client.? 172.21.15.69:0/3554215896' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-04-14T13:50:45.553 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 14 13:50:45 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-y[25054]: 2024-04-14T13:50:45.462+0000 7fd5bff82000 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-14T13:50:45.817 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 14 13:50:45 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-y[25054]: 2024-04-14T13:50:45.550+0000 7fd5bff82000 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-14T13:50:46.892 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 14 13:50:46 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-y[25054]: 2024-04-14T13:50:46.571+0000 7fd5bff82000 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-14T13:50:46.893 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 14 13:50:46 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-y[25054]: 2024-04-14T13:50:46.701+0000 7fd5bff82000 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-14T13:50:46.893 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 14 13:50:46 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-y[25054]: 2024-04-14T13:50:46.890+0000 7fd5bff82000 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-14T13:50:47.317 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 14 13:50:46 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-y[25054]: 2024-04-14T13:50:46.975+0000 7fd5bff82000 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-14T13:50:47.318 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 14 13:50:47 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-y[25054]: 2024-04-14T13:50:47.109+0000 7fd5bff82000 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-14T13:50:47.730 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 14 13:50:47 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-y[25054]: 2024-04-14T13:50:47.423+0000 7fd5bff82000 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-14T13:50:47.731 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 14 13:50:47 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-y[25054]: 2024-04-14T13:50:47.644+0000 7fd5bff82000 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-14T13:50:47.731 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 14 13:50:47 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-y[25054]: 2024-04-14T13:50:47.728+0000 7fd5bff82000 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-14T13:50:48.067 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 14 13:50:47 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-y[25054]: 2024-04-14T13:50:47.899+0000 7fd5bff82000 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-14T13:50:49.067 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 14 13:50:48 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-y[25054]: 2024-04-14T13:50:48.805+0000 7fd5bff82000 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-14T13:50:49.395 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 14 13:50:49 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-y[25054]: 2024-04-14T13:50:49.093+0000 7fd5bff82000 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-14T13:50:49.396 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 14 13:50:49 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-y[25054]: 2024-04-14T13:50:49.187+0000 7fd5bff82000 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-14T13:50:49.396 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 14 13:50:49 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-y[25054]: 2024-04-14T13:50:49.338+0000 7fd5bff82000 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-14T13:50:49.818 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:50:49 smithi069 ceph-mon[24818]: Active manager daemon y restarted 2024-04-14T13:50:49.818 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:50:49 smithi069 ceph-mon[24818]: Activating manager daemon y 2024-04-14T13:50:49.818 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:50:49 smithi069 ceph-mon[24818]: osdmap e2: 0 total, 0 up, 0 in 2024-04-14T13:50:50.408 INFO:teuthology.orchestra.run.smithi069.stderr:/usr/bin/ceph: { 2024-04-14T13:50:50.408 INFO:teuthology.orchestra.run.smithi069.stderr:/usr/bin/ceph: "mgrmap_epoch": 7, 2024-04-14T13:50:50.408 INFO:teuthology.orchestra.run.smithi069.stderr:/usr/bin/ceph: "initialized": true 2024-04-14T13:50:50.408 INFO:teuthology.orchestra.run.smithi069.stderr:/usr/bin/ceph: } 2024-04-14T13:50:50.763 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 14 13:50:50 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-y[25054]: [14/Apr/2024:13:50:50] ENGINE Bus STARTING 2024-04-14T13:50:50.763 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 14 13:50:50 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-y[25054]: [14/Apr/2024:13:50:50] ENGINE Serving on https://172.21.15.69:7150 2024-04-14T13:50:50.763 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 14 13:50:50 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-y[25054]: [14/Apr/2024:13:50:50] ENGINE Bus STARTED 2024-04-14T13:50:50.763 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:50:50 smithi069 ceph-mon[24818]: mgrmap e6: y(active, starting, since 0.0526963s) 2024-04-14T13:50:50.763 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:50:50 smithi069 ceph-mon[24818]: from='mgr.14116 172.21.15.69:0/2877187273' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-14T13:50:50.763 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:50:50 smithi069 ceph-mon[24818]: from='mgr.14116 172.21.15.69:0/2877187273' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-04-14T13:50:50.763 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:50:50 smithi069 ceph-mon[24818]: from='mgr.14116 172.21.15.69:0/2877187273' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-04-14T13:50:50.763 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:50:50 smithi069 ceph-mon[24818]: from='mgr.14116 172.21.15.69:0/2877187273' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-04-14T13:50:50.763 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:50:50 smithi069 ceph-mon[24818]: from='mgr.14116 172.21.15.69:0/2877187273' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-04-14T13:50:50.763 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:50:50 smithi069 ceph-mon[24818]: Manager daemon y is now available 2024-04-14T13:50:50.764 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:50:50 smithi069 ceph-mon[24818]: from='mgr.14116 172.21.15.69:0/2877187273' entity='mgr.y' 2024-04-14T13:50:50.764 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:50:50 smithi069 ceph-mon[24818]: from='mgr.14116 172.21.15.69:0/2877187273' entity='mgr.y' 2024-04-14T13:50:50.764 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:50:50 smithi069 ceph-mon[24818]: from='mgr.14116 172.21.15.69:0/2877187273' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-14T13:50:50.764 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:50:50 smithi069 ceph-mon[24818]: from='mgr.14116 172.21.15.69:0/2877187273' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-14T13:50:50.764 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:50:50 smithi069 ceph-mon[24818]: from='mgr.14116 172.21.15.69:0/2877187273' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-14T13:50:50.764 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:50:50 smithi069 ceph-mon[24818]: from='mgr.14116 172.21.15.69:0/2877187273' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-04-14T13:50:50.764 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:50:50 smithi069 ceph-mon[24818]: from='mgr.14116 172.21.15.69:0/2877187273' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-04-14T13:50:50.999 INFO:teuthology.orchestra.run.smithi069.stderr:mgr epoch 5 is available 2024-04-14T13:50:50.999 INFO:teuthology.orchestra.run.smithi069.stderr:Setting orchestrator backend to cephadm... 2024-04-14T13:50:51.667 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:50:51 smithi069 ceph-mon[24818]: [14/Apr/2024:13:50:50] ENGINE Bus STARTING 2024-04-14T13:50:51.667 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:50:51 smithi069 ceph-mon[24818]: from='client.14120 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-04-14T13:50:51.667 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:50:51 smithi069 ceph-mon[24818]: mgrmap e7: y(active, since 1.05939s) 2024-04-14T13:50:51.668 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:50:51 smithi069 ceph-mon[24818]: from='client.14120 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-04-14T13:50:51.668 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:50:51 smithi069 ceph-mon[24818]: [14/Apr/2024:13:50:50] ENGINE Serving on https://172.21.15.69:7150 2024-04-14T13:50:51.668 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:50:51 smithi069 ceph-mon[24818]: [14/Apr/2024:13:50:50] ENGINE Bus STARTED 2024-04-14T13:50:51.668 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:50:51 smithi069 ceph-mon[24818]: from='mgr.14116 172.21.15.69:0/2877187273' entity='mgr.y' 2024-04-14T13:50:51.668 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:50:51 smithi069 ceph-mon[24818]: from='mgr.14116 172.21.15.69:0/2877187273' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-14T13:50:52.689 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:50:52 smithi069 ceph-mon[24818]: mgrmap e8: y(active, since 2s) 2024-04-14T13:50:52.689 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:50:52 smithi069 ceph-mon[24818]: from='mgr.14116 172.21.15.69:0/2877187273' entity='mgr.y' 2024-04-14T13:50:52.689 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:50:52 smithi069 ceph-mon[24818]: from='mgr.14116 172.21.15.69:0/2877187273' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-14T13:50:52.689 INFO:teuthology.orchestra.run.smithi069.stderr:/usr/bin/ceph: value unchanged 2024-04-14T13:50:53.238 INFO:teuthology.orchestra.run.smithi069.stderr:Generating ssh key... 2024-04-14T13:50:53.612 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:50:53 smithi069 ceph-mon[24818]: from='client.14128 -' entity='client.admin' cmd=[{"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T13:50:54.318 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 14 13:50:54 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-y[25054]: Generating public/private rsa key pair. 2024-04-14T13:50:54.318 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 14 13:50:54 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-y[25054]: Your identification has been saved in /tmp/tmppng79dhh/key. 2024-04-14T13:50:54.318 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 14 13:50:54 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-y[25054]: Your public key has been saved in /tmp/tmppng79dhh/key.pub. 2024-04-14T13:50:54.318 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 14 13:50:54 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-y[25054]: The key fingerprint is: 2024-04-14T13:50:54.318 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 14 13:50:54 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-y[25054]: SHA256:p6l9o1dm0DVeo5jTCyIGyFWoGgL3Rg4umir+I0hUxcg ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968 2024-04-14T13:50:54.318 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 14 13:50:54 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-y[25054]: The key's randomart image is: 2024-04-14T13:50:54.318 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 14 13:50:54 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-y[25054]: +---[RSA 3072]----+ 2024-04-14T13:50:54.319 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 14 13:50:54 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-y[25054]: | o *oo. | 2024-04-14T13:50:54.319 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 14 13:50:54 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-y[25054]: |. oE.+ o..| 2024-04-14T13:50:54.319 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 14 13:50:54 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-y[25054]: |.o.=. . .+o.o.| 2024-04-14T13:50:54.319 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 14 13:50:54 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-y[25054]: |oo..+ o ..=.o. | 2024-04-14T13:50:54.319 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 14 13:50:54 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-y[25054]: |+oo. . S o.o . | 2024-04-14T13:50:54.319 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 14 13:50:54 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-y[25054]: |oo + +. | 2024-04-14T13:50:54.319 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 14 13:50:54 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-y[25054]: |+ o + | 2024-04-14T13:50:54.319 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 14 13:50:54 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-y[25054]: |+. . o + | 2024-04-14T13:50:54.319 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 14 13:50:54 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-y[25054]: |o.o.. . o+ . | 2024-04-14T13:50:54.319 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 14 13:50:54 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-y[25054]: +----[SHA256]-----+ 2024-04-14T13:50:54.659 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:50:54 smithi069 ceph-mon[24818]: from='client.14130 -' entity='client.admin' cmd=[{"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T13:50:54.659 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:50:54 smithi069 ceph-mon[24818]: from='mgr.14116 172.21.15.69:0/2877187273' entity='mgr.y' 2024-04-14T13:50:54.659 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:50:54 smithi069 ceph-mon[24818]: from='mgr.14116 172.21.15.69:0/2877187273' entity='mgr.y' 2024-04-14T13:50:55.107 INFO:teuthology.orchestra.run.smithi069.stderr:/usr/bin/ceph: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDBK4dYBdiNlF+sVmpUkmPiS8laU27/fc/CLMVJi20MPmnG51NsqpwKpfRdXdeq67Ls9GJLNG09UmjcRE49TGr3JmZtwu0DKTr6xEoDHsdd6wG0yIx21+cPJunvY0NMftr3XDwzgCJKSEugwasfg0IjTo4lrZ65gKBbGB30ZWA+/yBWZ/FgS/wPXmQimiyUVK3FQ6ZRvx0klpVrHpf55NhQnsP1ix6V8ikStd7lPh/O/Ho88plhrXmSBhwfAroZF9sAaxdCuQLybfsuYQpD0ETQpU+RPqTD3hay70aW82HjkYUZfSVe7AxQSVXBJzjucCCuM8yzwweM2rrrArWu4Jp8qox8BPhExgVAVX3172hX8FZHzOU7h4SaXIy61Wg24kivUnM3hieF8/9XD5KOKRK+a3nE0qOng0qdYMZuVQMVQZui/Tw/mQE1RbFMsITF5EHUPW1wM7Upe6UGzca8DNM/bE4VwqlBeytma7SvQzOFLyiqN3cZL2E9LXNWoujX8/0= ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968 2024-04-14T13:50:55.689 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:50:55 smithi069 ceph-mon[24818]: from='client.14132 -' entity='client.admin' cmd=[{"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T13:50:55.689 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:50:55 smithi069 ceph-mon[24818]: Generating ssh key... 2024-04-14T13:50:55.692 INFO:teuthology.orchestra.run.smithi069.stderr:Wrote public SSH key to /home/ubuntu/cephtest/ceph.pub 2024-04-14T13:50:55.692 INFO:teuthology.orchestra.run.smithi069.stderr:Adding key to root@localhost authorized_keys... 2024-04-14T13:50:55.692 INFO:teuthology.orchestra.run.smithi069.stderr:Adding host smithi069... 2024-04-14T13:50:56.567 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:50:56 smithi069 ceph-mon[24818]: from='client.14134 -' entity='client.admin' cmd=[{"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T13:50:57.277 INFO:teuthology.orchestra.run.smithi069.stderr:/usr/bin/ceph: Added host 'smithi069' with addr '172.21.15.69' 2024-04-14T13:50:57.563 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:50:57 smithi069 ceph-mon[24818]: from='client.14136 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi069", "addr": "172.21.15.69", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T13:50:57.563 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:50:57 smithi069 ceph-mon[24818]: from='mgr.14116 172.21.15.69:0/2877187273' entity='mgr.y' 2024-04-14T13:50:57.563 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:50:57 smithi069 ceph-mon[24818]: from='mgr.14116 172.21.15.69:0/2877187273' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-14T13:50:57.815 INFO:teuthology.orchestra.run.smithi069.stderr:Deploying unmanaged mon service... 2024-04-14T13:50:58.428 INFO:teuthology.orchestra.run.smithi069.stderr:/usr/bin/ceph: Scheduled mon update... 2024-04-14T13:50:58.735 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:50:58 smithi069 ceph-mon[24818]: Deploying cephadm binary to smithi069 2024-04-14T13:50:58.735 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:50:58 smithi069 ceph-mon[24818]: Added host smithi069 2024-04-14T13:50:58.735 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:50:58 smithi069 ceph-mon[24818]: from='mgr.14116 172.21.15.69:0/2877187273' entity='mgr.y' 2024-04-14T13:50:59.207 INFO:teuthology.orchestra.run.smithi069.stderr:Deploying unmanaged mgr service... 2024-04-14T13:50:59.515 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:50:59 smithi069 ceph-mon[24818]: from='client.14138 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "unmanaged": true, "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T13:50:59.515 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:50:59 smithi069 ceph-mon[24818]: Saving service mon spec with placement count:5 2024-04-14T13:50:59.940 INFO:teuthology.orchestra.run.smithi069.stderr:/usr/bin/ceph: Scheduled mgr update... 2024-04-14T13:51:01.166 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:51:00 smithi069 ceph-mon[24818]: from='client.14140 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "unmanaged": true, "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T13:51:01.167 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:51:00 smithi069 ceph-mon[24818]: Saving service mgr spec with placement count:2 2024-04-14T13:51:01.167 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:51:00 smithi069 ceph-mon[24818]: from='mgr.14116 172.21.15.69:0/2877187273' entity='mgr.y' 2024-04-14T13:51:02.299 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:51:02 smithi069 ceph-mon[24818]: from='mgr.14116 172.21.15.69:0/2877187273' entity='mgr.y' 2024-04-14T13:51:02.299 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:51:02 smithi069 ceph-mon[24818]: from='mgr.14116 172.21.15.69:0/2877187273' entity='mgr.y' 2024-04-14T13:51:02.299 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:51:02 smithi069 ceph-mon[24818]: from='client.? 172.21.15.69:0/2319438207' entity='client.admin' 2024-04-14T13:51:03.843 INFO:teuthology.orchestra.run.smithi069.stderr:Enabling the dashboard module... 2024-04-14T13:51:04.199 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:51:03 smithi069 ceph-mon[24818]: from='client.? 172.21.15.69:0/3078823788' entity='client.admin' 2024-04-14T13:51:04.199 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:51:03 smithi069 ceph-mon[24818]: from='mgr.14116 172.21.15.69:0/2877187273' entity='mgr.y' 2024-04-14T13:51:05.739 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:51:05 smithi069 ceph-mon[24818]: from='mgr.14116 172.21.15.69:0/2877187273' entity='mgr.y' 2024-04-14T13:51:05.739 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:51:05 smithi069 ceph-mon[24818]: from='client.? 172.21.15.69:0/498251550' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "dashboard"}]: dispatch 2024-04-14T13:51:05.739 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 14 13:51:05 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-y[25054]: ignoring --setuser ceph since I am not root 2024-04-14T13:51:05.739 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 14 13:51:05 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-y[25054]: ignoring --setgroup ceph since I am not root 2024-04-14T13:51:06.037 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 14 13:51:05 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-y[25054]: 2024-04-14T13:51:05.889+0000 7f08ef6ca000 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-14T13:51:06.037 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 14 13:51:05 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-y[25054]: 2024-04-14T13:51:05.979+0000 7f08ef6ca000 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-14T13:51:06.625 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:51:06 smithi069 ceph-mon[24818]: from='client.? 172.21.15.69:0/498251550' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "dashboard"}]': finished 2024-04-14T13:51:06.625 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:51:06 smithi069 ceph-mon[24818]: mgrmap e9: y(active, since 16s) 2024-04-14T13:51:06.625 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 14 13:51:06 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-y[25054]: 2024-04-14T13:51:06.589+0000 7f08ef6ca000 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-14T13:51:06.943 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 14 13:51:06 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-y[25054]: 2024-04-14T13:51:06.681+0000 7f08ef6ca000 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-14T13:51:06.943 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 14 13:51:06 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-y[25054]: 2024-04-14T13:51:06.781+0000 7f08ef6ca000 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-14T13:51:06.943 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 14 13:51:06 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-y[25054]: 2024-04-14T13:51:06.940+0000 7f08ef6ca000 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-14T13:51:07.090 INFO:teuthology.orchestra.run.smithi069.stderr:/usr/bin/ceph: { 2024-04-14T13:51:07.090 INFO:teuthology.orchestra.run.smithi069.stderr:/usr/bin/ceph: "epoch": 9, 2024-04-14T13:51:07.090 INFO:teuthology.orchestra.run.smithi069.stderr:/usr/bin/ceph: "available": true, 2024-04-14T13:51:07.090 INFO:teuthology.orchestra.run.smithi069.stderr:/usr/bin/ceph: "active_name": "y", 2024-04-14T13:51:07.090 INFO:teuthology.orchestra.run.smithi069.stderr:/usr/bin/ceph: "num_standby": 0 2024-04-14T13:51:07.090 INFO:teuthology.orchestra.run.smithi069.stderr:/usr/bin/ceph: } 2024-04-14T13:51:07.221 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 14 13:51:07 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-y[25054]: 2024-04-14T13:51:07.021+0000 7f08ef6ca000 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-14T13:51:07.222 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 14 13:51:07 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-y[25054]: 2024-04-14T13:51:07.134+0000 7f08ef6ca000 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-14T13:51:07.222 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 14 13:51:07 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-y[25054]: 2024-04-14T13:51:07.219+0000 7f08ef6ca000 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-14T13:51:07.476 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:51:07 smithi069 ceph-mon[24818]: from='client.? 172.21.15.69:0/1139134670' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-04-14T13:51:07.477 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 14 13:51:07 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-y[25054]: 2024-04-14T13:51:07.386+0000 7f08ef6ca000 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-14T13:51:07.477 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 14 13:51:07 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-y[25054]: 2024-04-14T13:51:07.473+0000 7f08ef6ca000 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-14T13:51:07.697 INFO:teuthology.orchestra.run.smithi069.stderr:Waiting for the mgr to restart... 2024-04-14T13:51:07.697 INFO:teuthology.orchestra.run.smithi069.stderr:Waiting for mgr epoch 9... 2024-04-14T13:51:08.817 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 14 13:51:08 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-y[25054]: 2024-04-14T13:51:08.523+0000 7f08ef6ca000 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-14T13:51:08.818 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 14 13:51:08 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-y[25054]: 2024-04-14T13:51:08.654+0000 7f08ef6ca000 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-14T13:51:09.317 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 14 13:51:08 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-y[25054]: 2024-04-14T13:51:08.846+0000 7f08ef6ca000 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-14T13:51:09.317 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 14 13:51:08 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-y[25054]: 2024-04-14T13:51:08.933+0000 7f08ef6ca000 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-14T13:51:09.318 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 14 13:51:09 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-y[25054]: 2024-04-14T13:51:09.069+0000 7f08ef6ca000 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-14T13:51:09.720 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 14 13:51:09 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-y[25054]: 2024-04-14T13:51:09.408+0000 7f08ef6ca000 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-14T13:51:09.721 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 14 13:51:09 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-y[25054]: 2024-04-14T13:51:09.633+0000 7f08ef6ca000 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-14T13:51:09.721 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 14 13:51:09 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-y[25054]: 2024-04-14T13:51:09.718+0000 7f08ef6ca000 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-14T13:51:10.067 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 14 13:51:09 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-y[25054]: 2024-04-14T13:51:09.890+0000 7f08ef6ca000 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-14T13:51:11.143 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 14 13:51:10 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-y[25054]: 2024-04-14T13:51:10.838+0000 7f08ef6ca000 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-14T13:51:11.143 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 14 13:51:11 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-y[25054]: 2024-04-14T13:51:11.140+0000 7f08ef6ca000 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-14T13:51:11.393 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 14 13:51:11 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-y[25054]: 2024-04-14T13:51:11.238+0000 7f08ef6ca000 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-14T13:51:11.393 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 14 13:51:11 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-y[25054]: 2024-04-14T13:51:11.390+0000 7f08ef6ca000 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-14T13:51:12.068 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 14 13:51:11 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-y[25054]: [14/Apr/2024:13:51:11] ENGINE Bus STARTING 2024-04-14T13:51:12.068 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 14 13:51:11 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-y[25054]: [14/Apr/2024:13:51:11] ENGINE Serving on https://172.21.15.69:7150 2024-04-14T13:51:12.068 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 14 13:51:11 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-y[25054]: [14/Apr/2024:13:51:11] ENGINE Bus STARTED 2024-04-14T13:51:12.458 INFO:teuthology.orchestra.run.smithi069.stderr:/usr/bin/ceph: { 2024-04-14T13:51:12.459 INFO:teuthology.orchestra.run.smithi069.stderr:/usr/bin/ceph: "mgrmap_epoch": 11, 2024-04-14T13:51:12.459 INFO:teuthology.orchestra.run.smithi069.stderr:/usr/bin/ceph: "initialized": true 2024-04-14T13:51:12.459 INFO:teuthology.orchestra.run.smithi069.stderr:/usr/bin/ceph: } 2024-04-14T13:51:12.567 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:51:12 smithi069 ceph-mon[24818]: Active manager daemon y restarted 2024-04-14T13:51:12.567 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:51:12 smithi069 ceph-mon[24818]: Activating manager daemon y 2024-04-14T13:51:12.567 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:51:12 smithi069 ceph-mon[24818]: osdmap e3: 0 total, 0 up, 0 in 2024-04-14T13:51:12.568 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:51:12 smithi069 ceph-mon[24818]: mgrmap e10: y(active, starting, since 0.0547431s) 2024-04-14T13:51:12.568 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:51:12 smithi069 ceph-mon[24818]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-14T13:51:12.568 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:51:12 smithi069 ceph-mon[24818]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-04-14T13:51:12.568 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:51:12 smithi069 ceph-mon[24818]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-04-14T13:51:12.568 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:51:12 smithi069 ceph-mon[24818]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-04-14T13:51:12.568 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:51:12 smithi069 ceph-mon[24818]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-04-14T13:51:12.568 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:51:12 smithi069 ceph-mon[24818]: Manager daemon y is now available 2024-04-14T13:51:12.568 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:51:12 smithi069 ceph-mon[24818]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-14T13:51:12.568 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:51:12 smithi069 ceph-mon[24818]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-14T13:51:12.568 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:51:12 smithi069 ceph-mon[24818]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-04-14T13:51:12.568 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:51:12 smithi069 ceph-mon[24818]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-04-14T13:51:12.568 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:51:12 smithi069 ceph-mon[24818]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' 2024-04-14T13:51:13.315 INFO:teuthology.orchestra.run.smithi069.stderr:mgr epoch 9 is available 2024-04-14T13:51:13.315 INFO:teuthology.orchestra.run.smithi069.stderr:Generating a dashboard self-signed certificate... 2024-04-14T13:51:13.567 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:51:13 smithi069 ceph-mon[24818]: [14/Apr/2024:13:51:11] ENGINE Bus STARTING 2024-04-14T13:51:13.567 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:51:13 smithi069 ceph-mon[24818]: [14/Apr/2024:13:51:11] ENGINE Serving on https://172.21.15.69:7150 2024-04-14T13:51:13.567 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:51:13 smithi069 ceph-mon[24818]: [14/Apr/2024:13:51:11] ENGINE Bus STARTED 2024-04-14T13:51:13.568 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:51:13 smithi069 ceph-mon[24818]: mgrmap e11: y(active, since 1.05759s) 2024-04-14T13:51:13.568 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:51:13 smithi069 ceph-mon[24818]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' 2024-04-14T13:51:14.115 INFO:teuthology.orchestra.run.smithi069.stderr:/usr/bin/ceph: Self-signed certificate created 2024-04-14T13:51:14.797 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:51:14 smithi069 ceph-mon[24818]: from='client.14152 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-04-14T13:51:14.797 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:51:14 smithi069 ceph-mon[24818]: from='client.14152 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-04-14T13:51:14.797 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:51:14 smithi069 ceph-mon[24818]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' 2024-04-14T13:51:14.797 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:51:14 smithi069 ceph-mon[24818]: mgrmap e12: y(active, since 2s) 2024-04-14T13:51:14.797 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:51:14 smithi069 ceph-mon[24818]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' 2024-04-14T13:51:14.797 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:51:14 smithi069 ceph-mon[24818]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' 2024-04-14T13:51:14.798 INFO:teuthology.orchestra.run.smithi069.stderr:Creating initial admin user... 2024-04-14T13:51:15.606 INFO:teuthology.orchestra.run.smithi069.stderr:/usr/bin/ceph: {"username": "admin", "password": "$2b$12$U1UJjH6zoOopot9eEax1JOGZcc8.BkRT57kYPNtzANgBpd.ga/IdW", "roles": ["administrator"], "name": null, "email": null, "lastUpdate": 1713102675, "enabled": true, "pwdExpirationDate": null, "pwdUpdateRequired": true} 2024-04-14T13:51:15.817 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:51:15 smithi069 ceph-mon[24818]: from='client.14160 -' entity='client.admin' cmd=[{"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T13:51:16.241 INFO:teuthology.orchestra.run.smithi069.stderr:Fetching dashboard port number... 2024-04-14T13:51:16.567 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:51:16 smithi069 ceph-mon[24818]: from='client.14162 -' entity='client.admin' cmd=[{"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T13:51:16.567 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:51:16 smithi069 ceph-mon[24818]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' 2024-04-14T13:51:17.251 INFO:teuthology.orchestra.run.smithi069.stderr:/usr/bin/ceph: 8443 2024-04-14T13:51:17.567 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:51:17 smithi069 ceph-mon[24818]: from='client.? 172.21.15.69:0/2653884964' entity='client.admin' cmd=[{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]: dispatch 2024-04-14T13:51:17.826 INFO:teuthology.orchestra.run.smithi069.stderr:systemctl: disabled 2024-04-14T13:51:17.833 INFO:teuthology.orchestra.run.smithi069.stderr:systemctl: inactive 2024-04-14T13:51:17.833 INFO:teuthology.orchestra.run.smithi069.stderr:firewalld.service is not enabled 2024-04-14T13:51:17.833 INFO:teuthology.orchestra.run.smithi069.stderr:Not possible to open ports <[8443]>. firewalld.service is not available 2024-04-14T13:51:17.835 INFO:teuthology.orchestra.run.smithi069.stderr:Ceph Dashboard is now available at: 2024-04-14T13:51:17.835 INFO:teuthology.orchestra.run.smithi069.stderr: 2024-04-14T13:51:17.835 INFO:teuthology.orchestra.run.smithi069.stderr: URL: https://smithi069.front.sepia.ceph.com:8443/ 2024-04-14T13:51:17.836 INFO:teuthology.orchestra.run.smithi069.stderr: User: admin 2024-04-14T13:51:17.836 INFO:teuthology.orchestra.run.smithi069.stderr: Password: doal8sh2np 2024-04-14T13:51:17.836 INFO:teuthology.orchestra.run.smithi069.stderr: 2024-04-14T13:51:17.836 INFO:teuthology.orchestra.run.smithi069.stderr:Enabling autotune for osd_memory_target 2024-04-14T13:51:18.812 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:51:18 smithi069 ceph-mon[24818]: mgrmap e13: y(active, since 6s) 2024-04-14T13:51:19.735 INFO:teuthology.orchestra.run.smithi069.stderr:/usr/bin/ceph: set mgr/dashboard/cluster/status 2024-04-14T13:51:20.322 INFO:teuthology.orchestra.run.smithi069.stderr:You can access the Ceph CLI with: 2024-04-14T13:51:20.322 INFO:teuthology.orchestra.run.smithi069.stderr: 2024-04-14T13:51:20.322 INFO:teuthology.orchestra.run.smithi069.stderr: sudo /home/ubuntu/cephtest/cephadm shell --fsid dc326fc2-fa65-11ee-bc8e-c7b262605968 -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring 2024-04-14T13:51:20.322 INFO:teuthology.orchestra.run.smithi069.stderr: 2024-04-14T13:51:20.322 INFO:teuthology.orchestra.run.smithi069.stderr:Please consider enabling telemetry to help improve Ceph: 2024-04-14T13:51:20.322 INFO:teuthology.orchestra.run.smithi069.stderr: 2024-04-14T13:51:20.323 INFO:teuthology.orchestra.run.smithi069.stderr: ceph telemetry on 2024-04-14T13:51:20.323 INFO:teuthology.orchestra.run.smithi069.stderr: 2024-04-14T13:51:20.323 INFO:teuthology.orchestra.run.smithi069.stderr:For more information see: 2024-04-14T13:51:20.323 INFO:teuthology.orchestra.run.smithi069.stderr: 2024-04-14T13:51:20.323 INFO:teuthology.orchestra.run.smithi069.stderr: https://docs.ceph.com/docs/master/mgr/telemetry/ 2024-04-14T13:51:20.323 INFO:teuthology.orchestra.run.smithi069.stderr: 2024-04-14T13:51:20.323 INFO:teuthology.orchestra.run.smithi069.stderr:Bootstrap complete. 2024-04-14T13:51:20.362 INFO:tasks.cephadm:Fetching config... 2024-04-14T13:51:20.362 DEBUG:teuthology.orchestra.run.smithi069:> set -ex 2024-04-14T13:51:20.362 DEBUG:teuthology.orchestra.run.smithi069:> dd if=/etc/ceph/ceph.conf of=/dev/stdout 2024-04-14T13:51:20.378 INFO:tasks.cephadm:Fetching client.admin keyring... 2024-04-14T13:51:20.378 DEBUG:teuthology.orchestra.run.smithi069:> set -ex 2024-04-14T13:51:20.378 DEBUG:teuthology.orchestra.run.smithi069:> dd if=/etc/ceph/ceph.client.admin.keyring of=/dev/stdout 2024-04-14T13:51:20.433 INFO:tasks.cephadm:Fetching mon keyring... 2024-04-14T13:51:20.433 DEBUG:teuthology.orchestra.run.smithi069:> set -ex 2024-04-14T13:51:20.433 DEBUG:teuthology.orchestra.run.smithi069:> sudo dd if=/var/lib/ceph/dc326fc2-fa65-11ee-bc8e-c7b262605968/mon.a/keyring of=/dev/stdout 2024-04-14T13:51:20.500 INFO:tasks.cephadm:Fetching pub ssh key... 2024-04-14T13:51:20.500 DEBUG:teuthology.orchestra.run.smithi069:> set -ex 2024-04-14T13:51:20.500 DEBUG:teuthology.orchestra.run.smithi069:> dd if=/home/ubuntu/cephtest/ceph.pub of=/dev/stdout 2024-04-14T13:51:20.555 INFO:tasks.cephadm:Installing pub ssh key for root users... 2024-04-14T13:51:20.556 DEBUG:teuthology.orchestra.run.smithi069:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDBK4dYBdiNlF+sVmpUkmPiS8laU27/fc/CLMVJi20MPmnG51NsqpwKpfRdXdeq67Ls9GJLNG09UmjcRE49TGr3JmZtwu0DKTr6xEoDHsdd6wG0yIx21+cPJunvY0NMftr3XDwzgCJKSEugwasfg0IjTo4lrZ65gKBbGB30ZWA+/yBWZ/FgS/wPXmQimiyUVK3FQ6ZRvx0klpVrHpf55NhQnsP1ix6V8ikStd7lPh/O/Ho88plhrXmSBhwfAroZF9sAaxdCuQLybfsuYQpD0ETQpU+RPqTD3hay70aW82HjkYUZfSVe7AxQSVXBJzjucCCuM8yzwweM2rrrArWu4Jp8qox8BPhExgVAVX3172hX8FZHzOU7h4SaXIy61Wg24kivUnM3hieF8/9XD5KOKRK+a3nE0qOng0qdYMZuVQMVQZui/Tw/mQE1RbFMsITF5EHUPW1wM7Upe6UGzca8DNM/bE4VwqlBeytma7SvQzOFLyiqN3cZL2E9LXNWoujX8/0= ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-04-14T13:51:20.635 INFO:teuthology.orchestra.run.smithi069.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDBK4dYBdiNlF+sVmpUkmPiS8laU27/fc/CLMVJi20MPmnG51NsqpwKpfRdXdeq67Ls9GJLNG09UmjcRE49TGr3JmZtwu0DKTr6xEoDHsdd6wG0yIx21+cPJunvY0NMftr3XDwzgCJKSEugwasfg0IjTo4lrZ65gKBbGB30ZWA+/yBWZ/FgS/wPXmQimiyUVK3FQ6ZRvx0klpVrHpf55NhQnsP1ix6V8ikStd7lPh/O/Ho88plhrXmSBhwfAroZF9sAaxdCuQLybfsuYQpD0ETQpU+RPqTD3hay70aW82HjkYUZfSVe7AxQSVXBJzjucCCuM8yzwweM2rrrArWu4Jp8qox8BPhExgVAVX3172hX8FZHzOU7h4SaXIy61Wg24kivUnM3hieF8/9XD5KOKRK+a3nE0qOng0qdYMZuVQMVQZui/Tw/mQE1RbFMsITF5EHUPW1wM7Upe6UGzca8DNM/bE4VwqlBeytma7SvQzOFLyiqN3cZL2E9LXNWoujX8/0= ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968 2024-04-14T13:51:20.648 DEBUG:teuthology.orchestra.run.smithi122:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDBK4dYBdiNlF+sVmpUkmPiS8laU27/fc/CLMVJi20MPmnG51NsqpwKpfRdXdeq67Ls9GJLNG09UmjcRE49TGr3JmZtwu0DKTr6xEoDHsdd6wG0yIx21+cPJunvY0NMftr3XDwzgCJKSEugwasfg0IjTo4lrZ65gKBbGB30ZWA+/yBWZ/FgS/wPXmQimiyUVK3FQ6ZRvx0klpVrHpf55NhQnsP1ix6V8ikStd7lPh/O/Ho88plhrXmSBhwfAroZF9sAaxdCuQLybfsuYQpD0ETQpU+RPqTD3hay70aW82HjkYUZfSVe7AxQSVXBJzjucCCuM8yzwweM2rrrArWu4Jp8qox8BPhExgVAVX3172hX8FZHzOU7h4SaXIy61Wg24kivUnM3hieF8/9XD5KOKRK+a3nE0qOng0qdYMZuVQMVQZui/Tw/mQE1RbFMsITF5EHUPW1wM7Upe6UGzca8DNM/bE4VwqlBeytma7SvQzOFLyiqN3cZL2E9LXNWoujX8/0= ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-04-14T13:51:20.693 INFO:teuthology.orchestra.run.smithi122.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDBK4dYBdiNlF+sVmpUkmPiS8laU27/fc/CLMVJi20MPmnG51NsqpwKpfRdXdeq67Ls9GJLNG09UmjcRE49TGr3JmZtwu0DKTr6xEoDHsdd6wG0yIx21+cPJunvY0NMftr3XDwzgCJKSEugwasfg0IjTo4lrZ65gKBbGB30ZWA+/yBWZ/FgS/wPXmQimiyUVK3FQ6ZRvx0klpVrHpf55NhQnsP1ix6V8ikStd7lPh/O/Ho88plhrXmSBhwfAroZF9sAaxdCuQLybfsuYQpD0ETQpU+RPqTD3hay70aW82HjkYUZfSVe7AxQSVXBJzjucCCuM8yzwweM2rrrArWu4Jp8qox8BPhExgVAVX3172hX8FZHzOU7h4SaXIy61Wg24kivUnM3hieF8/9XD5KOKRK+a3nE0qOng0qdYMZuVQMVQZui/Tw/mQE1RbFMsITF5EHUPW1wM7Upe6UGzca8DNM/bE4VwqlBeytma7SvQzOFLyiqN3cZL2E9LXNWoujX8/0= ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968 2024-04-14T13:51:20.709 DEBUG:teuthology.orchestra.run.smithi069:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid dc326fc2-fa65-11ee-bc8e-c7b262605968 -- ceph config set mgr mgr/cephadm/allow_ptrace true 2024-04-14T13:51:20.856 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:51:20 smithi069 ceph-mon[24818]: from='client.? 172.21.15.69:0/3990410903' entity='client.admin' 2024-04-14T13:51:23.568 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:51:23 smithi069 ceph-mon[24818]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' 2024-04-14T13:51:23.568 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:51:23 smithi069 ceph-mon[24818]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi069", "name": "osd_memory_target"}]: dispatch 2024-04-14T13:51:23.568 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:51:23 smithi069 ceph-mon[24818]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' 2024-04-14T13:51:23.568 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:51:23 smithi069 ceph-mon[24818]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-14T13:51:23.568 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:51:23 smithi069 ceph-mon[24818]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' 2024-04-14T13:51:24.173 INFO:tasks.cephadm:Distributing conf and client.admin keyring to all hosts + 0755 2024-04-14T13:51:24.174 DEBUG:teuthology.orchestra.run.smithi069:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid dc326fc2-fa65-11ee-bc8e-c7b262605968 -- ceph orch client-keyring set client.admin '*' --mode 0755 2024-04-14T13:51:24.818 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:51:24 smithi069 ceph-mon[24818]: from='client.? 172.21.15.69:0/2198760377' entity='client.admin' 2024-04-14T13:51:24.818 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:51:24 smithi069 ceph-mon[24818]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-14T13:51:24.818 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:51:24 smithi069 ceph-mon[24818]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' 2024-04-14T13:51:26.709 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:51:26 smithi069 ceph-mon[24818]: from='client.14172 -' entity='client.admin' cmd=[{"prefix": "orch client-keyring set", "entity": "client.admin", "placement": "*", "mode": "0755", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T13:51:26.710 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:51:26 smithi069 ceph-mon[24818]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' 2024-04-14T13:51:26.710 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:51:26 smithi069 ceph-mon[24818]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-14T13:51:26.710 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:51:26 smithi069 ceph-mon[24818]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T13:51:26.710 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:51:26 smithi069 ceph-mon[24818]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-14T13:51:27.128 INFO:tasks.cephadm:Writing (initial) conf and keyring to smithi122 2024-04-14T13:51:27.128 DEBUG:teuthology.orchestra.run.smithi122:> set -ex 2024-04-14T13:51:27.128 DEBUG:teuthology.orchestra.run.smithi122:> dd of=/etc/ceph/ceph.conf 2024-04-14T13:51:27.149 DEBUG:teuthology.orchestra.run.smithi122:> set -ex 2024-04-14T13:51:27.149 DEBUG:teuthology.orchestra.run.smithi122:> dd of=/etc/ceph/ceph.client.admin.keyring 2024-04-14T13:51:27.207 INFO:tasks.cephadm:Adding host smithi122 to orchestrator... 2024-04-14T13:51:27.208 DEBUG:teuthology.orchestra.run.smithi069:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid dc326fc2-fa65-11ee-bc8e-c7b262605968 -- ceph orch host add smithi122 2024-04-14T13:51:28.165 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:51:27 smithi069 ceph-mon[24818]: Updating smithi069:/etc/ceph/ceph.conf 2024-04-14T13:51:28.165 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:51:27 smithi069 ceph-mon[24818]: Updating smithi069:/etc/ceph/ceph.client.admin.keyring 2024-04-14T13:51:28.166 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:51:27 smithi069 ceph-mon[24818]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' 2024-04-14T13:51:28.166 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:51:27 smithi069 ceph-mon[24818]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' 2024-04-14T13:51:29.817 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:51:29 smithi069 ceph-mon[24818]: from='client.14174 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi122", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T13:51:30.050 INFO:teuthology.orchestra.run.smithi069.stdout:Added host 'smithi122' with addr '172.21.15.122' 2024-04-14T13:51:30.585 DEBUG:teuthology.orchestra.run.smithi069:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid dc326fc2-fa65-11ee-bc8e-c7b262605968 -- ceph orch host ls --format=json 2024-04-14T13:51:31.183 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:51:31 smithi069 ceph-mon[24818]: Deploying cephadm binary to smithi122 2024-04-14T13:51:31.183 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:51:31 smithi069 ceph-mon[24818]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' 2024-04-14T13:51:31.183 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:51:31 smithi069 ceph-mon[24818]: Added host smithi122 2024-04-14T13:51:31.183 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:51:31 smithi069 ceph-mon[24818]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-14T13:51:31.183 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:51:31 smithi069 ceph-mon[24818]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T13:51:31.183 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:51:31 smithi069 ceph-mon[24818]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-14T13:51:31.183 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:51:31 smithi069 ceph-mon[24818]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' 2024-04-14T13:51:31.183 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:51:31 smithi069 ceph-mon[24818]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' 2024-04-14T13:51:32.424 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-14T13:51:32.424 INFO:teuthology.orchestra.run.smithi069.stdout:[{"addr": "172.21.15.69", "hostname": "smithi069", "labels": [], "status": ""}, {"addr": "172.21.15.122", "hostname": "smithi122", "labels": [], "status": ""}] 2024-04-14T13:51:32.950 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:51:32 smithi069 ceph-mon[24818]: pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-14T13:51:32.951 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:51:32 smithi069 ceph-mon[24818]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' 2024-04-14T13:51:32.951 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:51:32 smithi069 ceph-mon[24818]: from='client.14176 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-14T13:51:32.974 INFO:tasks.cephadm:Setting crush tunables to default 2024-04-14T13:51:32.974 DEBUG:teuthology.orchestra.run.smithi069:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid dc326fc2-fa65-11ee-bc8e-c7b262605968 -- ceph osd crush tunables default 2024-04-14T13:51:35.309 INFO:teuthology.orchestra.run.smithi069.stderr:adjusted tunables profile to default 2024-04-14T13:51:35.567 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:51:35 smithi069 ceph-mon[24818]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-14T13:51:35.568 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:51:35 smithi069 ceph-mon[24818]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' 2024-04-14T13:51:35.568 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:51:35 smithi069 ceph-mon[24818]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' 2024-04-14T13:51:35.568 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:51:35 smithi069 ceph-mon[24818]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi122", "name": "osd_memory_target"}]: dispatch 2024-04-14T13:51:35.568 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:51:35 smithi069 ceph-mon[24818]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' 2024-04-14T13:51:35.568 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:51:35 smithi069 ceph-mon[24818]: from='client.? 172.21.15.69:0/1281184648' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2024-04-14T13:51:35.841 INFO:tasks.cephadm:Adding mon.a on smithi069 2024-04-14T13:51:35.841 INFO:tasks.cephadm:Adding mon.c on smithi069 2024-04-14T13:51:35.841 INFO:tasks.cephadm:Adding mon.b on smithi122 2024-04-14T13:51:35.841 DEBUG:teuthology.orchestra.run.smithi122:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid dc326fc2-fa65-11ee-bc8e-c7b262605968 -- ceph orch apply mon '3;smithi069:172.21.15.69=a;smithi069:[v2:172.21.15.69:3301,v1:172.21.15.69:6790]=c;smithi122:172.21.15.122=b' 2024-04-14T13:51:36.567 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:51:36 smithi069 ceph-mon[24818]: from='client.? 172.21.15.69:0/1281184648' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-04-14T13:51:36.567 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:51:36 smithi069 ceph-mon[24818]: osdmap e4: 0 total, 0 up, 0 in 2024-04-14T13:51:37.567 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:51:37 smithi069 ceph-mon[24818]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-14T13:51:37.628 INFO:teuthology.orchestra.run.smithi122.stdout:Scheduled mon update... 2024-04-14T13:51:38.274 DEBUG:teuthology.orchestra.run.smithi069:mon.c> sudo journalctl -f -n 0 -u ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968@mon.c.service 2024-04-14T13:51:38.276 DEBUG:teuthology.orchestra.run.smithi122:mon.b> sudo journalctl -f -n 0 -u ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968@mon.b.service 2024-04-14T13:51:38.278 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-04-14T13:51:38.278 DEBUG:teuthology.orchestra.run.smithi122:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid dc326fc2-fa65-11ee-bc8e-c7b262605968 -- ceph mon dump -f json 2024-04-14T13:51:38.865 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:51:38 smithi069 ceph-mon[24818]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-14T13:51:38.865 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:51:38 smithi069 ceph-mon[24818]: from='client.14180 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi069:172.21.15.69=a;smithi069:[v2:172.21.15.69:3301,v1:172.21.15.69:6790]=c;smithi122:172.21.15.122=b", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T13:51:38.865 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:51:38 smithi069 ceph-mon[24818]: Saving service mon spec with placement smithi069:172.21.15.69=a;smithi069:[v2:172.21.15.69:3301,v1:172.21.15.69:6790]=c;smithi122:172.21.15.122=b;count:3 2024-04-14T13:51:38.865 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:51:38 smithi069 ceph-mon[24818]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' 2024-04-14T13:51:38.865 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:51:38 smithi069 ceph-mon[24818]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-14T13:51:38.865 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:51:38 smithi069 ceph-mon[24818]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T13:51:38.866 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:51:38 smithi069 ceph-mon[24818]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-14T13:51:38.866 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:51:38 smithi069 ceph-mon[24818]: Updating smithi122:/etc/ceph/ceph.conf 2024-04-14T13:51:38.866 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:51:38 smithi069 ceph-mon[24818]: Updating smithi122:/etc/ceph/ceph.client.admin.keyring 2024-04-14T13:51:38.866 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:51:38 smithi069 ceph-mon[24818]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' 2024-04-14T13:51:38.866 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:51:38 smithi069 ceph-mon[24818]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' 2024-04-14T13:51:38.866 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:51:38 smithi069 ceph-mon[24818]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-14T13:51:38.866 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:51:38 smithi069 ceph-mon[24818]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T13:51:38.866 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:51:38 smithi069 ceph-mon[24818]: Deploying daemon mon.c on smithi069 2024-04-14T13:51:40.211 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-04-14T13:51:40.212 INFO:teuthology.orchestra.run.smithi122.stdout:{"epoch":1,"fsid":"dc326fc2-fa65-11ee-bc8e-c7b262605968","modified":"2024-04-14T13:50:18.166086Z","created":"2024-04-14T13:50:18.166086Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:3300","nonce":0},{"type":"v1","addr":"172.21.15.69:6789","nonce":0}]},"addr":"172.21.15.69:6789/0","public_addr":"172.21.15.69:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-04-14T13:51:40.213 INFO:teuthology.orchestra.run.smithi122.stderr:dumped monmap epoch 1 2024-04-14T13:51:41.317 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:51:41 smithi069 ceph-mon[24818]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-14T13:51:41.317 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:51:41 smithi069 ceph-mon[24818]: from='client.? 172.21.15.122:0/3564277494' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-14T13:51:41.763 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-04-14T13:51:41.764 DEBUG:teuthology.orchestra.run.smithi122:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid dc326fc2-fa65-11ee-bc8e-c7b262605968 -- ceph mon dump -f json 2024-04-14T13:51:46.568 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:51:46 smithi069 ceph-mon[29472]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T13:51:46.568 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:51:46 smithi069 ceph-mon[24818]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-14T13:51:46.568 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:51:46 smithi069 ceph-mon[24818]: mon.a calling monitor election 2024-04-14T13:51:46.568 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:51:46 smithi069 ceph-mon[24818]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-14T13:51:46.569 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:51:46 smithi069 ceph-mon[24818]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-14T13:51:46.569 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:51:46 smithi069 ceph-mon[24818]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-14T13:51:46.569 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:51:46 smithi069 ceph-mon[24818]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-14T13:51:46.569 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:51:46 smithi069 ceph-mon[24818]: mon.c calling monitor election 2024-04-14T13:51:46.569 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:51:46 smithi069 ceph-mon[24818]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-14T13:51:46.569 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:51:46 smithi069 ceph-mon[24818]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-14T13:51:46.569 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:51:46 smithi069 ceph-mon[24818]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-14T13:51:46.569 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:51:46 smithi069 ceph-mon[24818]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-14T13:51:46.569 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:51:46 smithi069 ceph-mon[24818]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-04-14T13:51:46.569 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:51:46 smithi069 ceph-mon[24818]: monmap e2: 2 mons at {a=[v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0],c=[v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0]} 2024-04-14T13:51:46.569 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:51:46 smithi069 ceph-mon[24818]: fsmap 2024-04-14T13:51:46.569 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:51:46 smithi069 ceph-mon[24818]: osdmap e4: 0 total, 0 up, 0 in 2024-04-14T13:51:46.569 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:51:46 smithi069 ceph-mon[24818]: mgrmap e13: y(active, since 34s) 2024-04-14T13:51:46.570 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:51:46 smithi069 ceph-mon[24818]: overall HEALTH_OK 2024-04-14T13:51:46.570 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:51:46 smithi069 ceph-mon[24818]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' 2024-04-14T13:51:46.570 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:51:46 smithi069 ceph-mon[24818]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-14T13:51:46.570 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:51:46 smithi069 ceph-mon[24818]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T13:51:46.710 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-04-14T13:51:46.710 INFO:teuthology.orchestra.run.smithi122.stdout:{"epoch":2,"fsid":"dc326fc2-fa65-11ee-bc8e-c7b262605968","modified":"2024-04-14T13:51:41.165379Z","created":"2024-04-14T13:50:18.166086Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:3300","nonce":0},{"type":"v1","addr":"172.21.15.69:6789","nonce":0}]},"addr":"172.21.15.69:6789/0","public_addr":"172.21.15.69:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:3301","nonce":0},{"type":"v1","addr":"172.21.15.69:6790","nonce":0}]},"addr":"172.21.15.69:6790/0","public_addr":"172.21.15.69:6790/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1]} 2024-04-14T13:51:46.712 INFO:teuthology.orchestra.run.smithi122.stderr:dumped monmap epoch 2 2024-04-14T13:51:47.567 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:51:47 smithi069 ceph-mon[24818]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-14T13:51:47.568 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:51:47 smithi069 ceph-mon[24818]: Deploying daemon mon.b on smithi122 2024-04-14T13:51:47.568 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:51:47 smithi069 ceph-mon[24818]: from='client.? 172.21.15.122:0/2873611874' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-14T13:51:47.568 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:51:47 smithi069 ceph-mon[24818]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-14T13:51:47.568 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:51:47 smithi069 ceph-mon[29472]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-14T13:51:47.568 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:51:47 smithi069 ceph-mon[29472]: Deploying daemon mon.b on smithi122 2024-04-14T13:51:47.568 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:51:47 smithi069 ceph-mon[29472]: from='client.? 172.21.15.122:0/2873611874' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-14T13:51:47.568 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:51:47 smithi069 ceph-mon[29472]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-14T13:51:48.611 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-04-14T13:51:48.611 DEBUG:teuthology.orchestra.run.smithi122:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid dc326fc2-fa65-11ee-bc8e-c7b262605968 -- ceph mon dump -f json 2024-04-14T13:51:49.567 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:51:49 smithi069 ceph-mon[24818]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-14T13:51:49.568 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:51:49 smithi069 ceph-mon[29472]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-14T13:51:49.952 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:51:49 smithi122 systemd[1]: Starting Ceph mon.b for dc326fc2-fa65-11ee-bc8e-c7b262605968... 2024-04-14T13:51:50.949 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-04-14T13:51:50.949 INFO:teuthology.orchestra.run.smithi122.stdout:{"epoch":2,"fsid":"dc326fc2-fa65-11ee-bc8e-c7b262605968","modified":"2024-04-14T13:51:41.165379Z","created":"2024-04-14T13:50:18.166086Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:3300","nonce":0},{"type":"v1","addr":"172.21.15.69:6789","nonce":0}]},"addr":"172.21.15.69:6789/0","public_addr":"172.21.15.69:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:3301","nonce":0},{"type":"v1","addr":"172.21.15.69:6790","nonce":0}]},"addr":"172.21.15.69:6790/0","public_addr":"172.21.15.69:6790/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1]} 2024-04-14T13:51:50.950 INFO:teuthology.orchestra.run.smithi122.stderr:dumped monmap epoch 2 2024-04-14T13:51:51.027 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:51:50 smithi122 podman[25680]: 2024-04-14 13:51:50.770507167 +0000 UTC m=+0.284916654 container create 3b766bfce4be5663d61ecbbff3cf9815245b7c7a653cd757cf48d86884426543 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mon-b, RELEASE=HEAD, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, maintainer=Guillaume Abrioux , com.redhat.component=centos-stream-container, ceph=True, GIT_BRANCH=HEAD, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, vcs-type=git, io.openshift.tags=base centos centos-stream, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, version=8, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, io.openshift.expose-services=, distribution-scope=public, vendor=Red Hat, Inc., description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, io.k8s.display-name=CentOS Stream 8, GIT_REPO=https://github.com/ceph/ceph-container.git, io.buildah.version=1.19.8, CEPH_POINT_RELEASE=-17.2.0, release=754, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., com.redhat.license_terms=https://centos.org/legal/licensing-policy/, GIT_CLEAN=True, architecture=x86_64, build-date=2022-05-03T08:36:31.336870, name=centos-stream) 2024-04-14T13:51:51.027 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:51:50 smithi122 podman[25680]: 2024-04-14 13:51:50.683469839 +0000 UTC m=+0.197879328 image pull e1d6a67b021eb077ee22bf650f1a9fb1980a2cf5c36bdb9cba9eac6de8f702d9 quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a 2024-04-14T13:51:51.028 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:51:50 smithi122 podman[25680]: 2024-04-14 13:51:50.914415437 +0000 UTC m=+0.428824921 container init 3b766bfce4be5663d61ecbbff3cf9815245b7c7a653cd757cf48d86884426543 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mon-b, GIT_CLEAN=True, maintainer=Guillaume Abrioux , description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, io.openshift.expose-services=, CEPH_POINT_RELEASE=-17.2.0, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, RELEASE=HEAD, distribution-scope=public, GIT_BRANCH=HEAD, build-date=2022-05-03T08:36:31.336870, io.buildah.version=1.19.8, version=8, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, architecture=x86_64, name=centos-stream, com.redhat.component=centos-stream-container, io.openshift.tags=base centos centos-stream, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., com.redhat.license_terms=https://centos.org/legal/licensing-policy/, ceph=True, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, GIT_REPO=https://github.com/ceph/ceph-container.git, vendor=Red Hat, Inc., release=754, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, io.k8s.display-name=CentOS Stream 8, vcs-type=git) 2024-04-14T13:51:51.028 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:51:50 smithi122 podman[25680]: 2024-04-14 13:51:50.918392685 +0000 UTC m=+0.432802170 container start 3b766bfce4be5663d61ecbbff3cf9815245b7c7a653cd757cf48d86884426543 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mon-b, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, io.k8s.display-name=CentOS Stream 8, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, vcs-type=git, io.buildah.version=1.19.8, io.openshift.tags=base centos centos-stream, ceph=True, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, GIT_BRANCH=HEAD, version=8, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, GIT_CLEAN=True, maintainer=Guillaume Abrioux , build-date=2022-05-03T08:36:31.336870, GIT_REPO=https://github.com/ceph/ceph-container.git, RELEASE=HEAD, distribution-scope=public, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, com.redhat.component=centos-stream-container, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, vendor=Red Hat, Inc., CEPH_POINT_RELEASE=-17.2.0, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., io.openshift.expose-services=, architecture=x86_64, release=754, name=centos-stream) 2024-04-14T13:51:51.028 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:51:50 smithi122 ceph-mon[25721]: set uid:gid to 167:167 (ceph:ceph) 2024-04-14T13:51:51.028 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:51:50 smithi122 ceph-mon[25721]: ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable), process ceph-mon, pid 2 2024-04-14T13:51:51.028 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:51:50 smithi122 ceph-mon[25721]: pidfile_write: ignore empty --pid-file 2024-04-14T13:51:51.028 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:51:50 smithi122 ceph-mon[25721]: load: jerasure load: lrc 2024-04-14T13:51:51.028 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:51:50 smithi122 ceph-mon[25721]: rocksdb: RocksDB version: 6.15.5 2024-04-14T13:51:51.028 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:51:50 smithi122 ceph-mon[25721]: rocksdb: Git sha rocksdb_build_git_sha:@0@ 2024-04-14T13:51:51.028 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:51:50 smithi122 ceph-mon[25721]: rocksdb: Compile date Apr 18 2022 2024-04-14T13:51:51.028 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:51:50 smithi122 ceph-mon[25721]: rocksdb: DB SUMMARY 2024-04-14T13:51:51.028 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:51:50 smithi122 ceph-mon[25721]: rocksdb: DB Session ID: IUSIJZFA3K6LSW8A0IC9 2024-04-14T13:51:51.028 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:51:50 smithi122 ceph-mon[25721]: rocksdb: CURRENT file: CURRENT 2024-04-14T13:51:51.028 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:51:50 smithi122 ceph-mon[25721]: rocksdb: IDENTITY file: IDENTITY 2024-04-14T13:51:51.028 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:51:50 smithi122 ceph-mon[25721]: rocksdb: MANIFEST file: MANIFEST-000003 size: 57 Bytes 2024-04-14T13:51:51.028 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:51:50 smithi122 ceph-mon[25721]: rocksdb: SST files in /var/lib/ceph/mon/ceph-b/store.db dir, Total Num: 0, files: 2024-04-14T13:51:51.029 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:51:50 smithi122 ceph-mon[25721]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-b/store.db: 000004.log size: 636 ; 2024-04-14T13:51:51.029 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:51:50 smithi122 ceph-mon[25721]: rocksdb: Options.error_if_exists: 0 2024-04-14T13:51:51.029 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:51:50 smithi122 ceph-mon[25721]: rocksdb: Options.create_if_missing: 0 2024-04-14T13:51:51.029 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:51:50 smithi122 ceph-mon[25721]: rocksdb: Options.paranoid_checks: 1 2024-04-14T13:51:51.029 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:51:50 smithi122 ceph-mon[25721]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-04-14T13:51:51.029 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:51:50 smithi122 ceph-mon[25721]: rocksdb: Options.env: 0x55c2a1ac6860 2024-04-14T13:51:51.029 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:51:50 smithi122 ceph-mon[25721]: rocksdb: Options.fs: Posix File System 2024-04-14T13:51:51.029 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:51:50 smithi122 ceph-mon[25721]: rocksdb: Options.info_log: 0x55c2a4229dc0 2024-04-14T13:51:51.029 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:51:50 smithi122 ceph-mon[25721]: rocksdb: Options.max_file_opening_threads: 16 2024-04-14T13:51:51.029 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:51:50 smithi122 ceph-mon[25721]: rocksdb: Options.statistics: (nil) 2024-04-14T13:51:51.029 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:51:50 smithi122 ceph-mon[25721]: rocksdb: Options.use_fsync: 0 2024-04-14T13:51:51.031 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:51:50 smithi122 ceph-mon[25721]: rocksdb: Options.max_log_file_size: 0 2024-04-14T13:51:51.031 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:51:50 smithi122 ceph-mon[25721]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-04-14T13:51:51.031 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:51:50 smithi122 ceph-mon[25721]: rocksdb: Options.log_file_time_to_roll: 0 2024-04-14T13:51:51.031 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:51:50 smithi122 ceph-mon[25721]: rocksdb: Options.keep_log_file_num: 1000 2024-04-14T13:51:51.031 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:51:50 smithi122 ceph-mon[25721]: rocksdb: Options.recycle_log_file_num: 0 2024-04-14T13:51:51.031 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:51:50 smithi122 ceph-mon[25721]: rocksdb: Options.allow_fallocate: 1 2024-04-14T13:51:51.031 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:51:50 smithi122 ceph-mon[25721]: rocksdb: Options.allow_mmap_reads: 0 2024-04-14T13:51:51.031 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:51:50 smithi122 ceph-mon[25721]: rocksdb: Options.allow_mmap_writes: 0 2024-04-14T13:51:51.031 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:51:50 smithi122 ceph-mon[25721]: rocksdb: Options.use_direct_reads: 0 2024-04-14T13:51:51.031 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:51:50 smithi122 ceph-mon[25721]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-04-14T13:51:51.031 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:51:50 smithi122 ceph-mon[25721]: rocksdb: Options.create_missing_column_families: 0 2024-04-14T13:51:51.031 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:51:50 smithi122 ceph-mon[25721]: rocksdb: Options.db_log_dir: 2024-04-14T13:51:51.031 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:51:50 smithi122 ceph-mon[25721]: rocksdb: Options.wal_dir: /var/lib/ceph/mon/ceph-b/store.db 2024-04-14T13:51:51.032 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:51:50 smithi122 ceph-mon[25721]: rocksdb: Options.table_cache_numshardbits: 6 2024-04-14T13:51:51.032 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:51:50 smithi122 ceph-mon[25721]: rocksdb: Options.WAL_ttl_seconds: 0 2024-04-14T13:51:51.032 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:51:50 smithi122 ceph-mon[25721]: rocksdb: Options.WAL_size_limit_MB: 0 2024-04-14T13:51:51.032 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:51:50 smithi122 ceph-mon[25721]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-04-14T13:51:51.032 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:51:50 smithi122 ceph-mon[25721]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-04-14T13:51:51.032 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:51:50 smithi122 ceph-mon[25721]: rocksdb: Options.is_fd_close_on_exec: 1 2024-04-14T13:51:51.032 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:51:50 smithi122 ceph-mon[25721]: rocksdb: Options.advise_random_on_open: 1 2024-04-14T13:51:51.032 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:51:50 smithi122 ceph-mon[25721]: rocksdb: Options.db_write_buffer_size: 0 2024-04-14T13:51:51.032 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:51:50 smithi122 ceph-mon[25721]: rocksdb: Options.write_buffer_manager: 0x55c2a431a240 2024-04-14T13:51:51.032 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:51:50 smithi122 ceph-mon[25721]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-04-14T13:51:51.032 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:51:50 smithi122 ceph-mon[25721]: rocksdb: Options.new_table_reader_for_compaction_inputs: 0 2024-04-14T13:51:51.032 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:51:50 smithi122 ceph-mon[25721]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-04-14T13:51:51.032 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:51:50 smithi122 ceph-mon[25721]: rocksdb: Options.use_adaptive_mutex: 0 2024-04-14T13:51:51.032 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:51:50 smithi122 ceph-mon[25721]: rocksdb: Options.rate_limiter: (nil) 2024-04-14T13:51:51.032 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:51:50 smithi122 ceph-mon[25721]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-04-14T13:51:51.033 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:51:50 smithi122 ceph-mon[25721]: rocksdb: Options.wal_recovery_mode: 2 2024-04-14T13:51:51.033 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:51:50 smithi122 ceph-mon[25721]: rocksdb: Options.enable_thread_tracking: 0 2024-04-14T13:51:51.033 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:51:50 smithi122 ceph-mon[25721]: rocksdb: Options.enable_pipelined_write: 0 2024-04-14T13:51:51.033 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:51:50 smithi122 ceph-mon[25721]: rocksdb: Options.unordered_write: 0 2024-04-14T13:51:51.033 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:51:50 smithi122 ceph-mon[25721]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-04-14T13:51:51.033 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:51:50 smithi122 ceph-mon[25721]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-04-14T13:51:51.033 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:51:50 smithi122 ceph-mon[25721]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-04-14T13:51:51.033 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:51:50 smithi122 ceph-mon[25721]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-04-14T13:51:51.033 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:51:50 smithi122 ceph-mon[25721]: rocksdb: Options.row_cache: None 2024-04-14T13:51:51.033 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:51:50 smithi122 ceph-mon[25721]: rocksdb: Options.wal_filter: None 2024-04-14T13:51:51.033 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:51:50 smithi122 ceph-mon[25721]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-04-14T13:51:51.033 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:51:50 smithi122 ceph-mon[25721]: rocksdb: Options.allow_ingest_behind: 0 2024-04-14T13:51:51.033 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:51:50 smithi122 ceph-mon[25721]: rocksdb: Options.preserve_deletes: 0 2024-04-14T13:51:51.033 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:51:50 smithi122 ceph-mon[25721]: rocksdb: Options.two_write_queues: 0 2024-04-14T13:51:51.033 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:51:50 smithi122 ceph-mon[25721]: rocksdb: Options.manual_wal_flush: 0 2024-04-14T13:51:51.033 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:51:50 smithi122 ceph-mon[25721]: rocksdb: Options.atomic_flush: 0 2024-04-14T13:51:51.034 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:51:50 smithi122 ceph-mon[25721]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-04-14T13:51:51.034 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:51:50 smithi122 ceph-mon[25721]: rocksdb: Options.persist_stats_to_disk: 0 2024-04-14T13:51:51.034 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:51:50 smithi122 ceph-mon[25721]: rocksdb: Options.write_dbid_to_manifest: 0 2024-04-14T13:51:51.034 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:51:50 smithi122 ceph-mon[25721]: rocksdb: Options.log_readahead_size: 0 2024-04-14T13:51:51.034 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:51:50 smithi122 ceph-mon[25721]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-04-14T13:51:51.034 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:51:50 smithi122 ceph-mon[25721]: rocksdb: Options.best_efforts_recovery: 0 2024-04-14T13:51:51.034 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:51:50 smithi122 ceph-mon[25721]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-04-14T13:51:51.034 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:51:50 smithi122 ceph-mon[25721]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-04-14T13:51:51.034 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:51:50 smithi122 ceph-mon[25721]: rocksdb: Options.allow_data_in_errors: 0 2024-04-14T13:51:51.034 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:51:50 smithi122 ceph-mon[25721]: rocksdb: Options.db_host_id: __hostname__ 2024-04-14T13:51:51.034 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:51:50 smithi122 ceph-mon[25721]: rocksdb: Options.max_background_jobs: 2 2024-04-14T13:51:51.034 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:51:50 smithi122 ceph-mon[25721]: rocksdb: Options.max_background_compactions: -1 2024-04-14T13:51:51.034 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:51:50 smithi122 ceph-mon[25721]: rocksdb: Options.max_subcompactions: 1 2024-04-14T13:51:51.034 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:51:50 smithi122 ceph-mon[25721]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-04-14T13:51:51.034 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:51:50 smithi122 ceph-mon[25721]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-04-14T13:51:51.035 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:51:50 smithi122 ceph-mon[25721]: rocksdb: Options.delayed_write_rate : 16777216 2024-04-14T13:51:51.035 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:51:50 smithi122 ceph-mon[25721]: rocksdb: Options.max_total_wal_size: 0 2024-04-14T13:51:51.035 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:51:50 smithi122 ceph-mon[25721]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-04-14T13:51:51.035 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:51:50 smithi122 ceph-mon[25721]: rocksdb: Options.stats_dump_period_sec: 600 2024-04-14T13:51:51.035 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:51:50 smithi122 ceph-mon[25721]: rocksdb: Options.stats_persist_period_sec: 600 2024-04-14T13:51:51.035 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:51:50 smithi122 ceph-mon[25721]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-04-14T13:51:51.035 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:51:50 smithi122 ceph-mon[25721]: rocksdb: Options.max_open_files: -1 2024-04-14T13:51:51.035 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:51:50 smithi122 ceph-mon[25721]: rocksdb: Options.bytes_per_sync: 0 2024-04-14T13:51:51.035 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:51:50 smithi122 ceph-mon[25721]: rocksdb: Options.wal_bytes_per_sync: 0 2024-04-14T13:51:51.035 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:51:50 smithi122 ceph-mon[25721]: rocksdb: Options.strict_bytes_per_sync: 0 2024-04-14T13:51:51.035 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:51:50 smithi122 ceph-mon[25721]: rocksdb: Options.compaction_readahead_size: 0 2024-04-14T13:51:51.035 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:51:50 smithi122 ceph-mon[25721]: rocksdb: Options.max_background_flushes: -1 2024-04-14T13:51:51.035 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:51:50 smithi122 ceph-mon[25721]: rocksdb: Compression algorithms supported: 2024-04-14T13:51:51.035 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:51:50 smithi122 ceph-mon[25721]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-04-14T13:51:51.035 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:51:50 smithi122 ceph-mon[25721]: rocksdb: kZSTD supported: 0 2024-04-14T13:51:51.035 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:51:50 smithi122 ceph-mon[25721]: rocksdb: kXpressCompression supported: 0 2024-04-14T13:51:51.036 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:51:50 smithi122 ceph-mon[25721]: rocksdb: kLZ4HCCompression supported: 1 2024-04-14T13:51:51.036 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:51:50 smithi122 ceph-mon[25721]: rocksdb: kLZ4Compression supported: 1 2024-04-14T13:51:51.036 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:51:50 smithi122 ceph-mon[25721]: rocksdb: kBZip2Compression supported: 0 2024-04-14T13:51:51.036 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:51:50 smithi122 ceph-mon[25721]: rocksdb: kZlibCompression supported: 1 2024-04-14T13:51:51.036 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:51:50 smithi122 ceph-mon[25721]: rocksdb: kSnappyCompression supported: 1 2024-04-14T13:51:51.036 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:51:50 smithi122 ceph-mon[25721]: rocksdb: Fast CRC32 supported: Supported on x86 2024-04-14T13:51:51.036 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:51:50 smithi122 ceph-mon[25721]: rocksdb: [db/version_set.cc:4725] Recovering from manifest file: /var/lib/ceph/mon/ceph-b/store.db/MANIFEST-000003 2024-04-14T13:51:51.036 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:51:50 smithi122 ceph-mon[25721]: rocksdb: [db/column_family.cc:597] --------------- Options for column family [default]: 2024-04-14T13:51:51.036 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:51:50 smithi122 ceph-mon[25721]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-04-14T13:51:51.036 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:51:50 smithi122 ceph-mon[25721]: rocksdb: Options.merge_operator: 2024-04-14T13:51:51.036 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:51:50 smithi122 ceph-mon[25721]: rocksdb: Options.compaction_filter: None 2024-04-14T13:51:51.036 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:51:50 smithi122 ceph-mon[25721]: rocksdb: Options.compaction_filter_factory: None 2024-04-14T13:51:51.036 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:51:50 smithi122 ceph-mon[25721]: rocksdb: Options.sst_partitioner_factory: None 2024-04-14T13:51:51.036 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:51:50 smithi122 ceph-mon[25721]: rocksdb: Options.memtable_factory: SkipListFactory 2024-04-14T13:51:51.036 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:51:50 smithi122 ceph-mon[25721]: rocksdb: Options.table_factory: BlockBasedTable 2024-04-14T13:51:51.036 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:51:50 smithi122 ceph-mon[25721]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x55c2a41f5d00) 2024-04-14T13:51:51.037 INFO:journalctl@ceph.mon.b.smithi122.stdout: cache_index_and_filter_blocks: 1 2024-04-14T13:51:51.037 INFO:journalctl@ceph.mon.b.smithi122.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-04-14T13:51:51.037 INFO:journalctl@ceph.mon.b.smithi122.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-04-14T13:51:51.037 INFO:journalctl@ceph.mon.b.smithi122.stdout: pin_top_level_index_and_filter: 1 2024-04-14T13:51:51.037 INFO:journalctl@ceph.mon.b.smithi122.stdout: index_type: 0 2024-04-14T13:51:51.037 INFO:journalctl@ceph.mon.b.smithi122.stdout: data_block_index_type: 0 2024-04-14T13:51:51.037 INFO:journalctl@ceph.mon.b.smithi122.stdout: index_shortening: 1 2024-04-14T13:51:51.037 INFO:journalctl@ceph.mon.b.smithi122.stdout: data_block_hash_table_util_ratio: 0.750000 2024-04-14T13:51:51.037 INFO:journalctl@ceph.mon.b.smithi122.stdout: hash_index_allow_collision: 1 2024-04-14T13:51:51.037 INFO:journalctl@ceph.mon.b.smithi122.stdout: checksum: 1 2024-04-14T13:51:51.037 INFO:journalctl@ceph.mon.b.smithi122.stdout: no_block_cache: 0 2024-04-14T13:51:51.037 INFO:journalctl@ceph.mon.b.smithi122.stdout: block_cache: 0x55c2a4260170 2024-04-14T13:51:51.037 INFO:journalctl@ceph.mon.b.smithi122.stdout: block_cache_name: BinnedLRUCache 2024-04-14T13:51:51.037 INFO:journalctl@ceph.mon.b.smithi122.stdout: block_cache_options: 2024-04-14T13:51:51.037 INFO:journalctl@ceph.mon.b.smithi122.stdout: capacity : 536870912 2024-04-14T13:51:51.037 INFO:journalctl@ceph.mon.b.smithi122.stdout: num_shard_bits : 4 2024-04-14T13:51:51.038 INFO:journalctl@ceph.mon.b.smithi122.stdout: strict_capacity_limit : 0 2024-04-14T13:51:51.038 INFO:journalctl@ceph.mon.b.smithi122.stdout: high_pri_pool_ratio: 0.000 2024-04-14T13:51:51.038 INFO:journalctl@ceph.mon.b.smithi122.stdout: block_cache_compressed: (nil) 2024-04-14T13:51:51.038 INFO:journalctl@ceph.mon.b.smithi122.stdout: persistent_cache: (nil) 2024-04-14T13:51:51.038 INFO:journalctl@ceph.mon.b.smithi122.stdout: block_size: 4096 2024-04-14T13:51:51.038 INFO:journalctl@ceph.mon.b.smithi122.stdout: block_size_deviation: 10 2024-04-14T13:51:51.038 INFO:journalctl@ceph.mon.b.smithi122.stdout: block_restart_interval: 16 2024-04-14T13:51:51.038 INFO:journalctl@ceph.mon.b.smithi122.stdout: index_block_restart_interval: 1 2024-04-14T13:51:51.038 INFO:journalctl@ceph.mon.b.smithi122.stdout: metadata_block_size: 4096 2024-04-14T13:51:51.038 INFO:journalctl@ceph.mon.b.smithi122.stdout: partition_filters: 0 2024-04-14T13:51:51.038 INFO:journalctl@ceph.mon.b.smithi122.stdout: use_delta_encoding: 1 2024-04-14T13:51:51.038 INFO:journalctl@ceph.mon.b.smithi122.stdout: filter_policy: rocksdb.BuiltinBloomFilter 2024-04-14T13:51:51.038 INFO:journalctl@ceph.mon.b.smithi122.stdout: whole_key_filtering: 1 2024-04-14T13:51:51.038 INFO:journalctl@ceph.mon.b.smithi122.stdout: verify_compression: 0 2024-04-14T13:51:51.038 INFO:journalctl@ceph.mon.b.smithi122.stdout: read_amp_bytes_per_bit: 0 2024-04-14T13:51:51.038 INFO:journalctl@ceph.mon.b.smithi122.stdout: format_version: 4 2024-04-14T13:51:51.039 INFO:journalctl@ceph.mon.b.smithi122.stdout: enable_index_compression: 1 2024-04-14T13:51:51.039 INFO:journalctl@ceph.mon.b.smithi122.stdout: block_align: 0 2024-04-14T13:51:51.039 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:51:50 smithi122 ceph-mon[25721]: rocksdb: Options.write_buffer_size: 33554432 2024-04-14T13:51:51.039 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:51:50 smithi122 ceph-mon[25721]: rocksdb: Options.max_write_buffer_number: 2 2024-04-14T13:51:51.039 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:51:50 smithi122 ceph-mon[25721]: rocksdb: Options.compression: NoCompression 2024-04-14T13:51:51.039 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:51:50 smithi122 ceph-mon[25721]: rocksdb: Options.bottommost_compression: Disabled 2024-04-14T13:51:51.039 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:51:50 smithi122 ceph-mon[25721]: rocksdb: Options.prefix_extractor: nullptr 2024-04-14T13:51:51.039 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:51:50 smithi122 ceph-mon[25721]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-04-14T13:51:51.039 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:51:50 smithi122 ceph-mon[25721]: rocksdb: Options.num_levels: 7 2024-04-14T13:51:51.039 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:51:50 smithi122 ceph-mon[25721]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-04-14T13:51:51.039 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:51:50 smithi122 ceph-mon[25721]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-04-14T13:51:51.039 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:51:50 smithi122 ceph-mon[25721]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-04-14T13:51:51.039 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:51:50 smithi122 ceph-mon[25721]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-04-14T13:51:51.039 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:51:50 smithi122 ceph-mon[25721]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-04-14T13:51:51.039 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:51:50 smithi122 ceph-mon[25721]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-04-14T13:51:51.039 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:51:50 smithi122 ceph-mon[25721]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-04-14T13:51:51.040 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:51:50 smithi122 ceph-mon[25721]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-04-14T13:51:51.040 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:51:50 smithi122 ceph-mon[25721]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-04-14T13:51:51.040 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:51:50 smithi122 ceph-mon[25721]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-04-14T13:51:51.040 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:51:50 smithi122 ceph-mon[25721]: rocksdb: Options.compression_opts.window_bits: -14 2024-04-14T13:51:51.040 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:51:50 smithi122 ceph-mon[25721]: rocksdb: Options.compression_opts.level: 32767 2024-04-14T13:51:51.040 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:51:50 smithi122 ceph-mon[25721]: rocksdb: Options.compression_opts.strategy: 0 2024-04-14T13:51:51.040 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:51:50 smithi122 ceph-mon[25721]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-04-14T13:51:51.040 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:51:50 smithi122 ceph-mon[25721]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-04-14T13:51:51.040 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:51:50 smithi122 ceph-mon[25721]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-04-14T13:51:51.040 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:51:50 smithi122 ceph-mon[25721]: rocksdb: Options.compression_opts.enabled: false 2024-04-14T13:51:51.040 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:51:50 smithi122 ceph-mon[25721]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-04-14T13:51:51.040 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:51:50 smithi122 ceph-mon[25721]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-04-14T13:51:51.040 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:51:50 smithi122 ceph-mon[25721]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-04-14T13:51:51.040 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:51:50 smithi122 ceph-mon[25721]: rocksdb: Options.target_file_size_base: 67108864 2024-04-14T13:51:51.040 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:51:50 smithi122 ceph-mon[25721]: rocksdb: Options.target_file_size_multiplier: 1 2024-04-14T13:51:51.040 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:51:50 smithi122 ceph-mon[25721]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-04-14T13:51:51.040 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:51:50 smithi122 ceph-mon[25721]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-04-14T13:51:51.041 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:51:50 smithi122 ceph-mon[25721]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-04-14T13:51:51.041 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:51:50 smithi122 ceph-mon[25721]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-04-14T13:51:51.041 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:51:50 smithi122 ceph-mon[25721]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-04-14T13:51:51.041 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:51:50 smithi122 ceph-mon[25721]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-04-14T13:51:51.041 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:51:50 smithi122 ceph-mon[25721]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-04-14T13:51:51.041 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:51:50 smithi122 ceph-mon[25721]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-04-14T13:51:51.041 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:51:50 smithi122 ceph-mon[25721]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-04-14T13:51:51.041 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:51:50 smithi122 ceph-mon[25721]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-04-14T13:51:51.041 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:51:50 smithi122 ceph-mon[25721]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-04-14T13:51:51.041 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:51:50 smithi122 ceph-mon[25721]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-04-14T13:51:51.041 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:51:50 smithi122 ceph-mon[25721]: rocksdb: Options.arena_block_size: 4194304 2024-04-14T13:51:51.041 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:51:50 smithi122 ceph-mon[25721]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-04-14T13:51:51.041 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:51:50 smithi122 ceph-mon[25721]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-04-14T13:51:51.041 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:51:50 smithi122 ceph-mon[25721]: rocksdb: Options.rate_limit_delay_max_milliseconds: 100 2024-04-14T13:51:51.041 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:51:50 smithi122 ceph-mon[25721]: rocksdb: Options.disable_auto_compactions: 0 2024-04-14T13:51:51.042 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:51:50 smithi122 ceph-mon[25721]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-04-14T13:51:51.042 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:51:50 smithi122 ceph-mon[25721]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-04-14T13:51:51.042 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:51:50 smithi122 ceph-mon[25721]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-04-14T13:51:51.042 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:51:50 smithi122 ceph-mon[25721]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-04-14T13:51:51.042 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:51:50 smithi122 ceph-mon[25721]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-04-14T13:51:51.042 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:51:50 smithi122 ceph-mon[25721]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-04-14T13:51:51.042 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:51:50 smithi122 ceph-mon[25721]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-04-14T13:51:51.042 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:51:50 smithi122 ceph-mon[25721]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-04-14T13:51:51.042 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:51:50 smithi122 ceph-mon[25721]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-04-14T13:51:51.042 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:51:50 smithi122 ceph-mon[25721]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-04-14T13:51:51.042 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:51:50 smithi122 ceph-mon[25721]: rocksdb: Options.table_properties_collectors: 2024-04-14T13:51:51.042 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:51:50 smithi122 ceph-mon[25721]: rocksdb: Options.inplace_update_support: 0 2024-04-14T13:51:51.042 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:51:50 smithi122 ceph-mon[25721]: rocksdb: Options.inplace_update_num_locks: 10000 2024-04-14T13:51:51.042 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:51:50 smithi122 ceph-mon[25721]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-04-14T13:51:51.042 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:51:50 smithi122 ceph-mon[25721]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-04-14T13:51:51.042 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:51:50 smithi122 ceph-mon[25721]: rocksdb: Options.memtable_huge_page_size: 0 2024-04-14T13:51:51.043 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:51:50 smithi122 ceph-mon[25721]: rocksdb: Options.bloom_locality: 0 2024-04-14T13:51:51.043 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:51:50 smithi122 ceph-mon[25721]: rocksdb: Options.max_successive_merges: 0 2024-04-14T13:51:51.043 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:51:50 smithi122 ceph-mon[25721]: rocksdb: Options.optimize_filters_for_hits: 0 2024-04-14T13:51:51.043 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:51:50 smithi122 ceph-mon[25721]: rocksdb: Options.paranoid_file_checks: 0 2024-04-14T13:51:51.043 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:51:50 smithi122 ceph-mon[25721]: rocksdb: Options.force_consistency_checks: 1 2024-04-14T13:51:51.043 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:51:50 smithi122 ceph-mon[25721]: rocksdb: Options.report_bg_io_stats: 0 2024-04-14T13:51:51.043 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:51:50 smithi122 ceph-mon[25721]: rocksdb: Options.ttl: 2592000 2024-04-14T13:51:51.043 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:51:50 smithi122 ceph-mon[25721]: rocksdb: Options.periodic_compaction_seconds: 0 2024-04-14T13:51:51.043 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:51:50 smithi122 ceph-mon[25721]: rocksdb: Options.enable_blob_files: false 2024-04-14T13:51:51.043 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:51:50 smithi122 ceph-mon[25721]: rocksdb: Options.min_blob_size: 0 2024-04-14T13:51:51.043 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:51:50 smithi122 ceph-mon[25721]: rocksdb: Options.blob_file_size: 268435456 2024-04-14T13:51:51.043 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:51:50 smithi122 ceph-mon[25721]: rocksdb: Options.blob_compression_type: NoCompression 2024-04-14T13:51:51.043 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:51:50 smithi122 ceph-mon[25721]: rocksdb: Options.enable_blob_garbage_collection: false 2024-04-14T13:51:51.043 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:51:50 smithi122 ceph-mon[25721]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-04-14T13:51:51.043 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:51:50 smithi122 ceph-mon[25721]: rocksdb: [db/version_set.cc:4773] Recovered from manifest file:/var/lib/ceph/mon/ceph-b/store.db/MANIFEST-000003 succeeded,manifest_file_number is 3, next_file_number is 5, last_sequence is 0, log_number is 0,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2024-04-14T13:51:51.043 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:51:50 smithi122 ceph-mon[25721]: rocksdb: [db/version_set.cc:4782] Column family [default] (ID 0), log number is 0 2024-04-14T13:51:51.043 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:51:50 smithi122 ceph-mon[25721]: rocksdb: [db/version_set.cc:4083] Creating manifest 7 2024-04-14T13:51:51.044 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:51:50 smithi122 ceph-mon[25721]: rocksdb: EVENT_LOG_v1 {"time_micros": 1713102710944613, "job": 1, "event": "recovery_started", "wal_files": [4]} 2024-04-14T13:51:51.044 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:51:50 smithi122 ceph-mon[25721]: rocksdb: [db/db_impl/db_impl_open.cc:847] Recovering log #4 mode 2 2024-04-14T13:51:51.044 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:51:50 smithi122 ceph-mon[25721]: rocksdb: [table/block_based/filter_policy.cc:996] Using legacy Bloom filter with high (20) bits/key. Dramatic filter space and/or accuracy improvement is available with format_version>=5. 2024-04-14T13:51:51.044 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:51:50 smithi122 ceph-mon[25721]: rocksdb: EVENT_LOG_v1 {"time_micros": 1713102710944910, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 8, "file_size": 1670, "file_checksum": "", "file_checksum_func_name": "Unknown", "table_properties": {"data_size": 648, "index_size": 31, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 69, "raw_key_size": 115, "raw_average_key_size": 23, "raw_value_size": 526, "raw_average_value_size": 105, "num_data_blocks": 1, "num_entries": 5, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "rocksdb.BuiltinBloomFilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; ", "creation_time": 1713102710, "oldest_key_time": 0, "file_creation_time": 0, "db_id": "1068778e-f7d8-49e5-b49e-6136684706ae", "db_session_id": "IUSIJZFA3K6LSW8A0IC9"}} 2024-04-14T13:51:51.044 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:51:50 smithi122 ceph-mon[25721]: rocksdb: [db/version_set.cc:4083] Creating manifest 9 2024-04-14T13:51:51.044 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:51:50 smithi122 ceph-mon[25721]: rocksdb: EVENT_LOG_v1 {"time_micros": 1713102710945224, "job": 1, "event": "recovery_finished"} 2024-04-14T13:51:51.044 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:51:50 smithi122 ceph-mon[25721]: rocksdb: [file/delete_scheduler.cc:73] Deleted file /var/lib/ceph/mon/ceph-b/store.db/000004.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-04-14T13:51:51.044 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:51:50 smithi122 ceph-mon[25721]: rocksdb: [db/db_impl/db_impl_open.cc:1701] SstFileManager instance 0x55c2a4246a80 2024-04-14T13:51:51.044 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:51:50 smithi122 ceph-mon[25721]: rocksdb: DB pointer 0x55c2a4256000 2024-04-14T13:51:51.044 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:51:50 smithi122 ceph-mon[25721]: rocksdb: [db/db_impl/db_impl.cc:902] ------- DUMPING STATS ------- 2024-04-14T13:51:51.044 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:51:50 smithi122 ceph-mon[25721]: rocksdb: [db/db_impl/db_impl.cc:903] 2024-04-14T13:51:51.044 INFO:journalctl@ceph.mon.b.smithi122.stdout: ** DB Stats ** 2024-04-14T13:51:51.044 INFO:journalctl@ceph.mon.b.smithi122.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-04-14T13:51:51.044 INFO:journalctl@ceph.mon.b.smithi122.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-04-14T13:51:51.044 INFO:journalctl@ceph.mon.b.smithi122.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-04-14T13:51:51.044 INFO:journalctl@ceph.mon.b.smithi122.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-04-14T13:51:51.045 INFO:journalctl@ceph.mon.b.smithi122.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-04-14T13:51:51.045 INFO:journalctl@ceph.mon.b.smithi122.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 MB, 0.00 MB/s 2024-04-14T13:51:51.045 INFO:journalctl@ceph.mon.b.smithi122.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-04-14T13:51:51.045 INFO:journalctl@ceph.mon.b.smithi122.stdout: 2024-04-14T13:51:51.045 INFO:journalctl@ceph.mon.b.smithi122.stdout: ** Compaction Stats [default] ** 2024-04-14T13:51:51.045 INFO:journalctl@ceph.mon.b.smithi122.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-04-14T13:51:51.045 INFO:journalctl@ceph.mon.b.smithi122.stdout: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-04-14T13:51:51.045 INFO:journalctl@ceph.mon.b.smithi122.stdout: L0 1/0 1.63 KB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 5.6 0.00 0.00 1 0.000 0 0 2024-04-14T13:51:51.045 INFO:journalctl@ceph.mon.b.smithi122.stdout: Sum 1/0 1.63 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 5.6 0.00 0.00 1 0.000 0 0 2024-04-14T13:51:51.045 INFO:journalctl@ceph.mon.b.smithi122.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 5.6 0.00 0.00 1 0.000 0 0 2024-04-14T13:51:51.045 INFO:journalctl@ceph.mon.b.smithi122.stdout: 2024-04-14T13:51:51.045 INFO:journalctl@ceph.mon.b.smithi122.stdout: ** Compaction Stats [default] ** 2024-04-14T13:51:51.045 INFO:journalctl@ceph.mon.b.smithi122.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-04-14T13:51:51.045 INFO:journalctl@ceph.mon.b.smithi122.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-04-14T13:51:51.045 INFO:journalctl@ceph.mon.b.smithi122.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 5.6 0.00 0.00 1 0.000 0 0 2024-04-14T13:51:51.046 INFO:journalctl@ceph.mon.b.smithi122.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-04-14T13:51:51.046 INFO:journalctl@ceph.mon.b.smithi122.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-04-14T13:51:51.046 INFO:journalctl@ceph.mon.b.smithi122.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-04-14T13:51:51.046 INFO:journalctl@ceph.mon.b.smithi122.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-04-14T13:51:51.046 INFO:journalctl@ceph.mon.b.smithi122.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-04-14T13:51:51.046 INFO:journalctl@ceph.mon.b.smithi122.stdout: AddFile(Keys): cumulative 0, interval 0 2024-04-14T13:51:51.046 INFO:journalctl@ceph.mon.b.smithi122.stdout: Cumulative compaction: 0.00 GB write, 0.61 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-04-14T13:51:51.046 INFO:journalctl@ceph.mon.b.smithi122.stdout: Interval compaction: 0.00 GB write, 0.61 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-04-14T13:51:51.046 INFO:journalctl@ceph.mon.b.smithi122.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-04-14T13:51:51.046 INFO:journalctl@ceph.mon.b.smithi122.stdout: 2024-04-14T13:51:51.046 INFO:journalctl@ceph.mon.b.smithi122.stdout: ** File Read Latency Histogram By Level [default] ** 2024-04-14T13:51:51.046 INFO:journalctl@ceph.mon.b.smithi122.stdout: 2024-04-14T13:51:51.046 INFO:journalctl@ceph.mon.b.smithi122.stdout: ** Compaction Stats [default] ** 2024-04-14T13:51:51.046 INFO:journalctl@ceph.mon.b.smithi122.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-04-14T13:51:51.046 INFO:journalctl@ceph.mon.b.smithi122.stdout: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-04-14T13:51:51.046 INFO:journalctl@ceph.mon.b.smithi122.stdout: L0 1/0 1.63 KB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 5.6 0.00 0.00 1 0.000 0 0 2024-04-14T13:51:51.047 INFO:journalctl@ceph.mon.b.smithi122.stdout: Sum 1/0 1.63 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 5.6 0.00 0.00 1 0.000 0 0 2024-04-14T13:51:51.047 INFO:journalctl@ceph.mon.b.smithi122.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2024-04-14T13:51:51.047 INFO:journalctl@ceph.mon.b.smithi122.stdout: 2024-04-14T13:51:51.047 INFO:journalctl@ceph.mon.b.smithi122.stdout: ** Compaction Stats [default] ** 2024-04-14T13:51:51.047 INFO:journalctl@ceph.mon.b.smithi122.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-04-14T13:51:51.047 INFO:journalctl@ceph.mon.b.smithi122.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-04-14T13:51:51.047 INFO:journalctl@ceph.mon.b.smithi122.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 5.6 0.00 0.00 1 0.000 0 0 2024-04-14T13:51:51.047 INFO:journalctl@ceph.mon.b.smithi122.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-04-14T13:51:51.047 INFO:journalctl@ceph.mon.b.smithi122.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-04-14T13:51:51.047 INFO:journalctl@ceph.mon.b.smithi122.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-04-14T13:51:51.047 INFO:journalctl@ceph.mon.b.smithi122.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-04-14T13:51:51.047 INFO:journalctl@ceph.mon.b.smithi122.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-04-14T13:51:51.047 INFO:journalctl@ceph.mon.b.smithi122.stdout: AddFile(Keys): cumulative 0, interval 0 2024-04-14T13:51:51.047 INFO:journalctl@ceph.mon.b.smithi122.stdout: Cumulative compaction: 0.00 GB write, 0.60 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-04-14T13:51:51.047 INFO:journalctl@ceph.mon.b.smithi122.stdout: Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-04-14T13:51:51.047 INFO:journalctl@ceph.mon.b.smithi122.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-04-14T13:51:51.048 INFO:journalctl@ceph.mon.b.smithi122.stdout: 2024-04-14T13:51:51.048 INFO:journalctl@ceph.mon.b.smithi122.stdout: ** File Read Latency Histogram By Level [default] ** 2024-04-14T13:51:51.048 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:51:50 smithi122 ceph-mon[25721]: mon.b does not exist in monmap, will attempt to join an existing cluster 2024-04-14T13:51:51.048 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:51:50 smithi122 ceph-mon[25721]: using public_addr v2:172.21.15.122:0/0 -> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] 2024-04-14T13:51:51.048 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:51:50 smithi122 ceph-mon[25721]: starting mon.b rank -1 at public addrs [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] at bind addrs [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] mon_data /var/lib/ceph/mon/ceph-b fsid dc326fc2-fa65-11ee-bc8e-c7b262605968 2024-04-14T13:51:51.048 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:51:50 smithi122 ceph-mon[25721]: mon.b@-1(???) e0 preinit fsid dc326fc2-fa65-11ee-bc8e-c7b262605968 2024-04-14T13:51:51.048 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:51:50 smithi122 ceph-mon[25721]: mon.b@-1(synchronizing).mds e1 new map 2024-04-14T13:51:51.048 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:51:50 smithi122 ceph-mon[25721]: mon.b@-1(synchronizing).mds e1 print_map 2024-04-14T13:51:51.048 INFO:journalctl@ceph.mon.b.smithi122.stdout: e1 2024-04-14T13:51:51.048 INFO:journalctl@ceph.mon.b.smithi122.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-04-14T13:51:51.048 INFO:journalctl@ceph.mon.b.smithi122.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-14T13:51:51.048 INFO:journalctl@ceph.mon.b.smithi122.stdout: legacy client fscid: -1 2024-04-14T13:51:51.048 INFO:journalctl@ceph.mon.b.smithi122.stdout: 2024-04-14T13:51:51.048 INFO:journalctl@ceph.mon.b.smithi122.stdout: No filesystems configured 2024-04-14T13:51:51.048 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:51:50 smithi122 ceph-mon[25721]: mon.b@-1(synchronizing).osd e0 _set_cache_ratios kv ratio 0.25 inc ratio 0.375 full ratio 0.375 2024-04-14T13:51:51.048 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:51:50 smithi122 ceph-mon[25721]: mon.b@-1(synchronizing).osd e0 register_cache_with_pcm pcm target: 2147483648 pcm max: 1020054732 pcm min: 134217728 inc_osd_cache size: 1 2024-04-14T13:51:51.049 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:51:50 smithi122 ceph-mon[25721]: mon.b@-1(synchronizing).osd e1 e1: 0 total, 0 up, 0 in 2024-04-14T13:51:51.049 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:51:50 smithi122 ceph-mon[25721]: mon.b@-1(synchronizing).osd e2 e2: 0 total, 0 up, 0 in 2024-04-14T13:51:51.049 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:51:50 smithi122 ceph-mon[25721]: mon.b@-1(synchronizing).osd e3 e3: 0 total, 0 up, 0 in 2024-04-14T13:51:51.049 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:51:50 smithi122 ceph-mon[25721]: mon.b@-1(synchronizing).osd e4 e4: 0 total, 0 up, 0 in 2024-04-14T13:51:51.049 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:51:50 smithi122 ceph-mon[25721]: mon.b@-1(synchronizing).osd e4 crush map has features 3314932999778484224, adjusting msgr requires 2024-04-14T13:51:51.049 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:51:50 smithi122 ceph-mon[25721]: mon.b@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-04-14T13:51:51.049 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:51:50 smithi122 ceph-mon[25721]: mon.b@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-04-14T13:51:51.049 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:51:50 smithi122 ceph-mon[25721]: mon.b@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-04-14T13:51:51.049 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:51:50 smithi122 ceph-mon[25721]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-14T13:51:51.049 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:51:50 smithi122 ceph-mon[25721]: from='client.14180 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi069:172.21.15.69=a;smithi069:[v2:172.21.15.69:3301,v1:172.21.15.69:6790]=c;smithi122:172.21.15.122=b", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T13:51:51.049 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:51:50 smithi122 ceph-mon[25721]: Saving service mon spec with placement smithi069:172.21.15.69=a;smithi069:[v2:172.21.15.69:3301,v1:172.21.15.69:6790]=c;smithi122:172.21.15.122=b;count:3 2024-04-14T13:51:51.049 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:51:50 smithi122 ceph-mon[25721]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' 2024-04-14T13:51:51.049 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:51:50 smithi122 ceph-mon[25721]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-14T13:51:51.049 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:51:50 smithi122 ceph-mon[25721]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T13:51:51.049 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:51:50 smithi122 ceph-mon[25721]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-14T13:51:51.050 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:51:50 smithi122 ceph-mon[25721]: Updating smithi122:/etc/ceph/ceph.conf 2024-04-14T13:51:51.050 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:51:50 smithi122 ceph-mon[25721]: Updating smithi122:/etc/ceph/ceph.client.admin.keyring 2024-04-14T13:51:51.050 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:51:50 smithi122 ceph-mon[25721]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' 2024-04-14T13:51:51.050 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:51:50 smithi122 ceph-mon[25721]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' 2024-04-14T13:51:51.050 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:51:50 smithi122 ceph-mon[25721]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-14T13:51:51.050 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:51:50 smithi122 ceph-mon[25721]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T13:51:51.050 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:51:50 smithi122 ceph-mon[25721]: Deploying daemon mon.c on smithi069 2024-04-14T13:51:51.050 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:51:50 smithi122 ceph-mon[25721]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-14T13:51:51.050 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:51:50 smithi122 ceph-mon[25721]: from='client.? 172.21.15.122:0/3564277494' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-14T13:51:51.050 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:51:50 smithi122 ceph-mon[25721]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-14T13:51:51.050 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:51:50 smithi122 ceph-mon[25721]: mon.a calling monitor election 2024-04-14T13:51:51.050 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:51:50 smithi122 ceph-mon[25721]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-14T13:51:51.050 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:51:50 smithi122 ceph-mon[25721]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-14T13:51:51.050 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:51:50 smithi122 ceph-mon[25721]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-14T13:51:51.050 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:51:50 smithi122 ceph-mon[25721]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-14T13:51:51.051 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:51:50 smithi122 ceph-mon[25721]: mon.c calling monitor election 2024-04-14T13:51:51.051 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:51:50 smithi122 ceph-mon[25721]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-14T13:51:51.051 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:51:50 smithi122 ceph-mon[25721]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-14T13:51:51.051 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:51:50 smithi122 ceph-mon[25721]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-14T13:51:51.051 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:51:50 smithi122 ceph-mon[25721]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-14T13:51:51.051 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:51:50 smithi122 ceph-mon[25721]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-04-14T13:51:51.051 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:51:50 smithi122 ceph-mon[25721]: monmap e2: 2 mons at {a=[v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0],c=[v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0]} 2024-04-14T13:51:51.051 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:51:50 smithi122 ceph-mon[25721]: fsmap 2024-04-14T13:51:51.051 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:51:50 smithi122 ceph-mon[25721]: osdmap e4: 0 total, 0 up, 0 in 2024-04-14T13:51:51.051 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:51:50 smithi122 ceph-mon[25721]: mgrmap e13: y(active, since 34s) 2024-04-14T13:51:51.051 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:51:50 smithi122 ceph-mon[25721]: overall HEALTH_OK 2024-04-14T13:51:51.051 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:51:50 smithi122 ceph-mon[25721]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' 2024-04-14T13:51:51.051 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:51:50 smithi122 ceph-mon[25721]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-14T13:51:51.051 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:51:50 smithi122 ceph-mon[25721]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T13:51:51.051 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:51:50 smithi122 ceph-mon[25721]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-14T13:51:51.051 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:51:50 smithi122 ceph-mon[25721]: Deploying daemon mon.b on smithi122 2024-04-14T13:51:51.051 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:51:50 smithi122 ceph-mon[25721]: from='client.? 172.21.15.122:0/2873611874' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-14T13:51:51.052 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:51:50 smithi122 ceph-mon[25721]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-14T13:51:51.052 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:51:50 smithi122 ceph-mon[25721]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-14T13:51:51.052 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:51:50 smithi122 ceph-mon[25721]: mon.b@-1(synchronizing).paxosservice(auth 1..3) refresh upgraded, format 0 -> 3 2024-04-14T13:51:51.052 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:51:50 smithi122 ceph-mon[25721]: mon.b@-1(synchronizing) e2 handle_conf_change mon_allow_pool_delete,mon_cluster_log_to_file 2024-04-14T13:51:51.052 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:51:51 smithi122 bash[25680]: 3b766bfce4be5663d61ecbbff3cf9815245b7c7a653cd757cf48d86884426543 2024-04-14T13:51:51.052 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:51:51 smithi122 systemd[1]: Started Ceph mon.b for dc326fc2-fa65-11ee-bc8e-c7b262605968. 2024-04-14T13:51:52.577 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-04-14T13:51:52.577 DEBUG:teuthology.orchestra.run.smithi122:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid dc326fc2-fa65-11ee-bc8e-c7b262605968 -- ceph mon dump -f json 2024-04-14T13:51:56.318 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:51:56 smithi069 ceph-mon[29472]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-14T13:51:56.318 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:51:56 smithi069 ceph-mon[29472]: mon.a calling monitor election 2024-04-14T13:51:56.318 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:51:56 smithi069 ceph-mon[29472]: mon.c calling monitor election 2024-04-14T13:51:56.318 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:51:56 smithi069 ceph-mon[29472]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-14T13:51:56.318 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:51:56 smithi069 ceph-mon[29472]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-14T13:51:56.318 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:51:56 smithi069 ceph-mon[29472]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-14T13:51:56.318 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:51:56 smithi069 ceph-mon[29472]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-14T13:51:56.318 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:51:56 smithi069 ceph-mon[29472]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-14T13:51:56.318 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:51:56 smithi069 ceph-mon[29472]: mon.b calling monitor election 2024-04-14T13:51:56.319 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:51:56 smithi069 ceph-mon[29472]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-14T13:51:56.319 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:51:56 smithi069 ceph-mon[29472]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-14T13:51:56.319 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:51:56 smithi069 ceph-mon[29472]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-14T13:51:56.319 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:51:56 smithi069 ceph-mon[29472]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-14T13:51:56.319 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:51:56 smithi069 ceph-mon[29472]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-04-14T13:51:56.319 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:51:56 smithi069 ceph-mon[29472]: monmap e3: 3 mons at {a=[v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0],b=[v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0],c=[v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0]} 2024-04-14T13:51:56.319 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:51:56 smithi069 ceph-mon[29472]: fsmap 2024-04-14T13:51:56.319 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:51:56 smithi069 ceph-mon[29472]: osdmap e4: 0 total, 0 up, 0 in 2024-04-14T13:51:56.319 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:51:56 smithi069 ceph-mon[29472]: mgrmap e13: y(active, since 44s) 2024-04-14T13:51:56.319 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:51:56 smithi069 ceph-mon[29472]: overall HEALTH_OK 2024-04-14T13:51:56.319 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:51:56 smithi069 ceph-mon[29472]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' 2024-04-14T13:51:56.319 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:51:56 smithi069 ceph-mon[29472]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-14T13:51:56.319 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:51:56 smithi069 ceph-mon[29472]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T13:51:56.320 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:51:56 smithi069 ceph-mon[29472]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-14T13:51:56.320 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:51:56 smithi069 ceph-mon[24818]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-14T13:51:56.320 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:51:56 smithi069 ceph-mon[24818]: mon.a calling monitor election 2024-04-14T13:51:56.320 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:51:56 smithi069 ceph-mon[24818]: mon.c calling monitor election 2024-04-14T13:51:56.320 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:51:56 smithi069 ceph-mon[24818]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-14T13:51:56.320 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:51:56 smithi069 ceph-mon[24818]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-14T13:51:56.320 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:51:56 smithi069 ceph-mon[24818]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-14T13:51:56.320 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:51:56 smithi069 ceph-mon[24818]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-14T13:51:56.320 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:51:56 smithi069 ceph-mon[24818]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-14T13:51:56.320 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:51:56 smithi069 ceph-mon[24818]: mon.b calling monitor election 2024-04-14T13:51:56.321 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:51:56 smithi069 ceph-mon[24818]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-14T13:51:56.321 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:51:56 smithi069 ceph-mon[24818]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-14T13:51:56.321 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:51:56 smithi069 ceph-mon[24818]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-14T13:51:56.321 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:51:56 smithi069 ceph-mon[24818]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-14T13:51:56.321 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:51:56 smithi069 ceph-mon[24818]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-04-14T13:51:56.321 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:51:56 smithi069 ceph-mon[24818]: monmap e3: 3 mons at {a=[v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0],b=[v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0],c=[v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0]} 2024-04-14T13:51:56.321 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:51:56 smithi069 ceph-mon[24818]: fsmap 2024-04-14T13:51:56.321 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:51:56 smithi069 ceph-mon[24818]: osdmap e4: 0 total, 0 up, 0 in 2024-04-14T13:51:56.321 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:51:56 smithi069 ceph-mon[24818]: mgrmap e13: y(active, since 44s) 2024-04-14T13:51:56.321 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:51:56 smithi069 ceph-mon[24818]: overall HEALTH_OK 2024-04-14T13:51:56.321 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:51:56 smithi069 ceph-mon[24818]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' 2024-04-14T13:51:56.321 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:51:56 smithi069 ceph-mon[24818]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-14T13:51:56.321 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:51:56 smithi069 ceph-mon[24818]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T13:51:56.322 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:51:56 smithi069 ceph-mon[24818]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-14T13:51:56.346 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:51:56 smithi122 ceph-mon[25721]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-14T13:51:56.346 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:51:56 smithi122 ceph-mon[25721]: mon.a calling monitor election 2024-04-14T13:51:56.346 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:51:56 smithi122 ceph-mon[25721]: mon.c calling monitor election 2024-04-14T13:51:56.346 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:51:56 smithi122 ceph-mon[25721]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-14T13:51:56.346 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:51:56 smithi122 ceph-mon[25721]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-14T13:51:56.346 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:51:56 smithi122 ceph-mon[25721]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-14T13:51:56.346 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:51:56 smithi122 ceph-mon[25721]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-14T13:51:56.346 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:51:56 smithi122 ceph-mon[25721]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-14T13:51:56.346 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:51:56 smithi122 ceph-mon[25721]: mon.b calling monitor election 2024-04-14T13:51:56.346 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:51:56 smithi122 ceph-mon[25721]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-14T13:51:56.347 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:51:56 smithi122 ceph-mon[25721]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-14T13:51:56.347 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:51:56 smithi122 ceph-mon[25721]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-14T13:51:56.347 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:51:56 smithi122 ceph-mon[25721]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-14T13:51:56.347 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:51:56 smithi122 ceph-mon[25721]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-04-14T13:51:56.347 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:51:56 smithi122 ceph-mon[25721]: monmap e3: 3 mons at {a=[v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0],b=[v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0],c=[v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0]} 2024-04-14T13:51:56.347 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:51:56 smithi122 ceph-mon[25721]: fsmap 2024-04-14T13:51:56.347 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:51:56 smithi122 ceph-mon[25721]: osdmap e4: 0 total, 0 up, 0 in 2024-04-14T13:51:56.347 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:51:56 smithi122 ceph-mon[25721]: mgrmap e13: y(active, since 44s) 2024-04-14T13:51:56.347 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:51:56 smithi122 ceph-mon[25721]: overall HEALTH_OK 2024-04-14T13:51:56.347 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:51:56 smithi122 ceph-mon[25721]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' 2024-04-14T13:51:56.347 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:51:56 smithi122 ceph-mon[25721]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-14T13:51:56.347 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:51:56 smithi122 ceph-mon[25721]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T13:51:56.347 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:51:56 smithi122 ceph-mon[25721]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-14T13:51:57.346 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:51:57 smithi122 ceph-mon[25721]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-14T13:51:57.346 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:51:57 smithi122 ceph-mon[25721]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' 2024-04-14T13:51:57.346 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:51:57 smithi122 ceph-mon[25721]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' 2024-04-14T13:51:57.346 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:51:57 smithi122 ceph-mon[25721]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' 2024-04-14T13:51:57.346 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:51:57 smithi122 ceph-mon[25721]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-14T13:51:57.346 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:51:57 smithi122 ceph-mon[25721]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' 2024-04-14T13:51:57.346 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:51:57 smithi122 ceph-mon[25721]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' 2024-04-14T13:51:57.452 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-04-14T13:51:57.452 INFO:teuthology.orchestra.run.smithi122.stdout:{"epoch":3,"fsid":"dc326fc2-fa65-11ee-bc8e-c7b262605968","modified":"2024-04-14T13:51:50.965882Z","created":"2024-04-14T13:50:18.166086Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:3300","nonce":0},{"type":"v1","addr":"172.21.15.69:6789","nonce":0}]},"addr":"172.21.15.69:6789/0","public_addr":"172.21.15.69:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:3301","nonce":0},{"type":"v1","addr":"172.21.15.69:6790","nonce":0}]},"addr":"172.21.15.69:6790/0","public_addr":"172.21.15.69:6790/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:3300","nonce":0},{"type":"v1","addr":"172.21.15.122:6789","nonce":0}]},"addr":"172.21.15.122:6789/0","public_addr":"172.21.15.122:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1,2]} 2024-04-14T13:51:57.453 INFO:teuthology.orchestra.run.smithi122.stderr:dumped monmap epoch 3 2024-04-14T13:51:57.518 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:51:57 smithi069 ceph-mon[24818]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-14T13:51:57.518 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:51:57 smithi069 ceph-mon[24818]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' 2024-04-14T13:51:57.518 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:51:57 smithi069 ceph-mon[24818]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' 2024-04-14T13:51:57.518 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:51:57 smithi069 ceph-mon[24818]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' 2024-04-14T13:51:57.518 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:51:57 smithi069 ceph-mon[24818]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-14T13:51:57.518 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:51:57 smithi069 ceph-mon[24818]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' 2024-04-14T13:51:57.518 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:51:57 smithi069 ceph-mon[24818]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' 2024-04-14T13:51:57.518 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:51:57 smithi069 ceph-mon[29472]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-14T13:51:57.518 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:51:57 smithi069 ceph-mon[29472]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' 2024-04-14T13:51:57.519 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:51:57 smithi069 ceph-mon[29472]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' 2024-04-14T13:51:57.519 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:51:57 smithi069 ceph-mon[29472]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' 2024-04-14T13:51:57.519 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:51:57 smithi069 ceph-mon[29472]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-14T13:51:57.519 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:51:57 smithi069 ceph-mon[29472]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' 2024-04-14T13:51:57.519 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:51:57 smithi069 ceph-mon[29472]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' 2024-04-14T13:51:58.043 INFO:tasks.cephadm:Generating final ceph.conf file... 2024-04-14T13:51:58.043 DEBUG:teuthology.orchestra.run.smithi069:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid dc326fc2-fa65-11ee-bc8e-c7b262605968 -- ceph config generate-minimal-conf 2024-04-14T13:51:58.345 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:51:58 smithi122 ceph-mon[25721]: Updating smithi122:/etc/ceph/ceph.conf 2024-04-14T13:51:58.345 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:51:58 smithi122 ceph-mon[25721]: Updating smithi069:/etc/ceph/ceph.conf 2024-04-14T13:51:58.345 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:51:58 smithi122 ceph-mon[25721]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' 2024-04-14T13:51:58.346 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:51:58 smithi122 ceph-mon[25721]: Reconfiguring mon.a (unknown last config time)... 2024-04-14T13:51:58.346 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:51:58 smithi122 ceph-mon[25721]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-14T13:51:58.346 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:51:58 smithi122 ceph-mon[25721]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-14T13:51:58.346 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:51:58 smithi122 ceph-mon[25721]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T13:51:58.346 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:51:58 smithi122 ceph-mon[25721]: Reconfiguring daemon mon.a on smithi069 2024-04-14T13:51:58.346 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:51:58 smithi122 ceph-mon[25721]: from='client.? 172.21.15.122:0/3615017858' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-14T13:51:58.462 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:51:58 smithi069 ceph-mon[29472]: Updating smithi122:/etc/ceph/ceph.conf 2024-04-14T13:51:58.462 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:51:58 smithi069 ceph-mon[29472]: Updating smithi069:/etc/ceph/ceph.conf 2024-04-14T13:51:58.462 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:51:58 smithi069 ceph-mon[29472]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' 2024-04-14T13:51:58.462 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:51:58 smithi069 ceph-mon[29472]: Reconfiguring mon.a (unknown last config time)... 2024-04-14T13:51:58.462 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:51:58 smithi069 ceph-mon[29472]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-14T13:51:58.462 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:51:58 smithi069 ceph-mon[29472]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-14T13:51:58.462 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:51:58 smithi069 ceph-mon[29472]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T13:51:58.463 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:51:58 smithi069 ceph-mon[29472]: Reconfiguring daemon mon.a on smithi069 2024-04-14T13:51:58.463 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:51:58 smithi069 ceph-mon[29472]: from='client.? 172.21.15.122:0/3615017858' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-14T13:51:58.463 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:51:58 smithi069 ceph-mon[24818]: Updating smithi122:/etc/ceph/ceph.conf 2024-04-14T13:51:58.463 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:51:58 smithi069 ceph-mon[24818]: Updating smithi069:/etc/ceph/ceph.conf 2024-04-14T13:51:58.463 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:51:58 smithi069 ceph-mon[24818]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' 2024-04-14T13:51:58.463 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:51:58 smithi069 ceph-mon[24818]: Reconfiguring mon.a (unknown last config time)... 2024-04-14T13:51:58.463 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:51:58 smithi069 ceph-mon[24818]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-14T13:51:58.463 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:51:58 smithi069 ceph-mon[24818]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-14T13:51:58.463 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:51:58 smithi069 ceph-mon[24818]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T13:51:58.463 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:51:58 smithi069 ceph-mon[24818]: Reconfiguring daemon mon.a on smithi069 2024-04-14T13:51:58.463 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:51:58 smithi069 ceph-mon[24818]: from='client.? 172.21.15.122:0/3615017858' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-14T13:51:59.420 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:51:59 smithi069 ceph-mon[24818]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-14T13:51:59.420 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:51:59 smithi069 ceph-mon[24818]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' 2024-04-14T13:51:59.420 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:51:59 smithi069 ceph-mon[24818]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-14T13:51:59.420 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:51:59 smithi069 ceph-mon[24818]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-14T13:51:59.420 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:51:59 smithi069 ceph-mon[24818]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T13:51:59.421 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:51:59 smithi069 ceph-mon[29472]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-14T13:51:59.421 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:51:59 smithi069 ceph-mon[29472]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' 2024-04-14T13:51:59.421 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:51:59 smithi069 ceph-mon[29472]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-14T13:51:59.421 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:51:59 smithi069 ceph-mon[29472]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-14T13:51:59.421 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:51:59 smithi069 ceph-mon[29472]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T13:51:59.595 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:51:59 smithi122 ceph-mon[25721]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-14T13:51:59.595 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:51:59 smithi122 ceph-mon[25721]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' 2024-04-14T13:51:59.595 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:51:59 smithi122 ceph-mon[25721]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-14T13:51:59.596 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:51:59 smithi122 ceph-mon[25721]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-14T13:51:59.596 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:51:59 smithi122 ceph-mon[25721]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T13:52:00.348 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:52:00 smithi069 ceph-mon[24818]: Reconfiguring mon.c (monmap changed)... 2024-04-14T13:52:00.349 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:52:00 smithi069 ceph-mon[24818]: Reconfiguring daemon mon.c on smithi069 2024-04-14T13:52:00.349 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:52:00 smithi069 ceph-mon[29472]: Reconfiguring mon.c (monmap changed)... 2024-04-14T13:52:00.349 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:52:00 smithi069 ceph-mon[29472]: Reconfiguring daemon mon.c on smithi069 2024-04-14T13:52:00.595 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:52:00 smithi122 ceph-mon[25721]: Reconfiguring mon.c (monmap changed)... 2024-04-14T13:52:00.595 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:52:00 smithi122 ceph-mon[25721]: Reconfiguring daemon mon.c on smithi069 2024-04-14T13:52:00.713 INFO:teuthology.orchestra.run.smithi069.stdout:# minimal ceph.conf for dc326fc2-fa65-11ee-bc8e-c7b262605968 2024-04-14T13:52:00.714 INFO:teuthology.orchestra.run.smithi069.stdout:[global] 2024-04-14T13:52:00.714 INFO:teuthology.orchestra.run.smithi069.stdout: fsid = dc326fc2-fa65-11ee-bc8e-c7b262605968 2024-04-14T13:52:00.714 INFO:teuthology.orchestra.run.smithi069.stdout: mon_host = [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] 2024-04-14T13:52:01.306 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:52:01 smithi122 ceph-mon[25721]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-14T13:52:01.306 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:52:01 smithi122 ceph-mon[25721]: from='client.? 172.21.15.69:0/3352126517' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T13:52:01.306 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:52:01 smithi122 ceph-mon[25721]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' 2024-04-14T13:52:01.306 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:52:01 smithi122 ceph-mon[25721]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-14T13:52:01.306 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:52:01 smithi122 ceph-mon[25721]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-14T13:52:01.306 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:52:01 smithi122 ceph-mon[25721]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T13:52:01.505 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:52:01 smithi069 ceph-mon[29472]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-14T13:52:01.505 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:52:01 smithi069 ceph-mon[29472]: from='client.? 172.21.15.69:0/3352126517' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T13:52:01.505 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:52:01 smithi069 ceph-mon[29472]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' 2024-04-14T13:52:01.506 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:52:01 smithi069 ceph-mon[29472]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-14T13:52:01.506 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:52:01 smithi069 ceph-mon[29472]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-14T13:52:01.506 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:52:01 smithi069 ceph-mon[29472]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T13:52:01.506 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:52:01 smithi069 ceph-mon[24818]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-14T13:52:01.506 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:52:01 smithi069 ceph-mon[24818]: from='client.? 172.21.15.69:0/3352126517' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T13:52:01.506 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:52:01 smithi069 ceph-mon[24818]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' 2024-04-14T13:52:01.506 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:52:01 smithi069 ceph-mon[24818]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-14T13:52:01.506 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:52:01 smithi069 ceph-mon[24818]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-14T13:52:01.506 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:52:01 smithi069 ceph-mon[24818]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T13:52:01.528 INFO:tasks.cephadm:Distributing (final) config and client.admin keyring... 2024-04-14T13:52:01.528 DEBUG:teuthology.orchestra.run.smithi069:> set -ex 2024-04-14T13:52:01.528 DEBUG:teuthology.orchestra.run.smithi069:> sudo dd of=/etc/ceph/ceph.conf 2024-04-14T13:52:01.558 DEBUG:teuthology.orchestra.run.smithi069:> set -ex 2024-04-14T13:52:01.559 DEBUG:teuthology.orchestra.run.smithi069:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-04-14T13:52:01.627 DEBUG:teuthology.orchestra.run.smithi122:> set -ex 2024-04-14T13:52:01.627 DEBUG:teuthology.orchestra.run.smithi122:> sudo dd of=/etc/ceph/ceph.conf 2024-04-14T13:52:01.655 DEBUG:teuthology.orchestra.run.smithi122:> set -ex 2024-04-14T13:52:01.655 DEBUG:teuthology.orchestra.run.smithi122:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-04-14T13:52:01.723 INFO:tasks.cephadm:Adding mgr.y on smithi069 2024-04-14T13:52:01.723 INFO:tasks.cephadm:Adding mgr.x on smithi122 2024-04-14T13:52:01.723 DEBUG:teuthology.orchestra.run.smithi122:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid dc326fc2-fa65-11ee-bc8e-c7b262605968 -- ceph orch apply mgr '2;smithi069=y;smithi122=x' 2024-04-14T13:52:02.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:52:02 smithi122 ceph-mon[25721]: Reconfiguring mon.b (monmap changed)... 2024-04-14T13:52:02.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:52:02 smithi122 ceph-mon[25721]: Reconfiguring daemon mon.b on smithi122 2024-04-14T13:52:02.567 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:52:02 smithi069 ceph-mon[24818]: Reconfiguring mon.b (monmap changed)... 2024-04-14T13:52:02.567 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:52:02 smithi069 ceph-mon[24818]: Reconfiguring daemon mon.b on smithi122 2024-04-14T13:52:02.568 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:52:02 smithi069 ceph-mon[29472]: Reconfiguring mon.b (monmap changed)... 2024-04-14T13:52:02.568 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:52:02 smithi069 ceph-mon[29472]: Reconfiguring daemon mon.b on smithi122 2024-04-14T13:52:03.316 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:52:03 smithi122 ceph-mon[25721]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-14T13:52:03.316 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:52:03 smithi122 ceph-mon[25721]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' 2024-04-14T13:52:03.316 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:52:03 smithi122 ceph-mon[25721]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-14T13:52:03.316 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:52:03 smithi122 ceph-mon[25721]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T13:52:03.316 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:52:03 smithi122 ceph-mon[25721]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-14T13:52:03.316 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:52:03 smithi122 ceph-mon[25721]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' 2024-04-14T13:52:03.568 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:52:03 smithi069 ceph-mon[24818]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-14T13:52:03.568 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:52:03 smithi069 ceph-mon[24818]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' 2024-04-14T13:52:03.568 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:52:03 smithi069 ceph-mon[24818]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-14T13:52:03.568 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:52:03 smithi069 ceph-mon[24818]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T13:52:03.568 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:52:03 smithi069 ceph-mon[24818]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-14T13:52:03.568 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:52:03 smithi069 ceph-mon[24818]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' 2024-04-14T13:52:03.569 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:52:03 smithi069 ceph-mon[29472]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-14T13:52:03.569 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:52:03 smithi069 ceph-mon[29472]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' 2024-04-14T13:52:03.569 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:52:03 smithi069 ceph-mon[29472]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-14T13:52:03.569 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:52:03 smithi069 ceph-mon[29472]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T13:52:03.569 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:52:03 smithi069 ceph-mon[29472]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-14T13:52:03.569 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:52:03 smithi069 ceph-mon[29472]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' 2024-04-14T13:52:03.641 INFO:teuthology.orchestra.run.smithi122.stdout:Scheduled mgr update... 2024-04-14T13:52:04.278 DEBUG:teuthology.orchestra.run.smithi122:mgr.x> sudo journalctl -f -n 0 -u ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968@mgr.x.service 2024-04-14T13:52:04.281 INFO:tasks.cephadm:Deploying OSDs... 2024-04-14T13:52:04.281 DEBUG:teuthology.orchestra.run.smithi069:> set -ex 2024-04-14T13:52:04.281 DEBUG:teuthology.orchestra.run.smithi069:> dd if=/scratch_devs of=/dev/stdout 2024-04-14T13:52:04.300 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-04-14T13:52:04.300 DEBUG:teuthology.orchestra.run.smithi069:> stat /dev/vg_nvme/lv_1 2024-04-14T13:52:04.358 INFO:teuthology.orchestra.run.smithi069.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-04-14T13:52:04.358 INFO:teuthology.orchestra.run.smithi069.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-14T13:52:04.358 INFO:teuthology.orchestra.run.smithi069.stdout:Device: 5h/5d Inode: 667 Links: 1 2024-04-14T13:52:04.358 INFO:teuthology.orchestra.run.smithi069.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-14T13:52:04.358 INFO:teuthology.orchestra.run.smithi069.stdout:Context: system_u:object_r:device_t:s0 2024-04-14T13:52:04.358 INFO:teuthology.orchestra.run.smithi069.stdout:Access: 2024-04-14 13:51:40.671859237 +0000 2024-04-14T13:52:04.358 INFO:teuthology.orchestra.run.smithi069.stdout:Modify: 2024-04-14 13:49:23.025449452 +0000 2024-04-14T13:52:04.358 INFO:teuthology.orchestra.run.smithi069.stdout:Change: 2024-04-14 13:49:23.025449452 +0000 2024-04-14T13:52:04.358 INFO:teuthology.orchestra.run.smithi069.stdout: Birth: 2024-04-14 13:49:23.025449452 +0000 2024-04-14T13:52:04.359 DEBUG:teuthology.orchestra.run.smithi069:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-04-14T13:52:04.428 INFO:teuthology.orchestra.run.smithi069.stderr:1+0 records in 2024-04-14T13:52:04.429 INFO:teuthology.orchestra.run.smithi069.stderr:1+0 records out 2024-04-14T13:52:04.429 INFO:teuthology.orchestra.run.smithi069.stderr:512 bytes copied, 0.000233579 s, 2.2 MB/s 2024-04-14T13:52:04.430 DEBUG:teuthology.orchestra.run.smithi069:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-04-14T13:52:04.489 DEBUG:teuthology.orchestra.run.smithi069:> stat /dev/vg_nvme/lv_2 2024-04-14T13:52:04.546 INFO:teuthology.orchestra.run.smithi069.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-04-14T13:52:04.546 INFO:teuthology.orchestra.run.smithi069.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-14T13:52:04.546 INFO:teuthology.orchestra.run.smithi069.stdout:Device: 5h/5d Inode: 687 Links: 1 2024-04-14T13:52:04.546 INFO:teuthology.orchestra.run.smithi069.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-14T13:52:04.546 INFO:teuthology.orchestra.run.smithi069.stdout:Context: system_u:object_r:device_t:s0 2024-04-14T13:52:04.546 INFO:teuthology.orchestra.run.smithi069.stdout:Access: 2024-04-14 13:51:40.671859237 +0000 2024-04-14T13:52:04.546 INFO:teuthology.orchestra.run.smithi069.stdout:Modify: 2024-04-14 13:49:23.351441779 +0000 2024-04-14T13:52:04.546 INFO:teuthology.orchestra.run.smithi069.stdout:Change: 2024-04-14 13:49:23.351441779 +0000 2024-04-14T13:52:04.546 INFO:teuthology.orchestra.run.smithi069.stdout: Birth: 2024-04-14 13:49:23.351441779 +0000 2024-04-14T13:52:04.547 DEBUG:teuthology.orchestra.run.smithi069:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-04-14T13:52:04.614 INFO:teuthology.orchestra.run.smithi069.stderr:1+0 records in 2024-04-14T13:52:04.614 INFO:teuthology.orchestra.run.smithi069.stderr:1+0 records out 2024-04-14T13:52:04.614 INFO:teuthology.orchestra.run.smithi069.stderr:512 bytes copied, 0.000246688 s, 2.1 MB/s 2024-04-14T13:52:04.615 DEBUG:teuthology.orchestra.run.smithi069:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-04-14T13:52:04.672 DEBUG:teuthology.orchestra.run.smithi069:> stat /dev/vg_nvme/lv_3 2024-04-14T13:52:04.727 INFO:teuthology.orchestra.run.smithi069.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-04-14T13:52:04.728 INFO:teuthology.orchestra.run.smithi069.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-14T13:52:04.728 INFO:teuthology.orchestra.run.smithi069.stdout:Device: 5h/5d Inode: 703 Links: 1 2024-04-14T13:52:04.728 INFO:teuthology.orchestra.run.smithi069.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-14T13:52:04.728 INFO:teuthology.orchestra.run.smithi069.stdout:Context: system_u:object_r:device_t:s0 2024-04-14T13:52:04.728 INFO:teuthology.orchestra.run.smithi069.stdout:Access: 2024-04-14 13:51:40.671859237 +0000 2024-04-14T13:52:04.728 INFO:teuthology.orchestra.run.smithi069.stdout:Modify: 2024-04-14 13:49:23.686433895 +0000 2024-04-14T13:52:04.728 INFO:teuthology.orchestra.run.smithi069.stdout:Change: 2024-04-14 13:49:23.686433895 +0000 2024-04-14T13:52:04.728 INFO:teuthology.orchestra.run.smithi069.stdout: Birth: 2024-04-14 13:49:23.686433895 +0000 2024-04-14T13:52:04.728 DEBUG:teuthology.orchestra.run.smithi069:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-04-14T13:52:04.793 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:52:04 smithi069 ceph-mon[24818]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-14T13:52:04.793 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:52:04 smithi069 ceph-mon[24818]: from='client.24107 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "2;smithi069=y;smithi122=x", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T13:52:04.793 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:52:04 smithi069 ceph-mon[24818]: Saving service mgr spec with placement smithi069=y;smithi122=x;count:2 2024-04-14T13:52:04.793 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:52:04 smithi069 ceph-mon[24818]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' 2024-04-14T13:52:04.793 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:52:04 smithi069 ceph-mon[24818]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-14T13:52:04.793 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:52:04 smithi069 ceph-mon[24818]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T13:52:04.793 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:52:04 smithi069 ceph-mon[24818]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-14T13:52:04.793 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:52:04 smithi069 ceph-mon[24818]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' 2024-04-14T13:52:04.793 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:52:04 smithi069 ceph-mon[24818]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-14T13:52:04.793 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:52:04 smithi069 ceph-mon[24818]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-04-14T13:52:04.793 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:52:04 smithi069 ceph-mon[24818]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-14T13:52:04.794 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:52:04 smithi069 ceph-mon[24818]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T13:52:04.794 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:52:04 smithi069 ceph-mon[24818]: Deploying daemon mgr.x on smithi122 2024-04-14T13:52:04.794 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:52:04 smithi069 ceph-mon[29472]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-14T13:52:04.794 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:52:04 smithi069 ceph-mon[29472]: from='client.24107 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "2;smithi069=y;smithi122=x", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T13:52:04.794 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:52:04 smithi069 ceph-mon[29472]: Saving service mgr spec with placement smithi069=y;smithi122=x;count:2 2024-04-14T13:52:04.794 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:52:04 smithi069 ceph-mon[29472]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' 2024-04-14T13:52:04.794 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:52:04 smithi069 ceph-mon[29472]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-14T13:52:04.794 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:52:04 smithi069 ceph-mon[29472]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T13:52:04.794 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:52:04 smithi069 ceph-mon[29472]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-14T13:52:04.794 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:52:04 smithi069 ceph-mon[29472]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' 2024-04-14T13:52:04.794 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:52:04 smithi069 ceph-mon[29472]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-14T13:52:04.794 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:52:04 smithi069 ceph-mon[29472]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-04-14T13:52:04.794 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:52:04 smithi069 ceph-mon[29472]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-14T13:52:04.795 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:52:04 smithi069 ceph-mon[29472]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T13:52:04.795 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:52:04 smithi069 ceph-mon[29472]: Deploying daemon mgr.x on smithi122 2024-04-14T13:52:04.795 INFO:teuthology.orchestra.run.smithi069.stderr:1+0 records in 2024-04-14T13:52:04.795 INFO:teuthology.orchestra.run.smithi069.stderr:1+0 records out 2024-04-14T13:52:04.795 INFO:teuthology.orchestra.run.smithi069.stderr:512 bytes copied, 0.000182877 s, 2.8 MB/s 2024-04-14T13:52:04.796 DEBUG:teuthology.orchestra.run.smithi069:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-04-14T13:52:04.808 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:52:04 smithi122 ceph-mon[25721]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-14T13:52:04.808 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:52:04 smithi122 ceph-mon[25721]: from='client.24107 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "2;smithi069=y;smithi122=x", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T13:52:04.808 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:52:04 smithi122 ceph-mon[25721]: Saving service mgr spec with placement smithi069=y;smithi122=x;count:2 2024-04-14T13:52:04.808 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:52:04 smithi122 ceph-mon[25721]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' 2024-04-14T13:52:04.809 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:52:04 smithi122 ceph-mon[25721]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-14T13:52:04.809 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:52:04 smithi122 ceph-mon[25721]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T13:52:04.809 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:52:04 smithi122 ceph-mon[25721]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-14T13:52:04.809 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:52:04 smithi122 ceph-mon[25721]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' 2024-04-14T13:52:04.809 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:52:04 smithi122 ceph-mon[25721]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-14T13:52:04.809 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:52:04 smithi122 ceph-mon[25721]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-04-14T13:52:04.809 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:52:04 smithi122 ceph-mon[25721]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-14T13:52:04.809 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:52:04 smithi122 ceph-mon[25721]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T13:52:04.809 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:52:04 smithi122 ceph-mon[25721]: Deploying daemon mgr.x on smithi122 2024-04-14T13:52:04.852 DEBUG:teuthology.orchestra.run.smithi069:> stat /dev/vg_nvme/lv_4 2024-04-14T13:52:04.907 INFO:teuthology.orchestra.run.smithi069.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-04-14T13:52:04.907 INFO:teuthology.orchestra.run.smithi069.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-14T13:52:04.907 INFO:teuthology.orchestra.run.smithi069.stdout:Device: 5h/5d Inode: 723 Links: 1 2024-04-14T13:52:04.908 INFO:teuthology.orchestra.run.smithi069.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-14T13:52:04.908 INFO:teuthology.orchestra.run.smithi069.stdout:Context: system_u:object_r:device_t:s0 2024-04-14T13:52:04.908 INFO:teuthology.orchestra.run.smithi069.stdout:Access: 2024-04-14 13:51:40.671859237 +0000 2024-04-14T13:52:04.908 INFO:teuthology.orchestra.run.smithi069.stdout:Modify: 2024-04-14 13:49:24.023425963 +0000 2024-04-14T13:52:04.908 INFO:teuthology.orchestra.run.smithi069.stdout:Change: 2024-04-14 13:49:24.023425963 +0000 2024-04-14T13:52:04.908 INFO:teuthology.orchestra.run.smithi069.stdout: Birth: 2024-04-14 13:49:24.023425963 +0000 2024-04-14T13:52:04.908 DEBUG:teuthology.orchestra.run.smithi069:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-04-14T13:52:04.973 INFO:teuthology.orchestra.run.smithi069.stderr:1+0 records in 2024-04-14T13:52:04.973 INFO:teuthology.orchestra.run.smithi069.stderr:1+0 records out 2024-04-14T13:52:04.973 INFO:teuthology.orchestra.run.smithi069.stderr:512 bytes copied, 0.000229883 s, 2.2 MB/s 2024-04-14T13:52:04.974 DEBUG:teuthology.orchestra.run.smithi069:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-04-14T13:52:05.031 DEBUG:teuthology.orchestra.run.smithi122:> set -ex 2024-04-14T13:52:05.031 DEBUG:teuthology.orchestra.run.smithi122:> dd if=/scratch_devs of=/dev/stdout 2024-04-14T13:52:05.047 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-04-14T13:52:05.047 DEBUG:teuthology.orchestra.run.smithi122:> stat /dev/vg_nvme/lv_1 2024-04-14T13:52:05.102 INFO:teuthology.orchestra.run.smithi122.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-04-14T13:52:05.102 INFO:teuthology.orchestra.run.smithi122.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-14T13:52:05.102 INFO:teuthology.orchestra.run.smithi122.stdout:Device: 5h/5d Inode: 665 Links: 1 2024-04-14T13:52:05.102 INFO:teuthology.orchestra.run.smithi122.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-14T13:52:05.102 INFO:teuthology.orchestra.run.smithi122.stdout:Context: system_u:object_r:device_t:s0 2024-04-14T13:52:05.102 INFO:teuthology.orchestra.run.smithi122.stdout:Access: 2024-04-14 13:51:49.791808359 +0000 2024-04-14T13:52:05.102 INFO:teuthology.orchestra.run.smithi122.stdout:Modify: 2024-04-14 13:49:23.382606859 +0000 2024-04-14T13:52:05.102 INFO:teuthology.orchestra.run.smithi122.stdout:Change: 2024-04-14 13:49:23.382606859 +0000 2024-04-14T13:52:05.103 INFO:teuthology.orchestra.run.smithi122.stdout: Birth: 2024-04-14 13:49:23.382606859 +0000 2024-04-14T13:52:05.103 DEBUG:teuthology.orchestra.run.smithi122:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-04-14T13:52:05.168 INFO:teuthology.orchestra.run.smithi122.stderr:1+0 records in 2024-04-14T13:52:05.168 INFO:teuthology.orchestra.run.smithi122.stderr:1+0 records out 2024-04-14T13:52:05.168 INFO:teuthology.orchestra.run.smithi122.stderr:512 bytes copied, 0.000258246 s, 2.0 MB/s 2024-04-14T13:52:05.169 DEBUG:teuthology.orchestra.run.smithi122:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-04-14T13:52:05.224 DEBUG:teuthology.orchestra.run.smithi122:> stat /dev/vg_nvme/lv_2 2024-04-14T13:52:05.280 INFO:teuthology.orchestra.run.smithi122.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-04-14T13:52:05.280 INFO:teuthology.orchestra.run.smithi122.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-14T13:52:05.280 INFO:teuthology.orchestra.run.smithi122.stdout:Device: 5h/5d Inode: 685 Links: 1 2024-04-14T13:52:05.280 INFO:teuthology.orchestra.run.smithi122.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-14T13:52:05.280 INFO:teuthology.orchestra.run.smithi122.stdout:Context: system_u:object_r:device_t:s0 2024-04-14T13:52:05.280 INFO:teuthology.orchestra.run.smithi122.stdout:Access: 2024-04-14 13:51:49.791808359 +0000 2024-04-14T13:52:05.280 INFO:teuthology.orchestra.run.smithi122.stdout:Modify: 2024-04-14 13:49:23.730598882 +0000 2024-04-14T13:52:05.280 INFO:teuthology.orchestra.run.smithi122.stdout:Change: 2024-04-14 13:49:23.730598882 +0000 2024-04-14T13:52:05.280 INFO:teuthology.orchestra.run.smithi122.stdout: Birth: 2024-04-14 13:49:23.730598882 +0000 2024-04-14T13:52:05.281 DEBUG:teuthology.orchestra.run.smithi122:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-04-14T13:52:05.345 INFO:teuthology.orchestra.run.smithi122.stderr:1+0 records in 2024-04-14T13:52:05.345 INFO:teuthology.orchestra.run.smithi122.stderr:1+0 records out 2024-04-14T13:52:05.345 INFO:teuthology.orchestra.run.smithi122.stderr:512 bytes copied, 0.000191046 s, 2.7 MB/s 2024-04-14T13:52:05.346 DEBUG:teuthology.orchestra.run.smithi122:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-04-14T13:52:05.402 DEBUG:teuthology.orchestra.run.smithi122:> stat /dev/vg_nvme/lv_3 2024-04-14T13:52:05.457 INFO:teuthology.orchestra.run.smithi122.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-04-14T13:52:05.457 INFO:teuthology.orchestra.run.smithi122.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-14T13:52:05.457 INFO:teuthology.orchestra.run.smithi122.stdout:Device: 5h/5d Inode: 704 Links: 1 2024-04-14T13:52:05.458 INFO:teuthology.orchestra.run.smithi122.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-14T13:52:05.458 INFO:teuthology.orchestra.run.smithi122.stdout:Context: system_u:object_r:device_t:s0 2024-04-14T13:52:05.458 INFO:teuthology.orchestra.run.smithi122.stdout:Access: 2024-04-14 13:51:49.792808336 +0000 2024-04-14T13:52:05.458 INFO:teuthology.orchestra.run.smithi122.stdout:Modify: 2024-04-14 13:49:24.074590997 +0000 2024-04-14T13:52:05.458 INFO:teuthology.orchestra.run.smithi122.stdout:Change: 2024-04-14 13:49:24.074590997 +0000 2024-04-14T13:52:05.458 INFO:teuthology.orchestra.run.smithi122.stdout: Birth: 2024-04-14 13:49:24.074590997 +0000 2024-04-14T13:52:05.458 DEBUG:teuthology.orchestra.run.smithi122:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-04-14T13:52:05.524 INFO:teuthology.orchestra.run.smithi122.stderr:1+0 records in 2024-04-14T13:52:05.524 INFO:teuthology.orchestra.run.smithi122.stderr:1+0 records out 2024-04-14T13:52:05.524 INFO:teuthology.orchestra.run.smithi122.stderr:512 bytes copied, 0.000194286 s, 2.6 MB/s 2024-04-14T13:52:05.525 DEBUG:teuthology.orchestra.run.smithi122:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-04-14T13:52:05.584 DEBUG:teuthology.orchestra.run.smithi122:> stat /dev/vg_nvme/lv_4 2024-04-14T13:52:05.639 INFO:teuthology.orchestra.run.smithi122.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-04-14T13:52:05.639 INFO:teuthology.orchestra.run.smithi122.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-14T13:52:05.639 INFO:teuthology.orchestra.run.smithi122.stdout:Device: 5h/5d Inode: 723 Links: 1 2024-04-14T13:52:05.639 INFO:teuthology.orchestra.run.smithi122.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-14T13:52:05.639 INFO:teuthology.orchestra.run.smithi122.stdout:Context: system_u:object_r:device_t:s0 2024-04-14T13:52:05.640 INFO:teuthology.orchestra.run.smithi122.stdout:Access: 2024-04-14 13:52:05.478447192 +0000 2024-04-14T13:52:05.640 INFO:teuthology.orchestra.run.smithi122.stdout:Modify: 2024-04-14 13:49:24.412583249 +0000 2024-04-14T13:52:05.640 INFO:teuthology.orchestra.run.smithi122.stdout:Change: 2024-04-14 13:49:24.412583249 +0000 2024-04-14T13:52:05.640 INFO:teuthology.orchestra.run.smithi122.stdout: Birth: 2024-04-14 13:49:24.412583249 +0000 2024-04-14T13:52:05.640 DEBUG:teuthology.orchestra.run.smithi122:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-04-14T13:52:05.715 INFO:teuthology.orchestra.run.smithi122.stderr:1+0 records in 2024-04-14T13:52:05.715 INFO:teuthology.orchestra.run.smithi122.stderr:1+0 records out 2024-04-14T13:52:05.715 INFO:teuthology.orchestra.run.smithi122.stderr:512 bytes copied, 0.000182082 s, 2.8 MB/s 2024-04-14T13:52:05.716 DEBUG:teuthology.orchestra.run.smithi122:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-04-14T13:52:05.773 INFO:tasks.cephadm:Deploying osd.0 on smithi069 with /dev/vg_nvme/lv_4... 2024-04-14T13:52:05.773 DEBUG:teuthology.orchestra.run.smithi069:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid dc326fc2-fa65-11ee-bc8e-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-04-14T13:52:06.181 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Apr 14 13:52:06 smithi122 podman[26898]: 2024-04-14 13:52:06.15834385 +0000 UTC m=+0.347462667 container start 9aaf29b023b15d873a3c639dd2ab368d275242c3251b5d983ab68475c2451b3e (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-x, io.k8s.display-name=CentOS Stream 8, build-date=2022-05-03T08:36:31.336870, RELEASE=HEAD, io.openshift.expose-services=, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, io.buildah.version=1.19.8, name=centos-stream, distribution-scope=public, ceph=True, io.openshift.tags=base centos centos-stream, GIT_BRANCH=HEAD, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, com.redhat.component=centos-stream-container, maintainer=Guillaume Abrioux , vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, vendor=Red Hat, Inc., url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, architecture=x86_64, version=8, CEPH_POINT_RELEASE=-17.2.0, release=754, vcs-type=git, GIT_CLEAN=True, GIT_REPO=https://github.com/ceph/ceph-container.git, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., com.redhat.license_terms=https://centos.org/legal/licensing-policy/) 2024-04-14T13:52:06.182 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Apr 14 13:52:06 smithi122 ceph-mgr[26917]: -- 172.21.15.122:0/4230019410 <== mon.2 v2:172.21.15.122:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55d0bd67e340 con 0x55d0bd657000 2024-04-14T13:52:06.595 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Apr 14 13:52:06 smithi122 bash[26898]: 9aaf29b023b15d873a3c639dd2ab368d275242c3251b5d983ab68475c2451b3e 2024-04-14T13:52:06.595 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Apr 14 13:52:06 smithi122 systemd[1]: Started Ceph mgr.x for dc326fc2-fa65-11ee-bc8e-c7b262605968. 2024-04-14T13:52:06.957 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Apr 14 13:52:06 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-x[26913]: 2024-04-14T13:52:06.680+0000 7fc642c07000 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-14T13:52:06.958 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Apr 14 13:52:06 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-x[26913]: 2024-04-14T13:52:06.773+0000 7fc642c07000 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-14T13:52:07.551 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:52:07 smithi122 ceph-mon[25721]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-14T13:52:07.551 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:52:07 smithi122 ceph-mon[25721]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' 2024-04-14T13:52:07.551 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:52:07 smithi122 ceph-mon[25721]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-14T13:52:07.551 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:52:07 smithi122 ceph-mon[25721]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T13:52:07.551 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:52:07 smithi122 ceph-mon[25721]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-14T13:52:07.551 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:52:07 smithi122 ceph-mon[25721]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' 2024-04-14T13:52:07.551 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:52:07 smithi122 ceph-mon[25721]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' 2024-04-14T13:52:07.551 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:52:07 smithi122 ceph-mon[25721]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' 2024-04-14T13:52:07.551 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:52:07 smithi122 ceph-mon[25721]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-14T13:52:07.551 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:52:07 smithi122 ceph-mon[25721]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-14T13:52:07.552 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:52:07 smithi122 ceph-mon[25721]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T13:52:07.552 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Apr 14 13:52:07 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-x[26913]: 2024-04-14T13:52:07.381+0000 7fc642c07000 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-14T13:52:07.552 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Apr 14 13:52:07 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-x[26913]: 2024-04-14T13:52:07.467+0000 7fc642c07000 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-14T13:52:07.552 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Apr 14 13:52:07 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-x[26913]: 2024-04-14T13:52:07.548+0000 7fc642c07000 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-14T13:52:07.567 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:52:07 smithi069 ceph-mon[29472]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-14T13:52:07.568 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:52:07 smithi069 ceph-mon[29472]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' 2024-04-14T13:52:07.568 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:52:07 smithi069 ceph-mon[29472]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-14T13:52:07.568 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:52:07 smithi069 ceph-mon[29472]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T13:52:07.568 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:52:07 smithi069 ceph-mon[29472]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-14T13:52:07.568 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:52:07 smithi069 ceph-mon[29472]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' 2024-04-14T13:52:07.568 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:52:07 smithi069 ceph-mon[29472]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' 2024-04-14T13:52:07.568 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:52:07 smithi069 ceph-mon[29472]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' 2024-04-14T13:52:07.568 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:52:07 smithi069 ceph-mon[29472]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-14T13:52:07.568 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:52:07 smithi069 ceph-mon[29472]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-14T13:52:07.568 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:52:07 smithi069 ceph-mon[29472]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T13:52:07.568 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:52:07 smithi069 ceph-mon[24818]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-14T13:52:07.568 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:52:07 smithi069 ceph-mon[24818]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' 2024-04-14T13:52:07.569 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:52:07 smithi069 ceph-mon[24818]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-14T13:52:07.569 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:52:07 smithi069 ceph-mon[24818]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T13:52:07.569 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:52:07 smithi069 ceph-mon[24818]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-14T13:52:07.569 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:52:07 smithi069 ceph-mon[24818]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' 2024-04-14T13:52:07.569 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:52:07 smithi069 ceph-mon[24818]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' 2024-04-14T13:52:07.569 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:52:07 smithi069 ceph-mon[24818]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' 2024-04-14T13:52:07.569 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:52:07 smithi069 ceph-mon[24818]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-14T13:52:07.569 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:52:07 smithi069 ceph-mon[24818]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-14T13:52:07.569 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:52:07 smithi069 ceph-mon[24818]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T13:52:07.655 INFO:teuthology.orchestra.run.smithi069.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-04-14T13:52:07.655 INFO:teuthology.orchestra.run.smithi069.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-04-14T13:52:07.655 INFO:teuthology.orchestra.run.smithi069.stderr: stderr: 10+0 records in 2024-04-14T13:52:07.656 INFO:teuthology.orchestra.run.smithi069.stderr:10+0 records out 2024-04-14T13:52:07.656 INFO:teuthology.orchestra.run.smithi069.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0123654 s, 848 MB/s 2024-04-14T13:52:07.656 INFO:teuthology.orchestra.run.smithi069.stderr:--> Zapping successful for: 2024-04-14T13:52:07.845 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Apr 14 13:52:07 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-x[26913]: 2024-04-14T13:52:07.693+0000 7fc642c07000 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-14T13:52:07.845 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Apr 14 13:52:07 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-x[26913]: 2024-04-14T13:52:07.771+0000 7fc642c07000 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-14T13:52:08.201 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Apr 14 13:52:07 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-x[26913]: 2024-04-14T13:52:07.878+0000 7fc642c07000 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-14T13:52:08.201 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Apr 14 13:52:07 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-x[26913]: 2024-04-14T13:52:07.958+0000 7fc642c07000 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-14T13:52:08.201 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Apr 14 13:52:08 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-x[26913]: 2024-04-14T13:52:08.116+0000 7fc642c07000 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-14T13:52:08.201 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Apr 14 13:52:08 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-x[26913]: 2024-04-14T13:52:08.198+0000 7fc642c07000 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-14T13:52:08.314 DEBUG:teuthology.orchestra.run.smithi069:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid dc326fc2-fa65-11ee-bc8e-c7b262605968 -- ceph orch daemon add osd smithi069:vg_nvme/lv_4 2024-04-14T13:52:08.595 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:52:08 smithi122 ceph-mon[25721]: Reconfiguring mgr.y (unknown last config time)... 2024-04-14T13:52:08.595 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:52:08 smithi122 ceph-mon[25721]: Reconfiguring daemon mgr.y on smithi069 2024-04-14T13:52:08.817 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:52:08 smithi069 ceph-mon[24818]: Reconfiguring mgr.y (unknown last config time)... 2024-04-14T13:52:08.817 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:52:08 smithi069 ceph-mon[24818]: Reconfiguring daemon mgr.y on smithi069 2024-04-14T13:52:08.818 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:52:08 smithi069 ceph-mon[29472]: Reconfiguring mgr.y (unknown last config time)... 2024-04-14T13:52:08.818 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:52:08 smithi069 ceph-mon[29472]: Reconfiguring daemon mgr.y on smithi069 2024-04-14T13:52:09.461 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:52:09 smithi069 ceph-mon[24818]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-14T13:52:09.461 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:52:09 smithi069 ceph-mon[29472]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-14T13:52:09.521 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:52:09 smithi122 ceph-mon[25721]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-14T13:52:09.521 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Apr 14 13:52:09 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-x[26913]: 2024-04-14T13:52:09.207+0000 7fc642c07000 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-14T13:52:09.521 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Apr 14 13:52:09 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-x[26913]: 2024-04-14T13:52:09.332+0000 7fc642c07000 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-14T13:52:09.521 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Apr 14 13:52:09 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-x[26913]: 2024-04-14T13:52:09.518+0000 7fc642c07000 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-14T13:52:09.845 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Apr 14 13:52:09 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-x[26913]: 2024-04-14T13:52:09.603+0000 7fc642c07000 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-14T13:52:09.845 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Apr 14 13:52:09 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-x[26913]: 2024-04-14T13:52:09.734+0000 7fc642c07000 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-14T13:52:10.345 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Apr 14 13:52:10 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-x[26913]: 2024-04-14T13:52:10.046+0000 7fc642c07000 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-14T13:52:10.345 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Apr 14 13:52:10 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-x[26913]: 2024-04-14T13:52:10.261+0000 7fc642c07000 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-14T13:52:10.751 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Apr 14 13:52:10 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-x[26913]: 2024-04-14T13:52:10.344+0000 7fc642c07000 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-14T13:52:10.751 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Apr 14 13:52:10 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-x[26913]: 2024-04-14T13:52:10.510+0000 7fc642c07000 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-14T13:52:11.067 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:52:10 smithi069 ceph-mon[24818]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-14T13:52:11.067 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:52:10 smithi069 ceph-mon[24818]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' 2024-04-14T13:52:11.067 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:52:10 smithi069 ceph-mon[24818]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-14T13:52:11.068 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:52:10 smithi069 ceph-mon[24818]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T13:52:11.068 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:52:10 smithi069 ceph-mon[24818]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-14T13:52:11.068 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:52:10 smithi069 ceph-mon[24818]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' 2024-04-14T13:52:11.068 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:52:10 smithi069 ceph-mon[24818]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-14T13:52:11.068 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:52:10 smithi069 ceph-mon[24818]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-14T13:52:11.068 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:52:10 smithi069 ceph-mon[24818]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T13:52:11.068 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:52:10 smithi069 ceph-mon[29472]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-14T13:52:11.068 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:52:10 smithi069 ceph-mon[29472]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' 2024-04-14T13:52:11.068 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:52:10 smithi069 ceph-mon[29472]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-14T13:52:11.068 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:52:10 smithi069 ceph-mon[29472]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T13:52:11.068 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:52:10 smithi069 ceph-mon[29472]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-14T13:52:11.068 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:52:10 smithi069 ceph-mon[29472]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' 2024-04-14T13:52:11.069 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:52:10 smithi069 ceph-mon[29472]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-14T13:52:11.069 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:52:10 smithi069 ceph-mon[29472]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-14T13:52:11.069 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:52:10 smithi069 ceph-mon[29472]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T13:52:11.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:52:10 smithi122 ceph-mon[25721]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-14T13:52:11.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:52:10 smithi122 ceph-mon[25721]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' 2024-04-14T13:52:11.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:52:10 smithi122 ceph-mon[25721]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-14T13:52:11.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:52:10 smithi122 ceph-mon[25721]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T13:52:11.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:52:10 smithi122 ceph-mon[25721]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-14T13:52:11.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:52:10 smithi122 ceph-mon[25721]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' 2024-04-14T13:52:11.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:52:10 smithi122 ceph-mon[25721]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-14T13:52:11.096 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:52:10 smithi122 ceph-mon[25721]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-14T13:52:11.096 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:52:10 smithi122 ceph-mon[25721]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T13:52:11.690 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Apr 14 13:52:11 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-x[26913]: 2024-04-14T13:52:11.401+0000 7fc642c07000 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-14T13:52:11.690 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Apr 14 13:52:11 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-x[26913]: 2024-04-14T13:52:11.687+0000 7fc642c07000 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-14T13:52:12.031 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:52:11 smithi069 ceph-mon[24818]: from='client.24121 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi069:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T13:52:12.031 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:52:11 smithi069 ceph-mon[24818]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-04-14T13:52:12.032 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:52:11 smithi069 ceph-mon[24818]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-04-14T13:52:12.032 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:52:11 smithi069 ceph-mon[29472]: from='client.24121 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi069:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T13:52:12.032 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:52:11 smithi069 ceph-mon[29472]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-04-14T13:52:12.032 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:52:11 smithi069 ceph-mon[29472]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-04-14T13:52:12.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:52:11 smithi122 ceph-mon[25721]: from='client.24121 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi069:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T13:52:12.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:52:11 smithi122 ceph-mon[25721]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-04-14T13:52:12.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:52:11 smithi122 ceph-mon[25721]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-04-14T13:52:12.095 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Apr 14 13:52:11 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-x[26913]: 2024-04-14T13:52:11.781+0000 7fc642c07000 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-14T13:52:12.096 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Apr 14 13:52:11 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-x[26913]: 2024-04-14T13:52:11.926+0000 7fc642c07000 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-14T13:52:13.067 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:52:12 smithi069 ceph-mon[29472]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-14T13:52:13.068 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:52:12 smithi069 ceph-mon[29472]: Standby manager daemon x started 2024-04-14T13:52:13.068 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:52:12 smithi069 ceph-mon[29472]: from='mgr.? 172.21.15.122:0/4020834838' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-04-14T13:52:13.068 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:52:12 smithi069 ceph-mon[29472]: from='mgr.? 172.21.15.122:0/4020834838' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-14T13:52:13.068 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:52:12 smithi069 ceph-mon[29472]: from='mgr.? 172.21.15.122:0/4020834838' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-04-14T13:52:13.068 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:52:12 smithi069 ceph-mon[29472]: from='mgr.? 172.21.15.122:0/4020834838' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-14T13:52:13.068 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:52:12 smithi069 ceph-mon[24818]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-14T13:52:13.068 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:52:12 smithi069 ceph-mon[24818]: Standby manager daemon x started 2024-04-14T13:52:13.068 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:52:12 smithi069 ceph-mon[24818]: from='mgr.? 172.21.15.122:0/4020834838' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-04-14T13:52:13.068 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:52:12 smithi069 ceph-mon[24818]: from='mgr.? 172.21.15.122:0/4020834838' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-14T13:52:13.068 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:52:12 smithi069 ceph-mon[24818]: from='mgr.? 172.21.15.122:0/4020834838' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-04-14T13:52:13.068 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:52:12 smithi069 ceph-mon[24818]: from='mgr.? 172.21.15.122:0/4020834838' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-14T13:52:13.096 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:52:12 smithi122 ceph-mon[25721]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-14T13:52:13.096 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:52:12 smithi122 ceph-mon[25721]: Standby manager daemon x started 2024-04-14T13:52:13.096 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:52:12 smithi122 ceph-mon[25721]: from='mgr.? 172.21.15.122:0/4020834838' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-04-14T13:52:13.096 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:52:12 smithi122 ceph-mon[25721]: from='mgr.? 172.21.15.122:0/4020834838' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-14T13:52:13.096 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:52:12 smithi122 ceph-mon[25721]: from='mgr.? 172.21.15.122:0/4020834838' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-04-14T13:52:13.096 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:52:12 smithi122 ceph-mon[25721]: from='mgr.? 172.21.15.122:0/4020834838' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-14T13:52:14.067 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:52:13 smithi069 ceph-mon[24818]: mgrmap e14: y(active, since 61s), standbys: x 2024-04-14T13:52:14.068 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:52:13 smithi069 ceph-mon[24818]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-04-14T13:52:14.068 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:52:13 smithi069 ceph-mon[24818]: from='client.? 172.21.15.69:0/2488088945' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "99aae7c5-9755-49e8-91c7-fceed3ff30fb"}]: dispatch 2024-04-14T13:52:14.068 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:52:13 smithi069 ceph-mon[24818]: from='client.? 172.21.15.69:0/2488088945' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "99aae7c5-9755-49e8-91c7-fceed3ff30fb"}]': finished 2024-04-14T13:52:14.068 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:52:13 smithi069 ceph-mon[24818]: osdmap e5: 1 total, 0 up, 1 in 2024-04-14T13:52:14.068 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:52:13 smithi069 ceph-mon[24818]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-14T13:52:14.068 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:52:13 smithi069 ceph-mon[24818]: from='client.? 172.21.15.69:0/3554543114' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-14T13:52:14.068 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:52:13 smithi069 ceph-mon[29472]: mgrmap e14: y(active, since 61s), standbys: x 2024-04-14T13:52:14.068 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:52:13 smithi069 ceph-mon[29472]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-04-14T13:52:14.068 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:52:13 smithi069 ceph-mon[29472]: from='client.? 172.21.15.69:0/2488088945' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "99aae7c5-9755-49e8-91c7-fceed3ff30fb"}]: dispatch 2024-04-14T13:52:14.068 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:52:13 smithi069 ceph-mon[29472]: from='client.? 172.21.15.69:0/2488088945' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "99aae7c5-9755-49e8-91c7-fceed3ff30fb"}]': finished 2024-04-14T13:52:14.068 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:52:13 smithi069 ceph-mon[29472]: osdmap e5: 1 total, 0 up, 1 in 2024-04-14T13:52:14.069 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:52:13 smithi069 ceph-mon[29472]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-14T13:52:14.069 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:52:13 smithi069 ceph-mon[29472]: from='client.? 172.21.15.69:0/3554543114' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-14T13:52:14.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:52:13 smithi122 ceph-mon[25721]: mgrmap e14: y(active, since 61s), standbys: x 2024-04-14T13:52:14.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:52:13 smithi122 ceph-mon[25721]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-04-14T13:52:14.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:52:13 smithi122 ceph-mon[25721]: from='client.? 172.21.15.69:0/2488088945' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "99aae7c5-9755-49e8-91c7-fceed3ff30fb"}]: dispatch 2024-04-14T13:52:14.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:52:13 smithi122 ceph-mon[25721]: from='client.? 172.21.15.69:0/2488088945' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "99aae7c5-9755-49e8-91c7-fceed3ff30fb"}]': finished 2024-04-14T13:52:14.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:52:13 smithi122 ceph-mon[25721]: osdmap e5: 1 total, 0 up, 1 in 2024-04-14T13:52:14.096 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:52:13 smithi122 ceph-mon[25721]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-14T13:52:14.096 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:52:13 smithi122 ceph-mon[25721]: from='client.? 172.21.15.69:0/3554543114' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-14T13:52:15.317 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:52:14 smithi069 ceph-mon[29472]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-14T13:52:15.318 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:52:14 smithi069 ceph-mon[24818]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-14T13:52:15.345 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:52:14 smithi122 ceph-mon[25721]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-14T13:52:17.317 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:52:17 smithi069 ceph-mon[24818]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-14T13:52:17.318 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:52:17 smithi069 ceph-mon[29472]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-14T13:52:17.595 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:52:17 smithi122 ceph-mon[25721]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-14T13:52:19.462 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:52:19 smithi069 ceph-mon[24818]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-14T13:52:19.463 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:52:19 smithi069 ceph-mon[24818]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-04-14T13:52:19.463 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:52:19 smithi069 ceph-mon[24818]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T13:52:19.463 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:52:19 smithi069 ceph-mon[29472]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-14T13:52:19.463 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:52:19 smithi069 ceph-mon[29472]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-04-14T13:52:19.463 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:52:19 smithi069 ceph-mon[29472]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T13:52:19.595 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:52:19 smithi122 ceph-mon[25721]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-14T13:52:19.595 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:52:19 smithi122 ceph-mon[25721]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-04-14T13:52:19.595 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:52:19 smithi122 ceph-mon[25721]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T13:52:20.308 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:52:20 smithi069 ceph-mon[24818]: Deploying daemon osd.0 on smithi069 2024-04-14T13:52:20.308 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:52:20 smithi069 ceph-mon[29472]: Deploying daemon osd.0 on smithi069 2024-04-14T13:52:20.497 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:52:20 smithi122 ceph-mon[25721]: Deploying daemon osd.0 on smithi069 2024-04-14T13:52:21.540 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:52:21 smithi069 ceph-mon[24818]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-14T13:52:21.541 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:52:21 smithi069 ceph-mon[29472]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-14T13:52:21.595 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:52:21 smithi122 ceph-mon[25721]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-14T13:52:23.278 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:52:23 smithi069 ceph-mon[24818]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-14T13:52:23.279 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:52:23 smithi069 ceph-mon[29472]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-14T13:52:23.595 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:52:23 smithi122 ceph-mon[25721]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-14T13:52:24.817 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:52:24 smithi069 ceph-mon[24818]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-14T13:52:24.817 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:52:24 smithi069 ceph-mon[24818]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' 2024-04-14T13:52:24.818 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:52:24 smithi069 ceph-mon[24818]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-14T13:52:24.818 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:52:24 smithi069 ceph-mon[24818]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T13:52:24.818 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:52:24 smithi069 ceph-mon[24818]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-14T13:52:24.818 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:52:24 smithi069 ceph-mon[29472]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-14T13:52:24.818 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:52:24 smithi069 ceph-mon[29472]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' 2024-04-14T13:52:24.818 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:52:24 smithi069 ceph-mon[29472]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-14T13:52:24.818 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:52:24 smithi069 ceph-mon[29472]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T13:52:24.818 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:52:24 smithi069 ceph-mon[29472]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-14T13:52:24.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:52:24 smithi122 ceph-mon[25721]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-14T13:52:24.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:52:24 smithi122 ceph-mon[25721]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' 2024-04-14T13:52:24.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:52:24 smithi122 ceph-mon[25721]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-14T13:52:24.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:52:24 smithi122 ceph-mon[25721]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T13:52:24.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:52:24 smithi122 ceph-mon[25721]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-14T13:52:26.456 INFO:teuthology.orchestra.run.smithi069.stdout:Created osd(s) 0 on host 'smithi069' 2024-04-14T13:52:27.015 DEBUG:teuthology.orchestra.run.smithi069:osd.0> sudo journalctl -f -n 0 -u ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968@osd.0.service 2024-04-14T13:52:27.017 INFO:tasks.cephadm:Deploying osd.1 on smithi069 with /dev/vg_nvme/lv_3... 2024-04-14T13:52:27.017 DEBUG:teuthology.orchestra.run.smithi069:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid dc326fc2-fa65-11ee-bc8e-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-04-14T13:52:27.317 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:52:27 smithi069 ceph-mon[24818]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-14T13:52:27.317 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:52:27 smithi069 ceph-mon[24818]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' 2024-04-14T13:52:27.317 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:52:27 smithi069 ceph-mon[24818]: from='osd.0 [v2:172.21.15.69:6802/2517323608,v1:172.21.15.69:6803/2517323608]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-04-14T13:52:27.317 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:52:27 smithi069 ceph-mon[24818]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' 2024-04-14T13:52:27.318 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:52:27 smithi069 ceph-mon[29472]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-14T13:52:27.318 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:52:27 smithi069 ceph-mon[29472]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' 2024-04-14T13:52:27.318 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:52:27 smithi069 ceph-mon[29472]: from='osd.0 [v2:172.21.15.69:6802/2517323608,v1:172.21.15.69:6803/2517323608]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-04-14T13:52:27.318 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:52:27 smithi069 ceph-mon[29472]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' 2024-04-14T13:52:27.595 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:52:27 smithi122 ceph-mon[25721]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-14T13:52:27.595 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:52:27 smithi122 ceph-mon[25721]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' 2024-04-14T13:52:27.596 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:52:27 smithi122 ceph-mon[25721]: from='osd.0 [v2:172.21.15.69:6802/2517323608,v1:172.21.15.69:6803/2517323608]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-04-14T13:52:27.596 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:52:27 smithi122 ceph-mon[25721]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' 2024-04-14T13:52:28.751 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:52:28 smithi069 ceph-mon[24818]: from='osd.0 [v2:172.21.15.69:6802/2517323608,v1:172.21.15.69:6803/2517323608]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-04-14T13:52:28.751 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:52:28 smithi069 ceph-mon[24818]: osdmap e6: 1 total, 0 up, 1 in 2024-04-14T13:52:28.751 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:52:28 smithi069 ceph-mon[24818]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-14T13:52:28.751 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:52:28 smithi069 ceph-mon[24818]: from='osd.0 [v2:172.21.15.69:6802/2517323608,v1:172.21.15.69:6803/2517323608]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi069", "root=default"]}]: dispatch 2024-04-14T13:52:28.751 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:52:28 smithi069 ceph-mon[24818]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-14T13:52:28.751 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:52:28 smithi069 ceph-mon[29472]: from='osd.0 [v2:172.21.15.69:6802/2517323608,v1:172.21.15.69:6803/2517323608]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-04-14T13:52:28.752 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:52:28 smithi069 ceph-mon[29472]: osdmap e6: 1 total, 0 up, 1 in 2024-04-14T13:52:28.752 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:52:28 smithi069 ceph-mon[29472]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-14T13:52:28.752 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:52:28 smithi069 ceph-mon[29472]: from='osd.0 [v2:172.21.15.69:6802/2517323608,v1:172.21.15.69:6803/2517323608]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi069", "root=default"]}]: dispatch 2024-04-14T13:52:28.752 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:52:28 smithi069 ceph-mon[29472]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-14T13:52:28.752 INFO:journalctl@ceph.osd.0.smithi069.stdout:Apr 14 13:52:28 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-osd-0[32331]: 2024-04-14T13:52:28.460+0000 7ffa49781700 -1 osd.0 0 waiting for initial osdmap 2024-04-14T13:52:28.752 INFO:journalctl@ceph.osd.0.smithi069.stdout:Apr 14 13:52:28 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-osd-0[32331]: 2024-04-14T13:52:28.464+0000 7ffa42114700 -1 osd.0 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-14T13:52:28.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:52:28 smithi122 ceph-mon[25721]: from='osd.0 [v2:172.21.15.69:6802/2517323608,v1:172.21.15.69:6803/2517323608]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-04-14T13:52:28.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:52:28 smithi122 ceph-mon[25721]: osdmap e6: 1 total, 0 up, 1 in 2024-04-14T13:52:28.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:52:28 smithi122 ceph-mon[25721]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-14T13:52:28.846 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:52:28 smithi122 ceph-mon[25721]: from='osd.0 [v2:172.21.15.69:6802/2517323608,v1:172.21.15.69:6803/2517323608]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi069", "root=default"]}]: dispatch 2024-04-14T13:52:28.846 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:52:28 smithi122 ceph-mon[25721]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-14T13:52:29.633 INFO:teuthology.orchestra.run.smithi069.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-04-14T13:52:29.633 INFO:teuthology.orchestra.run.smithi069.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-04-14T13:52:29.633 INFO:teuthology.orchestra.run.smithi069.stderr: stderr: 10+0 records in 2024-04-14T13:52:29.633 INFO:teuthology.orchestra.run.smithi069.stderr:10+0 records out 2024-04-14T13:52:29.633 INFO:teuthology.orchestra.run.smithi069.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0122594 s, 855 MB/s 2024-04-14T13:52:29.633 INFO:teuthology.orchestra.run.smithi069.stderr:--> Zapping successful for: 2024-04-14T13:52:29.782 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:52:29 smithi069 ceph-mon[24818]: from='osd.0 [v2:172.21.15.69:6802/2517323608,v1:172.21.15.69:6803/2517323608]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi069", "root=default"]}]': finished 2024-04-14T13:52:29.782 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:52:29 smithi069 ceph-mon[24818]: osdmap e7: 1 total, 0 up, 1 in 2024-04-14T13:52:29.782 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:52:29 smithi069 ceph-mon[24818]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-14T13:52:29.782 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:52:29 smithi069 ceph-mon[24818]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-14T13:52:29.782 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:52:29 smithi069 ceph-mon[29472]: from='osd.0 [v2:172.21.15.69:6802/2517323608,v1:172.21.15.69:6803/2517323608]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi069", "root=default"]}]': finished 2024-04-14T13:52:29.782 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:52:29 smithi069 ceph-mon[29472]: osdmap e7: 1 total, 0 up, 1 in 2024-04-14T13:52:29.783 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:52:29 smithi069 ceph-mon[29472]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-14T13:52:29.783 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:52:29 smithi069 ceph-mon[29472]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-14T13:52:29.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:52:29 smithi122 ceph-mon[25721]: from='osd.0 [v2:172.21.15.69:6802/2517323608,v1:172.21.15.69:6803/2517323608]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi069", "root=default"]}]': finished 2024-04-14T13:52:29.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:52:29 smithi122 ceph-mon[25721]: osdmap e7: 1 total, 0 up, 1 in 2024-04-14T13:52:29.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:52:29 smithi122 ceph-mon[25721]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-14T13:52:29.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:52:29 smithi122 ceph-mon[25721]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-14T13:52:30.626 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:52:30 smithi069 ceph-mon[24818]: purged_snaps scrub starts 2024-04-14T13:52:30.626 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:52:30 smithi069 ceph-mon[24818]: purged_snaps scrub ok 2024-04-14T13:52:30.626 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:52:30 smithi069 ceph-mon[24818]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-14T13:52:30.627 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:52:30 smithi069 ceph-mon[24818]: osd.0 [v2:172.21.15.69:6802/2517323608,v1:172.21.15.69:6803/2517323608] boot 2024-04-14T13:52:30.627 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:52:30 smithi069 ceph-mon[24818]: osdmap e8: 1 total, 1 up, 1 in 2024-04-14T13:52:30.627 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:52:30 smithi069 ceph-mon[24818]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-14T13:52:30.627 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:52:30 smithi069 ceph-mon[29472]: purged_snaps scrub starts 2024-04-14T13:52:30.627 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:52:30 smithi069 ceph-mon[29472]: purged_snaps scrub ok 2024-04-14T13:52:30.627 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:52:30 smithi069 ceph-mon[29472]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-14T13:52:30.627 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:52:30 smithi069 ceph-mon[29472]: osd.0 [v2:172.21.15.69:6802/2517323608,v1:172.21.15.69:6803/2517323608] boot 2024-04-14T13:52:30.627 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:52:30 smithi069 ceph-mon[29472]: osdmap e8: 1 total, 1 up, 1 in 2024-04-14T13:52:30.627 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:52:30 smithi069 ceph-mon[29472]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-14T13:52:30.823 DEBUG:teuthology.orchestra.run.smithi069:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid dc326fc2-fa65-11ee-bc8e-c7b262605968 -- ceph orch daemon add osd smithi069:vg_nvme/lv_3 2024-04-14T13:52:30.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:52:30 smithi122 ceph-mon[25721]: purged_snaps scrub starts 2024-04-14T13:52:30.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:52:30 smithi122 ceph-mon[25721]: purged_snaps scrub ok 2024-04-14T13:52:30.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:52:30 smithi122 ceph-mon[25721]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-14T13:52:30.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:52:30 smithi122 ceph-mon[25721]: osd.0 [v2:172.21.15.69:6802/2517323608,v1:172.21.15.69:6803/2517323608] boot 2024-04-14T13:52:30.846 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:52:30 smithi122 ceph-mon[25721]: osdmap e8: 1 total, 1 up, 1 in 2024-04-14T13:52:30.846 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:52:30 smithi122 ceph-mon[25721]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-14T13:52:31.567 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:52:31 smithi069 ceph-mon[24818]: osdmap e9: 1 total, 1 up, 1 in 2024-04-14T13:52:31.568 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:52:31 smithi069 ceph-mon[29472]: osdmap e9: 1 total, 1 up, 1 in 2024-04-14T13:52:31.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:52:31 smithi122 ceph-mon[25721]: osdmap e9: 1 total, 1 up, 1 in 2024-04-14T13:52:32.802 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:52:32 smithi069 ceph-mon[24818]: pgmap v39: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-04-14T13:52:32.802 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:52:32 smithi069 ceph-mon[24818]: Detected new or changed devices on smithi069 2024-04-14T13:52:32.802 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:52:32 smithi069 ceph-mon[24818]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' 2024-04-14T13:52:32.802 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:52:32 smithi069 ceph-mon[24818]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-14T13:52:32.802 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:52:32 smithi069 ceph-mon[24818]: Adjusting osd_memory_target on smithi069 to 16064M 2024-04-14T13:52:32.802 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:52:32 smithi069 ceph-mon[24818]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' 2024-04-14T13:52:32.802 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:52:32 smithi069 ceph-mon[24818]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' 2024-04-14T13:52:32.802 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:52:32 smithi069 ceph-mon[24818]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-14T13:52:32.802 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:52:32 smithi069 ceph-mon[24818]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T13:52:32.803 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:52:32 smithi069 ceph-mon[24818]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-14T13:52:32.803 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:52:32 smithi069 ceph-mon[24818]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' 2024-04-14T13:52:32.803 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:52:32 smithi069 ceph-mon[29472]: pgmap v39: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-04-14T13:52:32.803 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:52:32 smithi069 ceph-mon[29472]: Detected new or changed devices on smithi069 2024-04-14T13:52:32.803 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:52:32 smithi069 ceph-mon[29472]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' 2024-04-14T13:52:32.803 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:52:32 smithi069 ceph-mon[29472]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-14T13:52:32.803 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:52:32 smithi069 ceph-mon[29472]: Adjusting osd_memory_target on smithi069 to 16064M 2024-04-14T13:52:32.803 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:52:32 smithi069 ceph-mon[29472]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' 2024-04-14T13:52:32.803 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:52:32 smithi069 ceph-mon[29472]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' 2024-04-14T13:52:32.803 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:52:32 smithi069 ceph-mon[29472]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-14T13:52:32.803 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:52:32 smithi069 ceph-mon[29472]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T13:52:32.803 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:52:32 smithi069 ceph-mon[29472]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-14T13:52:32.803 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:52:32 smithi069 ceph-mon[29472]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' 2024-04-14T13:52:33.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:52:32 smithi122 ceph-mon[25721]: pgmap v39: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-04-14T13:52:33.096 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:52:32 smithi122 ceph-mon[25721]: Detected new or changed devices on smithi069 2024-04-14T13:52:33.096 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:52:32 smithi122 ceph-mon[25721]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' 2024-04-14T13:52:33.096 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:52:32 smithi122 ceph-mon[25721]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-14T13:52:33.096 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:52:32 smithi122 ceph-mon[25721]: Adjusting osd_memory_target on smithi069 to 16064M 2024-04-14T13:52:33.096 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:52:32 smithi122 ceph-mon[25721]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' 2024-04-14T13:52:33.096 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:52:32 smithi122 ceph-mon[25721]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' 2024-04-14T13:52:33.096 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:52:32 smithi122 ceph-mon[25721]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-14T13:52:33.096 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:52:32 smithi122 ceph-mon[25721]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T13:52:33.096 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:52:32 smithi122 ceph-mon[25721]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-14T13:52:33.096 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:52:32 smithi122 ceph-mon[25721]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' 2024-04-14T13:52:33.862 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:52:33 smithi069 ceph-mon[24818]: from='client.14244 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi069:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T13:52:33.863 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:52:33 smithi069 ceph-mon[24818]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-14T13:52:33.863 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:52:33 smithi069 ceph-mon[24818]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-14T13:52:33.863 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:52:33 smithi069 ceph-mon[24818]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T13:52:33.863 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:52:33 smithi069 ceph-mon[29472]: from='client.14244 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi069:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T13:52:33.863 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:52:33 smithi069 ceph-mon[29472]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-14T13:52:33.863 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:52:33 smithi069 ceph-mon[29472]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-14T13:52:33.863 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:52:33 smithi069 ceph-mon[29472]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T13:52:34.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:52:33 smithi122 ceph-mon[25721]: from='client.14244 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi069:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T13:52:34.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:52:33 smithi122 ceph-mon[25721]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-14T13:52:34.096 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:52:33 smithi122 ceph-mon[25721]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-14T13:52:34.096 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:52:33 smithi122 ceph-mon[25721]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T13:52:34.744 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:52:34 smithi069 ceph-mon[24818]: pgmap v40: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-04-14T13:52:34.744 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:52:34 smithi069 ceph-mon[29472]: pgmap v40: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-04-14T13:52:35.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:52:34 smithi122 ceph-mon[25721]: pgmap v40: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-04-14T13:52:36.567 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:52:36 smithi069 ceph-mon[24818]: from='client.? 172.21.15.69:0/2756024974' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "1d99b840-344d-422a-a466-13ed2d956280"}]: dispatch 2024-04-14T13:52:36.568 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:52:36 smithi069 ceph-mon[24818]: from='client.? 172.21.15.69:0/2756024974' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "1d99b840-344d-422a-a466-13ed2d956280"}]': finished 2024-04-14T13:52:36.568 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:52:36 smithi069 ceph-mon[24818]: osdmap e10: 2 total, 1 up, 2 in 2024-04-14T13:52:36.568 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:52:36 smithi069 ceph-mon[24818]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-14T13:52:36.568 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:52:36 smithi069 ceph-mon[24818]: from='client.? 172.21.15.69:0/1374856050' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-14T13:52:36.568 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:52:36 smithi069 ceph-mon[29472]: from='client.? 172.21.15.69:0/2756024974' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "1d99b840-344d-422a-a466-13ed2d956280"}]: dispatch 2024-04-14T13:52:36.569 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:52:36 smithi069 ceph-mon[29472]: from='client.? 172.21.15.69:0/2756024974' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "1d99b840-344d-422a-a466-13ed2d956280"}]': finished 2024-04-14T13:52:36.569 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:52:36 smithi069 ceph-mon[29472]: osdmap e10: 2 total, 1 up, 2 in 2024-04-14T13:52:36.569 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:52:36 smithi069 ceph-mon[29472]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-14T13:52:36.569 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:52:36 smithi069 ceph-mon[29472]: from='client.? 172.21.15.69:0/1374856050' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-14T13:52:36.595 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:52:36 smithi122 ceph-mon[25721]: from='client.? 172.21.15.69:0/2756024974' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "1d99b840-344d-422a-a466-13ed2d956280"}]: dispatch 2024-04-14T13:52:36.595 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:52:36 smithi122 ceph-mon[25721]: from='client.? 172.21.15.69:0/2756024974' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "1d99b840-344d-422a-a466-13ed2d956280"}]': finished 2024-04-14T13:52:36.596 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:52:36 smithi122 ceph-mon[25721]: osdmap e10: 2 total, 1 up, 2 in 2024-04-14T13:52:36.596 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:52:36 smithi122 ceph-mon[25721]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-14T13:52:36.596 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:52:36 smithi122 ceph-mon[25721]: from='client.? 172.21.15.69:0/1374856050' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-14T13:52:37.568 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:52:37 smithi069 ceph-mon[24818]: pgmap v42: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-04-14T13:52:37.568 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:52:37 smithi069 ceph-mon[29472]: pgmap v42: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-04-14T13:52:37.595 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:52:37 smithi122 ceph-mon[25721]: pgmap v42: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-04-14T13:52:39.468 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:52:39 smithi069 ceph-mon[24818]: pgmap v43: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-04-14T13:52:39.468 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:52:39 smithi069 ceph-mon[29472]: pgmap v43: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-04-14T13:52:39.595 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:52:39 smithi122 ceph-mon[25721]: pgmap v43: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-04-14T13:52:41.261 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:52:41 smithi069 ceph-mon[24818]: pgmap v44: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-04-14T13:52:41.261 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:52:41 smithi069 ceph-mon[29472]: pgmap v44: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-04-14T13:52:41.595 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:52:41 smithi122 ceph-mon[25721]: pgmap v44: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-04-14T13:52:42.317 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:52:42 smithi069 ceph-mon[24818]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-04-14T13:52:42.317 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:52:42 smithi069 ceph-mon[24818]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T13:52:42.318 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:52:42 smithi069 ceph-mon[24818]: Deploying daemon osd.1 on smithi069 2024-04-14T13:52:42.318 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:52:42 smithi069 ceph-mon[29472]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-04-14T13:52:42.318 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:52:42 smithi069 ceph-mon[29472]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T13:52:42.318 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:52:42 smithi069 ceph-mon[29472]: Deploying daemon osd.1 on smithi069 2024-04-14T13:52:42.595 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:52:42 smithi122 ceph-mon[25721]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-04-14T13:52:42.595 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:52:42 smithi122 ceph-mon[25721]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T13:52:42.595 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:52:42 smithi122 ceph-mon[25721]: Deploying daemon osd.1 on smithi069 2024-04-14T13:52:43.567 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:52:43 smithi069 ceph-mon[24818]: pgmap v45: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-04-14T13:52:43.567 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:52:43 smithi069 ceph-mon[29472]: pgmap v45: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-04-14T13:52:43.595 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:52:43 smithi122 ceph-mon[25721]: pgmap v45: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-04-14T13:52:45.318 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:52:45 smithi069 ceph-mon[24818]: pgmap v46: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-04-14T13:52:45.318 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:52:45 smithi069 ceph-mon[29472]: pgmap v46: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-04-14T13:52:45.595 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:52:45 smithi122 ceph-mon[25721]: pgmap v46: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-04-14T13:52:47.451 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:52:47 smithi069 ceph-mon[24818]: pgmap v47: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-04-14T13:52:47.451 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:52:47 smithi069 ceph-mon[24818]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' 2024-04-14T13:52:47.451 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:52:47 smithi069 ceph-mon[24818]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-14T13:52:47.451 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:52:47 smithi069 ceph-mon[24818]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T13:52:47.451 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:52:47 smithi069 ceph-mon[24818]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-14T13:52:47.451 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:52:47 smithi069 ceph-mon[29472]: pgmap v47: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-04-14T13:52:47.451 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:52:47 smithi069 ceph-mon[29472]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' 2024-04-14T13:52:47.451 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:52:47 smithi069 ceph-mon[29472]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-14T13:52:47.452 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:52:47 smithi069 ceph-mon[29472]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T13:52:47.452 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:52:47 smithi069 ceph-mon[29472]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-14T13:52:47.595 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:52:47 smithi122 ceph-mon[25721]: pgmap v47: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-04-14T13:52:47.595 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:52:47 smithi122 ceph-mon[25721]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' 2024-04-14T13:52:47.595 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:52:47 smithi122 ceph-mon[25721]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-14T13:52:47.595 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:52:47 smithi122 ceph-mon[25721]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T13:52:47.595 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:52:47 smithi122 ceph-mon[25721]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-14T13:52:49.036 INFO:teuthology.orchestra.run.smithi069.stdout:Created osd(s) 1 on host 'smithi069' 2024-04-14T13:52:49.317 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:52:49 smithi069 ceph-mon[24818]: pgmap v48: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-04-14T13:52:49.317 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:52:49 smithi069 ceph-mon[24818]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' 2024-04-14T13:52:49.317 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:52:49 smithi069 ceph-mon[29472]: pgmap v48: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-04-14T13:52:49.317 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:52:49 smithi069 ceph-mon[29472]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' 2024-04-14T13:52:49.595 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:52:49 smithi122 ceph-mon[25721]: pgmap v48: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-04-14T13:52:49.595 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:52:49 smithi122 ceph-mon[25721]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' 2024-04-14T13:52:49.640 DEBUG:teuthology.orchestra.run.smithi069:osd.1> sudo journalctl -f -n 0 -u ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968@osd.1.service 2024-04-14T13:52:49.643 INFO:tasks.cephadm:Deploying osd.2 on smithi069 with /dev/vg_nvme/lv_2... 2024-04-14T13:52:49.643 DEBUG:teuthology.orchestra.run.smithi069:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid dc326fc2-fa65-11ee-bc8e-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-04-14T13:52:50.496 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:52:50 smithi122 ceph-mon[25721]: from='osd.1 [v2:172.21.15.69:6810/1377309913,v1:172.21.15.69:6811/1377309913]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-04-14T13:52:50.496 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:52:50 smithi122 ceph-mon[25721]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' 2024-04-14T13:52:50.557 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:52:50 smithi069 ceph-mon[24818]: from='osd.1 [v2:172.21.15.69:6810/1377309913,v1:172.21.15.69:6811/1377309913]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-04-14T13:52:50.557 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:52:50 smithi069 ceph-mon[24818]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' 2024-04-14T13:52:50.557 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:52:50 smithi069 ceph-mon[29472]: from='osd.1 [v2:172.21.15.69:6810/1377309913,v1:172.21.15.69:6811/1377309913]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-04-14T13:52:50.557 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:52:50 smithi069 ceph-mon[29472]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' 2024-04-14T13:52:51.318 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:52:51 smithi069 ceph-mon[24818]: pgmap v49: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-04-14T13:52:51.318 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:52:51 smithi069 ceph-mon[24818]: from='osd.1 [v2:172.21.15.69:6810/1377309913,v1:172.21.15.69:6811/1377309913]' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-04-14T13:52:51.318 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:52:51 smithi069 ceph-mon[24818]: osdmap e11: 2 total, 1 up, 2 in 2024-04-14T13:52:51.318 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:52:51 smithi069 ceph-mon[24818]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-14T13:52:51.318 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:52:51 smithi069 ceph-mon[24818]: from='osd.1 [v2:172.21.15.69:6810/1377309913,v1:172.21.15.69:6811/1377309913]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi069", "root=default"]}]: dispatch 2024-04-14T13:52:51.319 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:52:51 smithi069 ceph-mon[29472]: pgmap v49: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-04-14T13:52:51.319 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:52:51 smithi069 ceph-mon[29472]: from='osd.1 [v2:172.21.15.69:6810/1377309913,v1:172.21.15.69:6811/1377309913]' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-04-14T13:52:51.319 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:52:51 smithi069 ceph-mon[29472]: osdmap e11: 2 total, 1 up, 2 in 2024-04-14T13:52:51.319 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:52:51 smithi069 ceph-mon[29472]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-14T13:52:51.319 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:52:51 smithi069 ceph-mon[29472]: from='osd.1 [v2:172.21.15.69:6810/1377309913,v1:172.21.15.69:6811/1377309913]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi069", "root=default"]}]: dispatch 2024-04-14T13:52:51.320 INFO:journalctl@ceph.osd.1.smithi069.stdout:Apr 14 13:52:51 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-osd-1[35300]: 2024-04-14T13:52:51.132+0000 7ff3a3288700 -1 osd.1 0 waiting for initial osdmap 2024-04-14T13:52:51.320 INFO:journalctl@ceph.osd.1.smithi069.stdout:Apr 14 13:52:51 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-osd-1[35300]: 2024-04-14T13:52:51.135+0000 7ff39dc1f700 -1 osd.1 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-14T13:52:51.550 INFO:teuthology.orchestra.run.smithi069.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-04-14T13:52:51.550 INFO:teuthology.orchestra.run.smithi069.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-04-14T13:52:51.551 INFO:teuthology.orchestra.run.smithi069.stderr: stderr: 10+0 records in 2024-04-14T13:52:51.551 INFO:teuthology.orchestra.run.smithi069.stderr:10+0 records out 2024-04-14T13:52:51.551 INFO:teuthology.orchestra.run.smithi069.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0127322 s, 824 MB/s 2024-04-14T13:52:51.551 INFO:teuthology.orchestra.run.smithi069.stderr:--> Zapping successful for: 2024-04-14T13:52:51.595 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:52:51 smithi122 ceph-mon[25721]: pgmap v49: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-04-14T13:52:51.595 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:52:51 smithi122 ceph-mon[25721]: from='osd.1 [v2:172.21.15.69:6810/1377309913,v1:172.21.15.69:6811/1377309913]' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-04-14T13:52:51.595 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:52:51 smithi122 ceph-mon[25721]: osdmap e11: 2 total, 1 up, 2 in 2024-04-14T13:52:51.595 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:52:51 smithi122 ceph-mon[25721]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-14T13:52:51.596 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:52:51 smithi122 ceph-mon[25721]: from='osd.1 [v2:172.21.15.69:6810/1377309913,v1:172.21.15.69:6811/1377309913]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi069", "root=default"]}]: dispatch 2024-04-14T13:52:52.166 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:52:52 smithi069 ceph-mon[24818]: from='osd.1 [v2:172.21.15.69:6810/1377309913,v1:172.21.15.69:6811/1377309913]' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi069", "root=default"]}]': finished 2024-04-14T13:52:52.166 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:52:52 smithi069 ceph-mon[24818]: osdmap e12: 2 total, 1 up, 2 in 2024-04-14T13:52:52.166 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:52:52 smithi069 ceph-mon[24818]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-14T13:52:52.166 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:52:52 smithi069 ceph-mon[24818]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-14T13:52:52.166 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:52:52 smithi069 ceph-mon[29472]: from='osd.1 [v2:172.21.15.69:6810/1377309913,v1:172.21.15.69:6811/1377309913]' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi069", "root=default"]}]': finished 2024-04-14T13:52:52.167 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:52:52 smithi069 ceph-mon[29472]: osdmap e12: 2 total, 1 up, 2 in 2024-04-14T13:52:52.167 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:52:52 smithi069 ceph-mon[29472]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-14T13:52:52.167 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:52:52 smithi069 ceph-mon[29472]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-14T13:52:52.188 DEBUG:teuthology.orchestra.run.smithi069:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid dc326fc2-fa65-11ee-bc8e-c7b262605968 -- ceph orch daemon add osd smithi069:vg_nvme/lv_2 2024-04-14T13:52:52.595 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:52:52 smithi122 ceph-mon[25721]: from='osd.1 [v2:172.21.15.69:6810/1377309913,v1:172.21.15.69:6811/1377309913]' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi069", "root=default"]}]': finished 2024-04-14T13:52:52.595 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:52:52 smithi122 ceph-mon[25721]: osdmap e12: 2 total, 1 up, 2 in 2024-04-14T13:52:52.595 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:52:52 smithi122 ceph-mon[25721]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-14T13:52:52.595 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:52:52 smithi122 ceph-mon[25721]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-14T13:52:53.223 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:52:53 smithi069 ceph-mon[24818]: purged_snaps scrub starts 2024-04-14T13:52:53.223 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:52:53 smithi069 ceph-mon[24818]: purged_snaps scrub ok 2024-04-14T13:52:53.223 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:52:53 smithi069 ceph-mon[24818]: pgmap v52: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-04-14T13:52:53.223 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:52:53 smithi069 ceph-mon[24818]: osd.1 [v2:172.21.15.69:6810/1377309913,v1:172.21.15.69:6811/1377309913] boot 2024-04-14T13:52:53.223 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:52:53 smithi069 ceph-mon[24818]: osdmap e13: 2 total, 2 up, 2 in 2024-04-14T13:52:53.223 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:52:53 smithi069 ceph-mon[24818]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-14T13:52:53.224 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:52:53 smithi069 ceph-mon[29472]: purged_snaps scrub starts 2024-04-14T13:52:53.224 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:52:53 smithi069 ceph-mon[29472]: purged_snaps scrub ok 2024-04-14T13:52:53.224 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:52:53 smithi069 ceph-mon[29472]: pgmap v52: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-04-14T13:52:53.224 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:52:53 smithi069 ceph-mon[29472]: osd.1 [v2:172.21.15.69:6810/1377309913,v1:172.21.15.69:6811/1377309913] boot 2024-04-14T13:52:53.224 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:52:53 smithi069 ceph-mon[29472]: osdmap e13: 2 total, 2 up, 2 in 2024-04-14T13:52:53.224 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:52:53 smithi069 ceph-mon[29472]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-14T13:52:53.595 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:52:53 smithi122 ceph-mon[25721]: purged_snaps scrub starts 2024-04-14T13:52:53.595 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:52:53 smithi122 ceph-mon[25721]: purged_snaps scrub ok 2024-04-14T13:52:53.595 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:52:53 smithi122 ceph-mon[25721]: pgmap v52: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-04-14T13:52:53.595 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:52:53 smithi122 ceph-mon[25721]: osd.1 [v2:172.21.15.69:6810/1377309913,v1:172.21.15.69:6811/1377309913] boot 2024-04-14T13:52:53.595 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:52:53 smithi122 ceph-mon[25721]: osdmap e13: 2 total, 2 up, 2 in 2024-04-14T13:52:53.595 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:52:53 smithi122 ceph-mon[25721]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-14T13:52:54.513 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:52:54 smithi069 ceph-mon[24818]: osdmap e14: 2 total, 2 up, 2 in 2024-04-14T13:52:54.513 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:52:54 smithi069 ceph-mon[29472]: osdmap e14: 2 total, 2 up, 2 in 2024-04-14T13:52:54.595 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:52:54 smithi122 ceph-mon[25721]: osdmap e14: 2 total, 2 up, 2 in 2024-04-14T13:52:55.526 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:52:55 smithi069 ceph-mon[24818]: pgmap v55: 0 pgs: ; 0 B data, 9.7 MiB used, 179 GiB / 179 GiB avail 2024-04-14T13:52:55.526 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:52:55 smithi069 ceph-mon[24818]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' 2024-04-14T13:52:55.526 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:52:55 smithi069 ceph-mon[24818]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-14T13:52:55.526 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:52:55 smithi069 ceph-mon[24818]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-14T13:52:55.526 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:52:55 smithi069 ceph-mon[24818]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' 2024-04-14T13:52:55.526 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:52:55 smithi069 ceph-mon[24818]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' 2024-04-14T13:52:55.526 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:52:55 smithi069 ceph-mon[24818]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-14T13:52:55.526 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:52:55 smithi069 ceph-mon[24818]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T13:52:55.526 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:52:55 smithi069 ceph-mon[24818]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-14T13:52:55.526 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:52:55 smithi069 ceph-mon[24818]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' 2024-04-14T13:52:55.526 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:52:55 smithi069 ceph-mon[29472]: pgmap v55: 0 pgs: ; 0 B data, 9.7 MiB used, 179 GiB / 179 GiB avail 2024-04-14T13:52:55.527 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:52:55 smithi069 ceph-mon[29472]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' 2024-04-14T13:52:55.527 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:52:55 smithi069 ceph-mon[29472]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-14T13:52:55.527 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:52:55 smithi069 ceph-mon[29472]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-14T13:52:55.527 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:52:55 smithi069 ceph-mon[29472]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' 2024-04-14T13:52:55.527 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:52:55 smithi069 ceph-mon[29472]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' 2024-04-14T13:52:55.527 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:52:55 smithi069 ceph-mon[29472]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-14T13:52:55.527 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:52:55 smithi069 ceph-mon[29472]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T13:52:55.527 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:52:55 smithi069 ceph-mon[29472]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-14T13:52:55.527 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:52:55 smithi069 ceph-mon[29472]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' 2024-04-14T13:52:55.595 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:52:55 smithi122 ceph-mon[25721]: pgmap v55: 0 pgs: ; 0 B data, 9.7 MiB used, 179 GiB / 179 GiB avail 2024-04-14T13:52:55.595 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:52:55 smithi122 ceph-mon[25721]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' 2024-04-14T13:52:55.595 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:52:55 smithi122 ceph-mon[25721]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-14T13:52:55.595 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:52:55 smithi122 ceph-mon[25721]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-14T13:52:55.596 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:52:55 smithi122 ceph-mon[25721]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' 2024-04-14T13:52:55.596 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:52:55 smithi122 ceph-mon[25721]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' 2024-04-14T13:52:55.596 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:52:55 smithi122 ceph-mon[25721]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-14T13:52:55.596 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:52:55 smithi122 ceph-mon[25721]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T13:52:55.596 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:52:55 smithi122 ceph-mon[25721]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-14T13:52:55.596 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:52:55 smithi122 ceph-mon[25721]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' 2024-04-14T13:52:56.567 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:52:56 smithi069 ceph-mon[24818]: Detected new or changed devices on smithi069 2024-04-14T13:52:56.567 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:52:56 smithi069 ceph-mon[24818]: Adjusting osd_memory_target on smithi069 to 8032M 2024-04-14T13:52:56.568 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:52:56 smithi069 ceph-mon[24818]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-14T13:52:56.568 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:52:56 smithi069 ceph-mon[24818]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-14T13:52:56.568 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:52:56 smithi069 ceph-mon[24818]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T13:52:56.568 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:52:56 smithi069 ceph-mon[29472]: Detected new or changed devices on smithi069 2024-04-14T13:52:56.568 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:52:56 smithi069 ceph-mon[29472]: Adjusting osd_memory_target on smithi069 to 8032M 2024-04-14T13:52:56.568 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:52:56 smithi069 ceph-mon[29472]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-14T13:52:56.568 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:52:56 smithi069 ceph-mon[29472]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-14T13:52:56.568 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:52:56 smithi069 ceph-mon[29472]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T13:52:56.595 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:52:56 smithi122 ceph-mon[25721]: Detected new or changed devices on smithi069 2024-04-14T13:52:56.595 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:52:56 smithi122 ceph-mon[25721]: Adjusting osd_memory_target on smithi069 to 8032M 2024-04-14T13:52:56.595 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:52:56 smithi122 ceph-mon[25721]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-14T13:52:56.595 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:52:56 smithi122 ceph-mon[25721]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-14T13:52:56.595 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:52:56 smithi122 ceph-mon[25721]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T13:52:57.567 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:52:57 smithi069 ceph-mon[29472]: pgmap v56: 0 pgs: ; 0 B data, 9.7 MiB used, 179 GiB / 179 GiB avail 2024-04-14T13:52:57.567 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:52:57 smithi069 ceph-mon[29472]: from='client.24175 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi069:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T13:52:57.568 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:52:57 smithi069 ceph-mon[24818]: pgmap v56: 0 pgs: ; 0 B data, 9.7 MiB used, 179 GiB / 179 GiB avail 2024-04-14T13:52:57.568 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:52:57 smithi069 ceph-mon[24818]: from='client.24175 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi069:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T13:52:57.595 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:52:57 smithi122 ceph-mon[25721]: pgmap v56: 0 pgs: ; 0 B data, 9.7 MiB used, 179 GiB / 179 GiB avail 2024-04-14T13:52:57.595 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:52:57 smithi122 ceph-mon[25721]: from='client.24175 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi069:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T13:52:58.567 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:52:58 smithi069 ceph-mon[24818]: from='client.? 172.21.15.69:0/2257974894' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "f4e0dcf9-5003-4a8f-8a51-fa36f58c0ac4"}]: dispatch 2024-04-14T13:52:58.567 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:52:58 smithi069 ceph-mon[24818]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "f4e0dcf9-5003-4a8f-8a51-fa36f58c0ac4"}]: dispatch 2024-04-14T13:52:58.568 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:52:58 smithi069 ceph-mon[24818]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "f4e0dcf9-5003-4a8f-8a51-fa36f58c0ac4"}]': finished 2024-04-14T13:52:58.568 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:52:58 smithi069 ceph-mon[24818]: osdmap e15: 3 total, 2 up, 3 in 2024-04-14T13:52:58.568 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:52:58 smithi069 ceph-mon[24818]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-14T13:52:58.568 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:52:58 smithi069 ceph-mon[29472]: from='client.? 172.21.15.69:0/2257974894' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "f4e0dcf9-5003-4a8f-8a51-fa36f58c0ac4"}]: dispatch 2024-04-14T13:52:58.568 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:52:58 smithi069 ceph-mon[29472]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "f4e0dcf9-5003-4a8f-8a51-fa36f58c0ac4"}]: dispatch 2024-04-14T13:52:58.568 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:52:58 smithi069 ceph-mon[29472]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "f4e0dcf9-5003-4a8f-8a51-fa36f58c0ac4"}]': finished 2024-04-14T13:52:58.568 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:52:58 smithi069 ceph-mon[29472]: osdmap e15: 3 total, 2 up, 3 in 2024-04-14T13:52:58.568 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:52:58 smithi069 ceph-mon[29472]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-14T13:52:58.595 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:52:58 smithi122 ceph-mon[25721]: from='client.? 172.21.15.69:0/2257974894' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "f4e0dcf9-5003-4a8f-8a51-fa36f58c0ac4"}]: dispatch 2024-04-14T13:52:58.595 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:52:58 smithi122 ceph-mon[25721]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "f4e0dcf9-5003-4a8f-8a51-fa36f58c0ac4"}]: dispatch 2024-04-14T13:52:58.595 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:52:58 smithi122 ceph-mon[25721]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "f4e0dcf9-5003-4a8f-8a51-fa36f58c0ac4"}]': finished 2024-04-14T13:52:58.595 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:52:58 smithi122 ceph-mon[25721]: osdmap e15: 3 total, 2 up, 3 in 2024-04-14T13:52:58.595 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:52:58 smithi122 ceph-mon[25721]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-14T13:52:59.567 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:52:59 smithi069 ceph-mon[24818]: pgmap v57: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-04-14T13:52:59.567 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:52:59 smithi069 ceph-mon[24818]: from='client.? 172.21.15.69:0/4056135015' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-14T13:52:59.567 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:52:59 smithi069 ceph-mon[29472]: pgmap v57: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-04-14T13:52:59.567 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:52:59 smithi069 ceph-mon[29472]: from='client.? 172.21.15.69:0/4056135015' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-14T13:52:59.595 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:52:59 smithi122 ceph-mon[25721]: pgmap v57: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-04-14T13:52:59.595 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:52:59 smithi122 ceph-mon[25721]: from='client.? 172.21.15.69:0/4056135015' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-14T13:53:01.317 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:53:01 smithi069 ceph-mon[24818]: pgmap v59: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-04-14T13:53:01.318 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:53:01 smithi069 ceph-mon[29472]: pgmap v59: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-04-14T13:53:01.595 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:53:01 smithi122 ceph-mon[25721]: pgmap v59: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-04-14T13:53:03.386 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:53:03 smithi069 ceph-mon[24818]: pgmap v60: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-04-14T13:53:03.386 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:53:03 smithi069 ceph-mon[29472]: pgmap v60: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-04-14T13:53:03.595 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:53:03 smithi122 ceph-mon[25721]: pgmap v60: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-04-14T13:53:04.450 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:53:04 smithi069 ceph-mon[24818]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-04-14T13:53:04.450 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:53:04 smithi069 ceph-mon[24818]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T13:53:04.450 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:53:04 smithi069 ceph-mon[29472]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-04-14T13:53:04.450 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:53:04 smithi069 ceph-mon[29472]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T13:53:04.595 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:53:04 smithi122 ceph-mon[25721]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-04-14T13:53:04.595 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:53:04 smithi122 ceph-mon[25721]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T13:53:05.317 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:53:05 smithi069 ceph-mon[24818]: pgmap v61: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-04-14T13:53:05.317 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:53:05 smithi069 ceph-mon[24818]: Deploying daemon osd.2 on smithi069 2024-04-14T13:53:05.318 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:53:05 smithi069 ceph-mon[29472]: pgmap v61: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-04-14T13:53:05.318 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:53:05 smithi069 ceph-mon[29472]: Deploying daemon osd.2 on smithi069 2024-04-14T13:53:05.595 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:53:05 smithi122 ceph-mon[25721]: pgmap v61: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-04-14T13:53:05.595 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:53:05 smithi122 ceph-mon[25721]: Deploying daemon osd.2 on smithi069 2024-04-14T13:53:07.465 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:53:07 smithi069 ceph-mon[24818]: pgmap v62: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-04-14T13:53:07.465 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:53:07 smithi069 ceph-mon[29472]: pgmap v62: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-04-14T13:53:07.595 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:53:07 smithi122 ceph-mon[25721]: pgmap v62: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-04-14T13:53:09.359 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:53:09 smithi069 ceph-mon[24818]: pgmap v63: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-04-14T13:53:09.360 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:53:09 smithi069 ceph-mon[24818]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' 2024-04-14T13:53:09.360 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:53:09 smithi069 ceph-mon[24818]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-14T13:53:09.360 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:53:09 smithi069 ceph-mon[24818]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T13:53:09.360 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:53:09 smithi069 ceph-mon[24818]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-14T13:53:09.360 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:53:09 smithi069 ceph-mon[29472]: pgmap v63: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-04-14T13:53:09.360 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:53:09 smithi069 ceph-mon[29472]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' 2024-04-14T13:53:09.360 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:53:09 smithi069 ceph-mon[29472]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-14T13:53:09.360 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:53:09 smithi069 ceph-mon[29472]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T13:53:09.360 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:53:09 smithi069 ceph-mon[29472]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-14T13:53:09.595 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:53:09 smithi122 ceph-mon[25721]: pgmap v63: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-04-14T13:53:09.595 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:53:09 smithi122 ceph-mon[25721]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' 2024-04-14T13:53:09.595 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:53:09 smithi122 ceph-mon[25721]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-14T13:53:09.595 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:53:09 smithi122 ceph-mon[25721]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T13:53:09.595 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:53:09 smithi122 ceph-mon[25721]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-14T13:53:11.317 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:53:11 smithi069 ceph-mon[29472]: pgmap v64: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-04-14T13:53:11.318 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:53:11 smithi069 ceph-mon[24818]: pgmap v64: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-04-14T13:53:11.595 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:53:11 smithi122 ceph-mon[25721]: pgmap v64: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-04-14T13:53:11.798 INFO:teuthology.orchestra.run.smithi069.stdout:Created osd(s) 2 on host 'smithi069' 2024-04-14T13:53:12.394 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:53:12 smithi069 ceph-mon[29472]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-04-14T13:53:12.394 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:53:12 smithi069 ceph-mon[29472]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-04-14T13:53:12.394 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:53:12 smithi069 ceph-mon[29472]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' 2024-04-14T13:53:12.394 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:53:12 smithi069 ceph-mon[29472]: from='osd.2 [v2:172.21.15.69:6818/2090584066,v1:172.21.15.69:6819/2090584066]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-04-14T13:53:12.394 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:53:12 smithi069 ceph-mon[29472]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-04-14T13:53:12.394 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:53:12 smithi069 ceph-mon[24818]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-04-14T13:53:12.394 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:53:12 smithi069 ceph-mon[24818]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-04-14T13:53:12.394 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:53:12 smithi069 ceph-mon[24818]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' 2024-04-14T13:53:12.395 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:53:12 smithi069 ceph-mon[24818]: from='osd.2 [v2:172.21.15.69:6818/2090584066,v1:172.21.15.69:6819/2090584066]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-04-14T13:53:12.395 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:53:12 smithi069 ceph-mon[24818]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-04-14T13:53:12.396 DEBUG:teuthology.orchestra.run.smithi069:osd.2> sudo journalctl -f -n 0 -u ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968@osd.2.service 2024-04-14T13:53:12.398 INFO:tasks.cephadm:Deploying osd.3 on smithi069 with /dev/vg_nvme/lv_1... 2024-04-14T13:53:12.399 DEBUG:teuthology.orchestra.run.smithi069:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid dc326fc2-fa65-11ee-bc8e-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2024-04-14T13:53:12.595 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:53:12 smithi122 ceph-mon[25721]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-04-14T13:53:12.595 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:53:12 smithi122 ceph-mon[25721]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-04-14T13:53:12.595 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:53:12 smithi122 ceph-mon[25721]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' 2024-04-14T13:53:12.595 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:53:12 smithi122 ceph-mon[25721]: from='osd.2 [v2:172.21.15.69:6818/2090584066,v1:172.21.15.69:6819/2090584066]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-04-14T13:53:12.595 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:53:12 smithi122 ceph-mon[25721]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-04-14T13:53:13.317 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:53:13 smithi069 ceph-mon[24818]: pgmap v65: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-04-14T13:53:13.317 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:53:13 smithi069 ceph-mon[24818]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' 2024-04-14T13:53:13.318 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:53:13 smithi069 ceph-mon[24818]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-04-14T13:53:13.318 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:53:13 smithi069 ceph-mon[24818]: osdmap e16: 3 total, 2 up, 3 in 2024-04-14T13:53:13.318 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:53:13 smithi069 ceph-mon[24818]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-14T13:53:13.318 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:53:13 smithi069 ceph-mon[24818]: from='osd.2 [v2:172.21.15.69:6818/2090584066,v1:172.21.15.69:6819/2090584066]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi069", "root=default"]}]: dispatch 2024-04-14T13:53:13.318 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:53:13 smithi069 ceph-mon[24818]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi069", "root=default"]}]: dispatch 2024-04-14T13:53:13.318 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:53:13 smithi069 ceph-mon[29472]: pgmap v65: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-04-14T13:53:13.318 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:53:13 smithi069 ceph-mon[29472]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' 2024-04-14T13:53:13.318 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:53:13 smithi069 ceph-mon[29472]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-04-14T13:53:13.318 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:53:13 smithi069 ceph-mon[29472]: osdmap e16: 3 total, 2 up, 3 in 2024-04-14T13:53:13.318 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:53:13 smithi069 ceph-mon[29472]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-14T13:53:13.318 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:53:13 smithi069 ceph-mon[29472]: from='osd.2 [v2:172.21.15.69:6818/2090584066,v1:172.21.15.69:6819/2090584066]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi069", "root=default"]}]: dispatch 2024-04-14T13:53:13.318 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:53:13 smithi069 ceph-mon[29472]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi069", "root=default"]}]: dispatch 2024-04-14T13:53:13.595 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:53:13 smithi122 ceph-mon[25721]: pgmap v65: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-04-14T13:53:13.595 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:53:13 smithi122 ceph-mon[25721]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' 2024-04-14T13:53:13.595 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:53:13 smithi122 ceph-mon[25721]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-04-14T13:53:13.595 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:53:13 smithi122 ceph-mon[25721]: osdmap e16: 3 total, 2 up, 3 in 2024-04-14T13:53:13.596 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:53:13 smithi122 ceph-mon[25721]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-14T13:53:13.596 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:53:13 smithi122 ceph-mon[25721]: from='osd.2 [v2:172.21.15.69:6818/2090584066,v1:172.21.15.69:6819/2090584066]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi069", "root=default"]}]: dispatch 2024-04-14T13:53:13.596 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:53:13 smithi122 ceph-mon[25721]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi069", "root=default"]}]: dispatch 2024-04-14T13:53:14.056 INFO:journalctl@ceph.osd.2.smithi069.stdout:Apr 14 13:53:13 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-osd-2[38223]: 2024-04-14T13:53:13.804+0000 7fce06af6700 -1 osd.2 0 waiting for initial osdmap 2024-04-14T13:53:14.057 INFO:journalctl@ceph.osd.2.smithi069.stdout:Apr 14 13:53:13 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-osd-2[38223]: 2024-04-14T13:53:13.809+0000 7fce0248f700 -1 osd.2 17 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-14T13:53:14.764 INFO:teuthology.orchestra.run.smithi069.stderr:--> Zapping: /dev/vg_nvme/lv_1 2024-04-14T13:53:14.765 INFO:teuthology.orchestra.run.smithi069.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2024-04-14T13:53:14.765 INFO:teuthology.orchestra.run.smithi069.stderr: stderr: 10+0 records in 2024-04-14T13:53:14.765 INFO:teuthology.orchestra.run.smithi069.stderr:10+0 records out 2024-04-14T13:53:14.765 INFO:teuthology.orchestra.run.smithi069.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0121444 s, 863 MB/s 2024-04-14T13:53:14.765 INFO:teuthology.orchestra.run.smithi069.stderr:--> Zapping successful for: 2024-04-14T13:53:15.067 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:53:14 smithi069 ceph-mon[29472]: pgmap v67: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-04-14T13:53:15.067 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:53:14 smithi069 ceph-mon[29472]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi069", "root=default"]}]': finished 2024-04-14T13:53:15.067 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:53:14 smithi069 ceph-mon[29472]: osdmap e17: 3 total, 2 up, 3 in 2024-04-14T13:53:15.067 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:53:14 smithi069 ceph-mon[29472]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-14T13:53:15.067 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:53:14 smithi069 ceph-mon[29472]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-14T13:53:15.068 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:53:14 smithi069 ceph-mon[24818]: pgmap v67: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-04-14T13:53:15.068 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:53:14 smithi069 ceph-mon[24818]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi069", "root=default"]}]': finished 2024-04-14T13:53:15.068 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:53:14 smithi069 ceph-mon[24818]: osdmap e17: 3 total, 2 up, 3 in 2024-04-14T13:53:15.068 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:53:14 smithi069 ceph-mon[24818]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-14T13:53:15.068 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:53:14 smithi069 ceph-mon[24818]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-14T13:53:15.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:53:14 smithi122 ceph-mon[25721]: pgmap v67: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-04-14T13:53:15.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:53:14 smithi122 ceph-mon[25721]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi069", "root=default"]}]': finished 2024-04-14T13:53:15.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:53:14 smithi122 ceph-mon[25721]: osdmap e17: 3 total, 2 up, 3 in 2024-04-14T13:53:15.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:53:14 smithi122 ceph-mon[25721]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-14T13:53:15.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:53:14 smithi122 ceph-mon[25721]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-14T13:53:15.420 DEBUG:teuthology.orchestra.run.smithi069:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid dc326fc2-fa65-11ee-bc8e-c7b262605968 -- ceph orch daemon add osd smithi069:vg_nvme/lv_1 2024-04-14T13:53:16.067 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:53:15 smithi069 ceph-mon[24818]: purged_snaps scrub starts 2024-04-14T13:53:16.067 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:53:15 smithi069 ceph-mon[24818]: purged_snaps scrub ok 2024-04-14T13:53:16.067 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:53:15 smithi069 ceph-mon[24818]: osd.2 [v2:172.21.15.69:6818/2090584066,v1:172.21.15.69:6819/2090584066] boot 2024-04-14T13:53:16.068 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:53:15 smithi069 ceph-mon[24818]: osdmap e18: 3 total, 3 up, 3 in 2024-04-14T13:53:16.068 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:53:15 smithi069 ceph-mon[24818]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-14T13:53:16.068 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:53:15 smithi069 ceph-mon[24818]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32}]: dispatch 2024-04-14T13:53:16.068 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:53:15 smithi069 ceph-mon[29472]: purged_snaps scrub starts 2024-04-14T13:53:16.068 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:53:15 smithi069 ceph-mon[29472]: purged_snaps scrub ok 2024-04-14T13:53:16.068 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:53:15 smithi069 ceph-mon[29472]: osd.2 [v2:172.21.15.69:6818/2090584066,v1:172.21.15.69:6819/2090584066] boot 2024-04-14T13:53:16.068 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:53:15 smithi069 ceph-mon[29472]: osdmap e18: 3 total, 3 up, 3 in 2024-04-14T13:53:16.068 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:53:15 smithi069 ceph-mon[29472]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-14T13:53:16.068 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:53:15 smithi069 ceph-mon[29472]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32}]: dispatch 2024-04-14T13:53:16.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:53:15 smithi122 ceph-mon[25721]: purged_snaps scrub starts 2024-04-14T13:53:16.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:53:15 smithi122 ceph-mon[25721]: purged_snaps scrub ok 2024-04-14T13:53:16.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:53:15 smithi122 ceph-mon[25721]: osd.2 [v2:172.21.15.69:6818/2090584066,v1:172.21.15.69:6819/2090584066] boot 2024-04-14T13:53:16.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:53:15 smithi122 ceph-mon[25721]: osdmap e18: 3 total, 3 up, 3 in 2024-04-14T13:53:16.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:53:15 smithi122 ceph-mon[25721]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-14T13:53:16.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:53:15 smithi122 ceph-mon[25721]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32}]: dispatch 2024-04-14T13:53:17.434 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:53:17 smithi069 ceph-mon[24818]: pgmap v70: 0 pgs: ; 0 B data, 15 MiB used, 268 GiB / 268 GiB avail 2024-04-14T13:53:17.434 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:53:17 smithi069 ceph-mon[24818]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32}]': finished 2024-04-14T13:53:17.434 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:53:17 smithi069 ceph-mon[24818]: osdmap e19: 3 total, 3 up, 3 in 2024-04-14T13:53:17.434 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:53:17 smithi069 ceph-mon[24818]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-04-14T13:53:17.435 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:53:17 smithi069 ceph-mon[29472]: pgmap v70: 0 pgs: ; 0 B data, 15 MiB used, 268 GiB / 268 GiB avail 2024-04-14T13:53:17.435 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:53:17 smithi069 ceph-mon[29472]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32}]': finished 2024-04-14T13:53:17.435 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:53:17 smithi069 ceph-mon[29472]: osdmap e19: 3 total, 3 up, 3 in 2024-04-14T13:53:17.435 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:53:17 smithi069 ceph-mon[29472]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-04-14T13:53:17.595 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:53:17 smithi122 ceph-mon[25721]: pgmap v70: 0 pgs: ; 0 B data, 15 MiB used, 268 GiB / 268 GiB avail 2024-04-14T13:53:17.595 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:53:17 smithi122 ceph-mon[25721]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32}]': finished 2024-04-14T13:53:17.595 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:53:17 smithi122 ceph-mon[25721]: osdmap e19: 3 total, 3 up, 3 in 2024-04-14T13:53:17.595 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:53:17 smithi122 ceph-mon[25721]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-04-14T13:53:17.745 INFO:journalctl@ceph.osd.0.smithi069.stdout:Apr 14 13:53:17 smithi069 sudo[39507]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/smartctl -x --json=o /dev/nvme0n1 2024-04-14T13:53:17.745 INFO:journalctl@ceph.osd.0.smithi069.stdout:Apr 14 13:53:17 smithi069 sudo[39507]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-04-14T13:53:17.745 INFO:journalctl@ceph.osd.0.smithi069.stdout:Apr 14 13:53:17 smithi069 sudo[39507]: pam_unix(sudo:session): session closed for user root 2024-04-14T13:53:18.017 INFO:journalctl@ceph.osd.0.smithi069.stdout:Apr 14 13:53:17 smithi069 sudo[39514]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-04-14T13:53:18.017 INFO:journalctl@ceph.osd.0.smithi069.stdout:Apr 14 13:53:17 smithi069 sudo[39514]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-04-14T13:53:18.017 INFO:journalctl@ceph.osd.0.smithi069.stdout:Apr 14 13:53:17 smithi069 sudo[39514]: pam_unix(sudo:session): session closed for user root 2024-04-14T13:53:18.281 INFO:journalctl@ceph.osd.1.smithi069.stdout:Apr 14 13:53:18 smithi069 sudo[39536]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/smartctl -x --json=o /dev/nvme0n1 2024-04-14T13:53:18.281 INFO:journalctl@ceph.osd.1.smithi069.stdout:Apr 14 13:53:18 smithi069 sudo[39536]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-04-14T13:53:18.281 INFO:journalctl@ceph.osd.1.smithi069.stdout:Apr 14 13:53:18 smithi069 sudo[39536]: pam_unix(sudo:session): session closed for user root 2024-04-14T13:53:18.281 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:53:18 smithi069 ceph-mon[24818]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-04-14T13:53:18.282 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:53:18 smithi069 ceph-mon[24818]: osdmap e20: 3 total, 3 up, 3 in 2024-04-14T13:53:18.282 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:53:18 smithi069 ceph-mon[24818]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' 2024-04-14T13:53:18.282 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:53:18 smithi069 ceph-mon[24818]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-14T13:53:18.282 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:53:18 smithi069 ceph-mon[24818]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-14T13:53:18.282 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:53:18 smithi069 ceph-mon[24818]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-14T13:53:18.282 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:53:18 smithi069 ceph-mon[24818]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' 2024-04-14T13:53:18.282 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:53:18 smithi069 ceph-mon[24818]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' 2024-04-14T13:53:18.282 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:53:18 smithi069 ceph-mon[24818]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-14T13:53:18.282 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:53:18 smithi069 ceph-mon[24818]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T13:53:18.282 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:53:18 smithi069 ceph-mon[24818]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-14T13:53:18.282 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:53:18 smithi069 ceph-mon[24818]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' 2024-04-14T13:53:18.282 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:53:18 smithi069 ceph-mon[24818]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' 2024-04-14T13:53:18.283 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:53:18 smithi069 ceph-mon[29472]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-04-14T13:53:18.283 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:53:18 smithi069 ceph-mon[29472]: osdmap e20: 3 total, 3 up, 3 in 2024-04-14T13:53:18.283 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:53:18 smithi069 ceph-mon[29472]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' 2024-04-14T13:53:18.283 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:53:18 smithi069 ceph-mon[29472]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-14T13:53:18.283 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:53:18 smithi069 ceph-mon[29472]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-14T13:53:18.283 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:53:18 smithi069 ceph-mon[29472]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-14T13:53:18.283 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:53:18 smithi069 ceph-mon[29472]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' 2024-04-14T13:53:18.283 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:53:18 smithi069 ceph-mon[29472]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' 2024-04-14T13:53:18.283 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:53:18 smithi069 ceph-mon[29472]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-14T13:53:18.283 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:53:18 smithi069 ceph-mon[29472]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T13:53:18.283 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:53:18 smithi069 ceph-mon[29472]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-14T13:53:18.283 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:53:18 smithi069 ceph-mon[29472]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' 2024-04-14T13:53:18.283 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:53:18 smithi069 ceph-mon[29472]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' 2024-04-14T13:53:18.531 INFO:journalctl@ceph.osd.2.smithi069.stdout:Apr 14 13:53:18 smithi069 sudo[39603]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/smartctl -x --json=o /dev/nvme0n1 2024-04-14T13:53:18.531 INFO:journalctl@ceph.osd.1.smithi069.stdout:Apr 14 13:53:18 smithi069 sudo[39563]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-04-14T13:53:18.531 INFO:journalctl@ceph.osd.1.smithi069.stdout:Apr 14 13:53:18 smithi069 sudo[39563]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-04-14T13:53:18.531 INFO:journalctl@ceph.osd.1.smithi069.stdout:Apr 14 13:53:18 smithi069 sudo[39563]: pam_unix(sudo:session): session closed for user root 2024-04-14T13:53:18.595 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:53:18 smithi122 ceph-mon[25721]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-04-14T13:53:18.595 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:53:18 smithi122 ceph-mon[25721]: osdmap e20: 3 total, 3 up, 3 in 2024-04-14T13:53:18.595 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:53:18 smithi122 ceph-mon[25721]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' 2024-04-14T13:53:18.596 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:53:18 smithi122 ceph-mon[25721]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-14T13:53:18.596 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:53:18 smithi122 ceph-mon[25721]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-14T13:53:18.596 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:53:18 smithi122 ceph-mon[25721]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-14T13:53:18.596 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:53:18 smithi122 ceph-mon[25721]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' 2024-04-14T13:53:18.596 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:53:18 smithi122 ceph-mon[25721]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' 2024-04-14T13:53:18.596 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:53:18 smithi122 ceph-mon[25721]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-14T13:53:18.596 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:53:18 smithi122 ceph-mon[25721]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T13:53:18.596 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:53:18 smithi122 ceph-mon[25721]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-14T13:53:18.596 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:53:18 smithi122 ceph-mon[25721]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' 2024-04-14T13:53:18.596 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:53:18 smithi122 ceph-mon[25721]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' 2024-04-14T13:53:18.793 INFO:journalctl@ceph.osd.2.smithi069.stdout:Apr 14 13:53:18 smithi069 sudo[39603]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-04-14T13:53:18.794 INFO:journalctl@ceph.osd.2.smithi069.stdout:Apr 14 13:53:18 smithi069 sudo[39603]: pam_unix(sudo:session): session closed for user root 2024-04-14T13:53:19.054 INFO:journalctl@ceph.osd.2.smithi069.stdout:Apr 14 13:53:18 smithi069 sudo[39648]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-04-14T13:53:19.055 INFO:journalctl@ceph.osd.2.smithi069.stdout:Apr 14 13:53:18 smithi069 sudo[39648]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-04-14T13:53:19.055 INFO:journalctl@ceph.osd.2.smithi069.stdout:Apr 14 13:53:18 smithi069 sudo[39648]: pam_unix(sudo:session): session closed for user root 2024-04-14T13:53:19.055 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:53:19 smithi069 sudo[39655]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/smartctl -x --json=o /dev/nvme0n1 2024-04-14T13:53:19.055 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:53:19 smithi069 sudo[39655]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-04-14T13:53:19.055 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:53:19 smithi069 sudo[39655]: pam_unix(sudo:session): session closed for user root 2024-04-14T13:53:19.317 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:53:19 smithi069 ceph-mon[29472]: Detected new or changed devices on smithi069 2024-04-14T13:53:19.317 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:53:19 smithi069 ceph-mon[29472]: Adjusting osd_memory_target on smithi069 to 5354M 2024-04-14T13:53:19.317 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:53:19 smithi069 ceph-mon[29472]: pgmap v73: 1 pgs: 1 unknown; 0 B data, 15 MiB used, 268 GiB / 268 GiB avail 2024-04-14T13:53:19.317 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:53:19 smithi069 ceph-mon[29472]: osdmap e21: 3 total, 3 up, 3 in 2024-04-14T13:53:19.317 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:53:19 smithi069 ceph-mon[29472]: mgrmap e15: y(active, since 2m), standbys: x 2024-04-14T13:53:19.318 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:53:19 smithi069 ceph-mon[29472]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-14T13:53:19.318 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:53:19 smithi069 ceph-mon[29472]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-14T13:53:19.318 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:53:19 smithi069 ceph-mon[29472]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T13:53:19.318 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:53:19 smithi069 sudo[39671]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-04-14T13:53:19.318 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:53:19 smithi069 sudo[39671]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-04-14T13:53:19.318 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:53:19 smithi069 sudo[39671]: pam_unix(sudo:session): session closed for user root 2024-04-14T13:53:19.318 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:53:19 smithi069 ceph-mon[24818]: Detected new or changed devices on smithi069 2024-04-14T13:53:19.318 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:53:19 smithi069 ceph-mon[24818]: Adjusting osd_memory_target on smithi069 to 5354M 2024-04-14T13:53:19.318 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:53:19 smithi069 ceph-mon[24818]: pgmap v73: 1 pgs: 1 unknown; 0 B data, 15 MiB used, 268 GiB / 268 GiB avail 2024-04-14T13:53:19.318 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:53:19 smithi069 ceph-mon[24818]: osdmap e21: 3 total, 3 up, 3 in 2024-04-14T13:53:19.318 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:53:19 smithi069 ceph-mon[24818]: mgrmap e15: y(active, since 2m), standbys: x 2024-04-14T13:53:19.319 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:53:19 smithi069 ceph-mon[24818]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-14T13:53:19.319 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:53:19 smithi069 ceph-mon[24818]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-14T13:53:19.319 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:53:19 smithi069 ceph-mon[24818]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T13:53:19.595 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:53:19 smithi122 ceph-mon[25721]: Detected new or changed devices on smithi069 2024-04-14T13:53:19.595 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:53:19 smithi122 ceph-mon[25721]: Adjusting osd_memory_target on smithi069 to 5354M 2024-04-14T13:53:19.595 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:53:19 smithi122 ceph-mon[25721]: pgmap v73: 1 pgs: 1 unknown; 0 B data, 15 MiB used, 268 GiB / 268 GiB avail 2024-04-14T13:53:19.596 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:53:19 smithi122 ceph-mon[25721]: osdmap e21: 3 total, 3 up, 3 in 2024-04-14T13:53:19.596 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:53:19 smithi122 ceph-mon[25721]: mgrmap e15: y(active, since 2m), standbys: x 2024-04-14T13:53:19.596 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:53:19 smithi122 ceph-mon[25721]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-14T13:53:19.596 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:53:19 smithi122 ceph-mon[25721]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-14T13:53:19.596 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:53:19 smithi122 ceph-mon[25721]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T13:53:19.797 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:53:19 smithi069 sudo[39677]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/smartctl -x --json=o /dev/nvme0n1 2024-04-14T13:53:19.797 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:53:19 smithi069 sudo[39677]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-04-14T13:53:19.797 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:53:19 smithi069 sudo[39677]: pam_unix(sudo:session): session closed for user root 2024-04-14T13:53:20.058 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:53:19 smithi069 sudo[39681]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-04-14T13:53:20.059 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:53:19 smithi069 sudo[39681]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-04-14T13:53:20.059 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:53:19 smithi069 sudo[39681]: pam_unix(sudo:session): session closed for user root 2024-04-14T13:53:20.297 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:53:20 smithi122 sudo[27240]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/smartctl -x --json=o /dev/nvme0n1 2024-04-14T13:53:20.297 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:53:20 smithi122 sudo[27240]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-04-14T13:53:20.297 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:53:20 smithi122 sudo[27240]: pam_unix(sudo:session): session closed for user root 2024-04-14T13:53:20.311 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:53:20 smithi069 ceph-mon[29472]: from='client.24208 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi069:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T13:53:20.311 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:53:20 smithi069 ceph-mon[29472]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-14T13:53:20.311 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:53:20 smithi069 ceph-mon[29472]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-14T13:53:20.311 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:53:20 smithi069 ceph-mon[29472]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-14T13:53:20.311 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:53:20 smithi069 ceph-mon[29472]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-14T13:53:20.311 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:53:20 smithi069 ceph-mon[29472]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-14T13:53:20.311 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:53:20 smithi069 ceph-mon[29472]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-14T13:53:20.311 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:53:20 smithi069 ceph-mon[29472]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-14T13:53:20.311 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:53:20 smithi069 ceph-mon[29472]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-14T13:53:20.311 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:53:20 smithi069 ceph-mon[29472]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-14T13:53:20.311 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:53:20 smithi069 ceph-mon[29472]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-14T13:53:20.312 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:53:20 smithi069 ceph-mon[29472]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-14T13:53:20.312 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:53:20 smithi069 ceph-mon[29472]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-14T13:53:20.312 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:53:20 smithi069 ceph-mon[29472]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-14T13:53:20.312 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:53:20 smithi069 ceph-mon[29472]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-14T13:53:20.567 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:53:20 smithi069 ceph-mon[24818]: from='client.24208 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi069:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T13:53:20.567 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:53:20 smithi069 ceph-mon[24818]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-14T13:53:20.568 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:53:20 smithi069 ceph-mon[24818]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-14T13:53:20.568 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:53:20 smithi069 ceph-mon[24818]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-14T13:53:20.568 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:53:20 smithi069 ceph-mon[24818]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-14T13:53:20.568 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:53:20 smithi069 ceph-mon[24818]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-14T13:53:20.568 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:53:20 smithi069 ceph-mon[24818]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-14T13:53:20.568 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:53:20 smithi069 ceph-mon[24818]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-14T13:53:20.568 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:53:20 smithi069 ceph-mon[24818]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-14T13:53:20.568 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:53:20 smithi069 ceph-mon[24818]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-14T13:53:20.568 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:53:20 smithi069 ceph-mon[24818]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-14T13:53:20.568 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:53:20 smithi069 ceph-mon[24818]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-14T13:53:20.568 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:53:20 smithi069 ceph-mon[24818]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-14T13:53:20.568 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:53:20 smithi069 ceph-mon[24818]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-14T13:53:20.568 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:53:20 smithi069 ceph-mon[24818]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-14T13:53:20.595 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:53:20 smithi122 sudo[27243]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-04-14T13:53:20.595 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:53:20 smithi122 sudo[27243]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-04-14T13:53:20.595 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:53:20 smithi122 sudo[27243]: pam_unix(sudo:session): session closed for user root 2024-04-14T13:53:20.595 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:53:20 smithi122 ceph-mon[25721]: from='client.24208 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi069:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T13:53:20.595 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:53:20 smithi122 ceph-mon[25721]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-14T13:53:20.595 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:53:20 smithi122 ceph-mon[25721]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-14T13:53:20.595 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:53:20 smithi122 ceph-mon[25721]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-14T13:53:20.595 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:53:20 smithi122 ceph-mon[25721]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-14T13:53:20.596 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:53:20 smithi122 ceph-mon[25721]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-14T13:53:20.596 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:53:20 smithi122 ceph-mon[25721]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-14T13:53:20.596 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:53:20 smithi122 ceph-mon[25721]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-14T13:53:20.596 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:53:20 smithi122 ceph-mon[25721]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-14T13:53:20.596 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:53:20 smithi122 ceph-mon[25721]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-14T13:53:20.596 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:53:20 smithi122 ceph-mon[25721]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-14T13:53:20.596 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:53:20 smithi122 ceph-mon[25721]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-14T13:53:20.596 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:53:20 smithi122 ceph-mon[25721]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-14T13:53:20.596 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:53:20 smithi122 ceph-mon[25721]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-14T13:53:20.596 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:53:20 smithi122 ceph-mon[25721]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-14T13:53:21.567 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:53:21 smithi069 ceph-mon[29472]: pgmap v75: 1 pgs: 1 active+clean; 577 KiB data, 15 MiB used, 268 GiB / 268 GiB avail 2024-04-14T13:53:21.567 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:53:21 smithi069 ceph-mon[29472]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-14T13:53:21.567 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:53:21 smithi069 ceph-mon[29472]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' 2024-04-14T13:53:21.568 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:53:21 smithi069 ceph-mon[29472]: from='client.? 172.21.15.69:0/2464956847' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "27e543e2-f5c0-462d-ab23-d439e06c056e"}]: dispatch 2024-04-14T13:53:21.568 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:53:21 smithi069 ceph-mon[29472]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "27e543e2-f5c0-462d-ab23-d439e06c056e"}]: dispatch 2024-04-14T13:53:21.568 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:53:21 smithi069 ceph-mon[29472]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "27e543e2-f5c0-462d-ab23-d439e06c056e"}]': finished 2024-04-14T13:53:21.568 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:53:21 smithi069 ceph-mon[29472]: osdmap e22: 4 total, 3 up, 4 in 2024-04-14T13:53:21.568 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:53:21 smithi069 ceph-mon[29472]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-14T13:53:21.568 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:53:21 smithi069 ceph-mon[24818]: pgmap v75: 1 pgs: 1 active+clean; 577 KiB data, 15 MiB used, 268 GiB / 268 GiB avail 2024-04-14T13:53:21.568 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:53:21 smithi069 ceph-mon[24818]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-14T13:53:21.568 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:53:21 smithi069 ceph-mon[24818]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' 2024-04-14T13:53:21.568 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:53:21 smithi069 ceph-mon[24818]: from='client.? 172.21.15.69:0/2464956847' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "27e543e2-f5c0-462d-ab23-d439e06c056e"}]: dispatch 2024-04-14T13:53:21.568 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:53:21 smithi069 ceph-mon[24818]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "27e543e2-f5c0-462d-ab23-d439e06c056e"}]: dispatch 2024-04-14T13:53:21.568 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:53:21 smithi069 ceph-mon[24818]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "27e543e2-f5c0-462d-ab23-d439e06c056e"}]': finished 2024-04-14T13:53:21.568 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:53:21 smithi069 ceph-mon[24818]: osdmap e22: 4 total, 3 up, 4 in 2024-04-14T13:53:21.568 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:53:21 smithi069 ceph-mon[24818]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-14T13:53:21.595 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:53:21 smithi122 ceph-mon[25721]: pgmap v75: 1 pgs: 1 active+clean; 577 KiB data, 15 MiB used, 268 GiB / 268 GiB avail 2024-04-14T13:53:21.595 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:53:21 smithi122 ceph-mon[25721]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-14T13:53:21.595 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:53:21 smithi122 ceph-mon[25721]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' 2024-04-14T13:53:21.595 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:53:21 smithi122 ceph-mon[25721]: from='client.? 172.21.15.69:0/2464956847' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "27e543e2-f5c0-462d-ab23-d439e06c056e"}]: dispatch 2024-04-14T13:53:21.595 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:53:21 smithi122 ceph-mon[25721]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "27e543e2-f5c0-462d-ab23-d439e06c056e"}]: dispatch 2024-04-14T13:53:21.595 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:53:21 smithi122 ceph-mon[25721]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "27e543e2-f5c0-462d-ab23-d439e06c056e"}]': finished 2024-04-14T13:53:21.595 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:53:21 smithi122 ceph-mon[25721]: osdmap e22: 4 total, 3 up, 4 in 2024-04-14T13:53:21.595 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:53:21 smithi122 ceph-mon[25721]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-14T13:53:22.567 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:53:22 smithi069 ceph-mon[24818]: from='client.? 172.21.15.69:0/4264286740' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-14T13:53:22.568 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:53:22 smithi069 ceph-mon[29472]: from='client.? 172.21.15.69:0/4264286740' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-14T13:53:22.595 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:53:22 smithi122 ceph-mon[25721]: from='client.? 172.21.15.69:0/4264286740' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-14T13:53:23.567 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:53:23 smithi069 ceph-mon[24818]: pgmap v77: 1 pgs: 1 active+clean; 577 KiB data, 17 MiB used, 268 GiB / 268 GiB avail 2024-04-14T13:53:23.568 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:53:23 smithi069 ceph-mon[29472]: pgmap v77: 1 pgs: 1 active+clean; 577 KiB data, 17 MiB used, 268 GiB / 268 GiB avail 2024-04-14T13:53:23.595 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:53:23 smithi122 ceph-mon[25721]: pgmap v77: 1 pgs: 1 active+clean; 577 KiB data, 17 MiB used, 268 GiB / 268 GiB avail 2024-04-14T13:53:25.552 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:53:25 smithi069 ceph-mon[24818]: pgmap v78: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail; 3.4 KiB/s rd, 148 KiB/s wr, 10 op/s 2024-04-14T13:53:25.553 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:53:25 smithi069 ceph-mon[29472]: pgmap v78: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail; 3.4 KiB/s rd, 148 KiB/s wr, 10 op/s 2024-04-14T13:53:25.595 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:53:25 smithi122 ceph-mon[25721]: pgmap v78: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail; 3.4 KiB/s rd, 148 KiB/s wr, 10 op/s 2024-04-14T13:53:27.595 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:53:27 smithi122 ceph-mon[25721]: pgmap v79: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail; 2.5 KiB/s rd, 107 KiB/s wr, 7 op/s 2024-04-14T13:53:27.595 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:53:27 smithi122 ceph-mon[25721]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-04-14T13:53:27.595 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:53:27 smithi122 ceph-mon[25721]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T13:53:27.621 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:53:27 smithi069 ceph-mon[24818]: pgmap v79: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail; 2.5 KiB/s rd, 107 KiB/s wr, 7 op/s 2024-04-14T13:53:27.621 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:53:27 smithi069 ceph-mon[24818]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-04-14T13:53:27.622 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:53:27 smithi069 ceph-mon[24818]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T13:53:27.622 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:53:27 smithi069 ceph-mon[29472]: pgmap v79: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail; 2.5 KiB/s rd, 107 KiB/s wr, 7 op/s 2024-04-14T13:53:27.622 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:53:27 smithi069 ceph-mon[29472]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-04-14T13:53:27.622 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:53:27 smithi069 ceph-mon[29472]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T13:53:28.531 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:53:28 smithi069 ceph-mon[24818]: Deploying daemon osd.3 on smithi069 2024-04-14T13:53:28.532 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:53:28 smithi069 ceph-mon[29472]: Deploying daemon osd.3 on smithi069 2024-04-14T13:53:28.595 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:53:28 smithi122 ceph-mon[25721]: Deploying daemon osd.3 on smithi069 2024-04-14T13:53:29.567 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:53:29 smithi069 ceph-mon[24818]: pgmap v80: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail; 1.9 KiB/s rd, 84 KiB/s wr, 6 op/s 2024-04-14T13:53:29.567 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:53:29 smithi069 ceph-mon[29472]: pgmap v80: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail; 1.9 KiB/s rd, 84 KiB/s wr, 6 op/s 2024-04-14T13:53:29.595 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:53:29 smithi122 ceph-mon[25721]: pgmap v80: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail; 1.9 KiB/s rd, 84 KiB/s wr, 6 op/s 2024-04-14T13:53:31.387 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:53:31 smithi069 ceph-mon[24818]: pgmap v81: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail; 1.8 KiB/s rd, 78 KiB/s wr, 5 op/s 2024-04-14T13:53:31.388 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:53:31 smithi069 ceph-mon[29472]: pgmap v81: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail; 1.8 KiB/s rd, 78 KiB/s wr, 5 op/s 2024-04-14T13:53:31.595 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:53:31 smithi122 ceph-mon[25721]: pgmap v81: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail; 1.8 KiB/s rd, 78 KiB/s wr, 5 op/s 2024-04-14T13:53:33.138 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:53:32 smithi069 ceph-mon[24818]: pgmap v82: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail; 1.7 KiB/s rd, 74 KiB/s wr, 5 op/s 2024-04-14T13:53:33.138 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:53:32 smithi069 ceph-mon[24818]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' 2024-04-14T13:53:33.138 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:53:32 smithi069 ceph-mon[24818]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-14T13:53:33.138 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:53:32 smithi069 ceph-mon[24818]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T13:53:33.138 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:53:32 smithi069 ceph-mon[24818]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-14T13:53:33.139 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:53:32 smithi069 ceph-mon[29472]: pgmap v82: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail; 1.7 KiB/s rd, 74 KiB/s wr, 5 op/s 2024-04-14T13:53:33.139 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:53:32 smithi069 ceph-mon[29472]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' 2024-04-14T13:53:33.139 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:53:32 smithi069 ceph-mon[29472]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-14T13:53:33.139 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:53:32 smithi069 ceph-mon[29472]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T13:53:33.139 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:53:32 smithi069 ceph-mon[29472]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-14T13:53:33.345 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:53:32 smithi122 ceph-mon[25721]: pgmap v82: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail; 1.7 KiB/s rd, 74 KiB/s wr, 5 op/s 2024-04-14T13:53:33.345 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:53:32 smithi122 ceph-mon[25721]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' 2024-04-14T13:53:33.345 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:53:32 smithi122 ceph-mon[25721]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-14T13:53:33.345 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:53:32 smithi122 ceph-mon[25721]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T13:53:33.345 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:53:32 smithi122 ceph-mon[25721]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-14T13:53:34.669 INFO:teuthology.orchestra.run.smithi069.stdout:Created osd(s) 3 on host 'smithi069' 2024-04-14T13:53:34.987 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:53:34 smithi069 ceph-mon[24818]: pgmap v83: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail; 1.5 KiB/s rd, 65 KiB/s wr, 4 op/s 2024-04-14T13:53:34.988 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:53:34 smithi069 ceph-mon[24818]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' 2024-04-14T13:53:34.988 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:53:34 smithi069 ceph-mon[29472]: pgmap v83: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail; 1.5 KiB/s rd, 65 KiB/s wr, 4 op/s 2024-04-14T13:53:34.988 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:53:34 smithi069 ceph-mon[29472]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' 2024-04-14T13:53:35.345 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:53:34 smithi122 ceph-mon[25721]: pgmap v83: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail; 1.5 KiB/s rd, 65 KiB/s wr, 4 op/s 2024-04-14T13:53:35.345 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:53:34 smithi122 ceph-mon[25721]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' 2024-04-14T13:53:35.593 DEBUG:teuthology.orchestra.run.smithi069:osd.3> sudo journalctl -f -n 0 -u ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968@osd.3.service 2024-04-14T13:53:35.596 INFO:tasks.cephadm:Deploying osd.4 on smithi122 with /dev/vg_nvme/lv_4... 2024-04-14T13:53:35.596 DEBUG:teuthology.orchestra.run.smithi122:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid dc326fc2-fa65-11ee-bc8e-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-04-14T13:53:36.109 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:53:35 smithi069 ceph-mon[24818]: from='osd.3 [v2:172.21.15.69:6826/148592620,v1:172.21.15.69:6827/148592620]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-04-14T13:53:36.109 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:53:35 smithi069 ceph-mon[24818]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' 2024-04-14T13:53:36.110 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:53:35 smithi069 ceph-mon[29472]: from='osd.3 [v2:172.21.15.69:6826/148592620,v1:172.21.15.69:6827/148592620]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-04-14T13:53:36.110 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:53:35 smithi069 ceph-mon[29472]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' 2024-04-14T13:53:36.191 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:53:35 smithi122 ceph-mon[25721]: from='osd.3 [v2:172.21.15.69:6826/148592620,v1:172.21.15.69:6827/148592620]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-04-14T13:53:36.191 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:53:35 smithi122 ceph-mon[25721]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' 2024-04-14T13:53:37.068 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:53:36 smithi069 ceph-mon[24818]: pgmap v84: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail 2024-04-14T13:53:37.068 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:53:36 smithi069 ceph-mon[24818]: from='osd.3 [v2:172.21.15.69:6826/148592620,v1:172.21.15.69:6827/148592620]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-04-14T13:53:37.068 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:53:36 smithi069 ceph-mon[24818]: osdmap e23: 4 total, 3 up, 4 in 2024-04-14T13:53:37.068 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:53:36 smithi069 ceph-mon[24818]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-14T13:53:37.068 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:53:36 smithi069 ceph-mon[24818]: from='osd.3 [v2:172.21.15.69:6826/148592620,v1:172.21.15.69:6827/148592620]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi069", "root=default"]}]: dispatch 2024-04-14T13:53:37.068 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:53:36 smithi069 ceph-mon[29472]: pgmap v84: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail 2024-04-14T13:53:37.068 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:53:36 smithi069 ceph-mon[29472]: from='osd.3 [v2:172.21.15.69:6826/148592620,v1:172.21.15.69:6827/148592620]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-04-14T13:53:37.068 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:53:36 smithi069 ceph-mon[29472]: osdmap e23: 4 total, 3 up, 4 in 2024-04-14T13:53:37.069 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:53:36 smithi069 ceph-mon[29472]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-14T13:53:37.069 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:53:36 smithi069 ceph-mon[29472]: from='osd.3 [v2:172.21.15.69:6826/148592620,v1:172.21.15.69:6827/148592620]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi069", "root=default"]}]: dispatch 2024-04-14T13:53:37.069 INFO:journalctl@ceph.osd.3.smithi069.stdout:Apr 14 13:53:36 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-osd-3[41193]: 2024-04-14T13:53:36.901+0000 7fc664af0700 -1 osd.3 0 waiting for initial osdmap 2024-04-14T13:53:37.069 INFO:journalctl@ceph.osd.3.smithi069.stdout:Apr 14 13:53:36 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-osd-3[41193]: 2024-04-14T13:53:36.908+0000 7fc66148b700 -1 osd.3 24 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-14T13:53:37.150 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:53:36 smithi122 ceph-mon[25721]: pgmap v84: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail 2024-04-14T13:53:37.150 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:53:36 smithi122 ceph-mon[25721]: from='osd.3 [v2:172.21.15.69:6826/148592620,v1:172.21.15.69:6827/148592620]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-04-14T13:53:37.150 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:53:36 smithi122 ceph-mon[25721]: osdmap e23: 4 total, 3 up, 4 in 2024-04-14T13:53:37.150 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:53:36 smithi122 ceph-mon[25721]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-14T13:53:37.150 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:53:36 smithi122 ceph-mon[25721]: from='osd.3 [v2:172.21.15.69:6826/148592620,v1:172.21.15.69:6827/148592620]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi069", "root=default"]}]: dispatch 2024-04-14T13:53:37.612 INFO:teuthology.orchestra.run.smithi122.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-04-14T13:53:37.612 INFO:teuthology.orchestra.run.smithi122.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-04-14T13:53:37.613 INFO:teuthology.orchestra.run.smithi122.stderr: stderr: 10+0 records in 2024-04-14T13:53:37.613 INFO:teuthology.orchestra.run.smithi122.stderr:10+0 records out 2024-04-14T13:53:37.613 INFO:teuthology.orchestra.run.smithi122.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0108464 s, 967 MB/s 2024-04-14T13:53:37.613 INFO:teuthology.orchestra.run.smithi122.stderr: stderr: 2024-04-14T13:53:37.613 INFO:teuthology.orchestra.run.smithi122.stderr:--> Zapping successful for: 2024-04-14T13:53:38.067 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:53:37 smithi069 ceph-mon[24818]: from='osd.3 [v2:172.21.15.69:6826/148592620,v1:172.21.15.69:6827/148592620]' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi069", "root=default"]}]': finished 2024-04-14T13:53:38.068 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:53:37 smithi069 ceph-mon[24818]: osdmap e24: 4 total, 3 up, 4 in 2024-04-14T13:53:38.068 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:53:37 smithi069 ceph-mon[24818]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-14T13:53:38.068 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:53:37 smithi069 ceph-mon[24818]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-14T13:53:38.068 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:53:37 smithi069 ceph-mon[29472]: from='osd.3 [v2:172.21.15.69:6826/148592620,v1:172.21.15.69:6827/148592620]' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi069", "root=default"]}]': finished 2024-04-14T13:53:38.068 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:53:37 smithi069 ceph-mon[29472]: osdmap e24: 4 total, 3 up, 4 in 2024-04-14T13:53:38.068 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:53:37 smithi069 ceph-mon[29472]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-14T13:53:38.068 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:53:37 smithi069 ceph-mon[29472]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-14T13:53:38.200 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:53:37 smithi122 ceph-mon[25721]: from='osd.3 [v2:172.21.15.69:6826/148592620,v1:172.21.15.69:6827/148592620]' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi069", "root=default"]}]': finished 2024-04-14T13:53:38.201 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:53:37 smithi122 ceph-mon[25721]: osdmap e24: 4 total, 3 up, 4 in 2024-04-14T13:53:38.201 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:53:37 smithi122 ceph-mon[25721]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-14T13:53:38.201 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:53:37 smithi122 ceph-mon[25721]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-14T13:53:38.223 DEBUG:teuthology.orchestra.run.smithi122:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid dc326fc2-fa65-11ee-bc8e-c7b262605968 -- ceph orch daemon add osd smithi122:vg_nvme/lv_4 2024-04-14T13:53:39.009 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:53:38 smithi122 ceph-mon[25721]: purged_snaps scrub starts 2024-04-14T13:53:39.009 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:53:38 smithi122 ceph-mon[25721]: purged_snaps scrub ok 2024-04-14T13:53:39.010 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:53:38 smithi122 ceph-mon[25721]: pgmap v87: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail 2024-04-14T13:53:39.010 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:53:38 smithi122 ceph-mon[25721]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-14T13:53:39.010 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:53:38 smithi122 ceph-mon[25721]: osd.3 [v2:172.21.15.69:6826/148592620,v1:172.21.15.69:6827/148592620] boot 2024-04-14T13:53:39.010 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:53:38 smithi122 ceph-mon[25721]: osdmap e25: 4 total, 4 up, 4 in 2024-04-14T13:53:39.010 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:53:38 smithi122 ceph-mon[25721]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-14T13:53:39.317 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:53:38 smithi069 ceph-mon[24818]: purged_snaps scrub starts 2024-04-14T13:53:39.317 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:53:38 smithi069 ceph-mon[24818]: purged_snaps scrub ok 2024-04-14T13:53:39.317 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:53:38 smithi069 ceph-mon[24818]: pgmap v87: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail 2024-04-14T13:53:39.317 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:53:38 smithi069 ceph-mon[24818]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-14T13:53:39.317 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:53:38 smithi069 ceph-mon[24818]: osd.3 [v2:172.21.15.69:6826/148592620,v1:172.21.15.69:6827/148592620] boot 2024-04-14T13:53:39.318 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:53:38 smithi069 ceph-mon[24818]: osdmap e25: 4 total, 4 up, 4 in 2024-04-14T13:53:39.318 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:53:38 smithi069 ceph-mon[24818]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-14T13:53:39.318 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:53:38 smithi069 ceph-mon[29472]: purged_snaps scrub starts 2024-04-14T13:53:39.318 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:53:38 smithi069 ceph-mon[29472]: purged_snaps scrub ok 2024-04-14T13:53:39.318 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:53:38 smithi069 ceph-mon[29472]: pgmap v87: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail 2024-04-14T13:53:39.318 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:53:38 smithi069 ceph-mon[29472]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-14T13:53:39.318 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:53:38 smithi069 ceph-mon[29472]: osd.3 [v2:172.21.15.69:6826/148592620,v1:172.21.15.69:6827/148592620] boot 2024-04-14T13:53:39.318 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:53:38 smithi069 ceph-mon[29472]: osdmap e25: 4 total, 4 up, 4 in 2024-04-14T13:53:39.318 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:53:38 smithi069 ceph-mon[29472]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-14T13:53:40.043 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:53:39 smithi122 ceph-mon[25721]: osdmap e26: 4 total, 4 up, 4 in 2024-04-14T13:53:40.043 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:53:39 smithi122 ceph-mon[25721]: Detected new or changed devices on smithi069 2024-04-14T13:53:40.043 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:53:39 smithi122 ceph-mon[25721]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' 2024-04-14T13:53:40.043 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:53:39 smithi122 ceph-mon[25721]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-14T13:53:40.043 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:53:39 smithi122 ceph-mon[25721]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-14T13:53:40.043 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:53:39 smithi122 ceph-mon[25721]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-14T13:53:40.043 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:53:39 smithi122 ceph-mon[25721]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-14T13:53:40.043 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:53:39 smithi122 ceph-mon[25721]: Adjusting osd_memory_target on smithi069 to 4016M 2024-04-14T13:53:40.043 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:53:39 smithi122 ceph-mon[25721]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' 2024-04-14T13:53:40.043 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:53:39 smithi122 ceph-mon[25721]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' 2024-04-14T13:53:40.043 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:53:39 smithi122 ceph-mon[25721]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-14T13:53:40.043 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:53:39 smithi122 ceph-mon[25721]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T13:53:40.043 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:53:39 smithi122 ceph-mon[25721]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-14T13:53:40.044 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:53:39 smithi122 ceph-mon[25721]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' 2024-04-14T13:53:40.219 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:53:39 smithi069 ceph-mon[24818]: osdmap e26: 4 total, 4 up, 4 in 2024-04-14T13:53:40.219 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:53:39 smithi069 ceph-mon[24818]: Detected new or changed devices on smithi069 2024-04-14T13:53:40.219 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:53:39 smithi069 ceph-mon[24818]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' 2024-04-14T13:53:40.219 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:53:39 smithi069 ceph-mon[24818]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-14T13:53:40.219 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:53:39 smithi069 ceph-mon[24818]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-14T13:53:40.219 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:53:39 smithi069 ceph-mon[24818]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-14T13:53:40.219 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:53:39 smithi069 ceph-mon[24818]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-14T13:53:40.219 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:53:39 smithi069 ceph-mon[24818]: Adjusting osd_memory_target on smithi069 to 4016M 2024-04-14T13:53:40.219 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:53:39 smithi069 ceph-mon[24818]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' 2024-04-14T13:53:40.220 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:53:39 smithi069 ceph-mon[24818]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' 2024-04-14T13:53:40.220 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:53:39 smithi069 ceph-mon[24818]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-14T13:53:40.220 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:53:39 smithi069 ceph-mon[24818]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T13:53:40.220 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:53:39 smithi069 ceph-mon[24818]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-14T13:53:40.220 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:53:39 smithi069 ceph-mon[24818]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' 2024-04-14T13:53:40.220 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:53:39 smithi069 ceph-mon[29472]: osdmap e26: 4 total, 4 up, 4 in 2024-04-14T13:53:40.220 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:53:39 smithi069 ceph-mon[29472]: Detected new or changed devices on smithi069 2024-04-14T13:53:40.220 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:53:39 smithi069 ceph-mon[29472]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' 2024-04-14T13:53:40.220 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:53:39 smithi069 ceph-mon[29472]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-14T13:53:40.220 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:53:39 smithi069 ceph-mon[29472]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-14T13:53:40.220 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:53:39 smithi069 ceph-mon[29472]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-14T13:53:40.220 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:53:39 smithi069 ceph-mon[29472]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-14T13:53:40.220 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:53:39 smithi069 ceph-mon[29472]: Adjusting osd_memory_target on smithi069 to 4016M 2024-04-14T13:53:40.221 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:53:39 smithi069 ceph-mon[29472]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' 2024-04-14T13:53:40.221 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:53:39 smithi069 ceph-mon[29472]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' 2024-04-14T13:53:40.221 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:53:39 smithi069 ceph-mon[29472]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-14T13:53:40.221 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:53:39 smithi069 ceph-mon[29472]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T13:53:40.221 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:53:39 smithi069 ceph-mon[29472]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-14T13:53:40.221 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:53:39 smithi069 ceph-mon[29472]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' 2024-04-14T13:53:41.230 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:53:40 smithi069 ceph-mon[24818]: pgmap v90: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-04-14T13:53:41.230 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:53:40 smithi069 ceph-mon[24818]: from='client.24188 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi122:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T13:53:41.230 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:53:40 smithi069 ceph-mon[24818]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-14T13:53:41.231 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:53:40 smithi069 ceph-mon[24818]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-14T13:53:41.231 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:53:40 smithi069 ceph-mon[24818]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T13:53:41.231 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:53:40 smithi069 ceph-mon[29472]: pgmap v90: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-04-14T13:53:41.231 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:53:40 smithi069 ceph-mon[29472]: from='client.24188 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi122:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T13:53:41.231 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:53:40 smithi069 ceph-mon[29472]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-14T13:53:41.231 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:53:40 smithi069 ceph-mon[29472]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-14T13:53:41.231 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:53:40 smithi069 ceph-mon[29472]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T13:53:41.291 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:53:40 smithi122 ceph-mon[25721]: pgmap v90: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-04-14T13:53:41.291 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:53:40 smithi122 ceph-mon[25721]: from='client.24188 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi122:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T13:53:41.291 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:53:40 smithi122 ceph-mon[25721]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-14T13:53:41.291 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:53:40 smithi122 ceph-mon[25721]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-14T13:53:41.291 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:53:40 smithi122 ceph-mon[25721]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T13:53:43.317 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:53:42 smithi069 ceph-mon[24818]: pgmap v91: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-04-14T13:53:43.317 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:53:42 smithi069 ceph-mon[24818]: from='client.? 172.21.15.122:0/160859774' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "cc9f056c-1e48-41c6-80db-dd1ce38c11db"}]: dispatch 2024-04-14T13:53:43.317 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:53:42 smithi069 ceph-mon[24818]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "cc9f056c-1e48-41c6-80db-dd1ce38c11db"}]: dispatch 2024-04-14T13:53:43.317 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:53:42 smithi069 ceph-mon[24818]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "cc9f056c-1e48-41c6-80db-dd1ce38c11db"}]': finished 2024-04-14T13:53:43.318 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:53:42 smithi069 ceph-mon[24818]: osdmap e27: 5 total, 4 up, 5 in 2024-04-14T13:53:43.318 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:53:42 smithi069 ceph-mon[24818]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-14T13:53:43.318 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:53:42 smithi069 ceph-mon[24818]: from='client.? 172.21.15.122:0/1360092850' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-14T13:53:43.318 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:53:42 smithi069 ceph-mon[29472]: pgmap v91: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-04-14T13:53:43.318 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:53:42 smithi069 ceph-mon[29472]: from='client.? 172.21.15.122:0/160859774' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "cc9f056c-1e48-41c6-80db-dd1ce38c11db"}]: dispatch 2024-04-14T13:53:43.318 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:53:42 smithi069 ceph-mon[29472]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "cc9f056c-1e48-41c6-80db-dd1ce38c11db"}]: dispatch 2024-04-14T13:53:43.318 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:53:42 smithi069 ceph-mon[29472]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "cc9f056c-1e48-41c6-80db-dd1ce38c11db"}]': finished 2024-04-14T13:53:43.318 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:53:42 smithi069 ceph-mon[29472]: osdmap e27: 5 total, 4 up, 5 in 2024-04-14T13:53:43.318 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:53:42 smithi069 ceph-mon[29472]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-14T13:53:43.318 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:53:42 smithi069 ceph-mon[29472]: from='client.? 172.21.15.122:0/1360092850' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-14T13:53:43.345 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:53:42 smithi122 ceph-mon[25721]: pgmap v91: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-04-14T13:53:43.345 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:53:42 smithi122 ceph-mon[25721]: from='client.? 172.21.15.122:0/160859774' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "cc9f056c-1e48-41c6-80db-dd1ce38c11db"}]: dispatch 2024-04-14T13:53:43.345 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:53:42 smithi122 ceph-mon[25721]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "cc9f056c-1e48-41c6-80db-dd1ce38c11db"}]: dispatch 2024-04-14T13:53:43.345 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:53:42 smithi122 ceph-mon[25721]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "cc9f056c-1e48-41c6-80db-dd1ce38c11db"}]': finished 2024-04-14T13:53:43.346 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:53:42 smithi122 ceph-mon[25721]: osdmap e27: 5 total, 4 up, 5 in 2024-04-14T13:53:43.346 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:53:42 smithi122 ceph-mon[25721]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-14T13:53:43.346 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:53:42 smithi122 ceph-mon[25721]: from='client.? 172.21.15.122:0/1360092850' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-14T13:53:45.317 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:53:44 smithi069 ceph-mon[24818]: pgmap v93: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-04-14T13:53:45.318 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:53:44 smithi069 ceph-mon[29472]: pgmap v93: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-04-14T13:53:45.323 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:53:44 smithi122 ceph-mon[25721]: pgmap v93: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-04-14T13:53:47.545 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:53:47 smithi122 ceph-mon[25721]: pgmap v94: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-04-14T13:53:47.567 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:53:47 smithi069 ceph-mon[24818]: pgmap v94: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-04-14T13:53:47.567 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:53:47 smithi069 ceph-mon[29472]: pgmap v94: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-04-14T13:53:48.345 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:53:48 smithi122 ceph-mon[25721]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-04-14T13:53:48.345 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:53:48 smithi122 ceph-mon[25721]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T13:53:48.567 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:53:48 smithi069 ceph-mon[24818]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-04-14T13:53:48.567 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:53:48 smithi069 ceph-mon[24818]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T13:53:48.567 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:53:48 smithi069 ceph-mon[29472]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-04-14T13:53:48.567 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:53:48 smithi069 ceph-mon[29472]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T13:53:49.433 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:53:49 smithi122 ceph-mon[25721]: pgmap v95: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-04-14T13:53:49.433 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:53:49 smithi122 ceph-mon[25721]: Deploying daemon osd.4 on smithi122 2024-04-14T13:53:49.567 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:53:49 smithi069 ceph-mon[24818]: pgmap v95: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-04-14T13:53:49.567 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:53:49 smithi069 ceph-mon[24818]: Deploying daemon osd.4 on smithi122 2024-04-14T13:53:49.567 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:53:49 smithi069 ceph-mon[29472]: pgmap v95: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-04-14T13:53:49.567 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:53:49 smithi069 ceph-mon[29472]: Deploying daemon osd.4 on smithi122 2024-04-14T13:53:51.345 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:53:51 smithi122 ceph-mon[25721]: pgmap v96: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-04-14T13:53:51.567 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:53:51 smithi069 ceph-mon[24818]: pgmap v96: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-04-14T13:53:51.567 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:53:51 smithi069 ceph-mon[29472]: pgmap v96: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-04-14T13:53:53.474 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:53:53 smithi122 ceph-mon[25721]: pgmap v97: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-04-14T13:53:53.474 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:53:53 smithi122 ceph-mon[25721]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' 2024-04-14T13:53:53.474 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:53:53 smithi122 ceph-mon[25721]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-14T13:53:53.474 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:53:53 smithi122 ceph-mon[25721]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T13:53:53.474 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:53:53 smithi122 ceph-mon[25721]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-14T13:53:53.567 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:53:53 smithi069 ceph-mon[24818]: pgmap v97: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-04-14T13:53:53.568 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:53:53 smithi069 ceph-mon[24818]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' 2024-04-14T13:53:53.568 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:53:53 smithi069 ceph-mon[24818]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-14T13:53:53.568 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:53:53 smithi069 ceph-mon[24818]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T13:53:53.568 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:53:53 smithi069 ceph-mon[24818]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-14T13:53:53.568 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:53:53 smithi069 ceph-mon[29472]: pgmap v97: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-04-14T13:53:53.568 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:53:53 smithi069 ceph-mon[29472]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' 2024-04-14T13:53:53.568 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:53:53 smithi069 ceph-mon[29472]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-14T13:53:53.568 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:53:53 smithi069 ceph-mon[29472]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T13:53:53.568 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:53:53 smithi069 ceph-mon[29472]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-14T13:53:55.508 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:53:55 smithi122 ceph-mon[25721]: pgmap v98: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-04-14T13:53:55.567 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:53:55 smithi069 ceph-mon[24818]: pgmap v98: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-04-14T13:53:55.567 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:53:55 smithi069 ceph-mon[29472]: pgmap v98: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-04-14T13:53:56.089 INFO:teuthology.orchestra.run.smithi122.stdout:Created osd(s) 4 on host 'smithi122' 2024-04-14T13:53:56.346 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:53:56 smithi122 ceph-mon[25721]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' 2024-04-14T13:53:56.346 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:53:56 smithi122 ceph-mon[25721]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' 2024-04-14T13:53:56.346 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:53:56 smithi122 ceph-mon[25721]: from='osd.4 [v2:172.21.15.122:6800/3006695914,v1:172.21.15.122:6801/3006695914]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-04-14T13:53:56.346 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:53:56 smithi122 ceph-mon[25721]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-04-14T13:53:56.346 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:53:56 smithi122 ceph-mon[25721]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' 2024-04-14T13:53:56.346 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:53:56 smithi122 ceph-mon[25721]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-14T13:53:56.346 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:53:56 smithi122 ceph-mon[25721]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T13:53:56.347 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:53:56 smithi122 ceph-mon[25721]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-14T13:53:56.567 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:53:56 smithi069 ceph-mon[24818]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' 2024-04-14T13:53:56.567 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:53:56 smithi069 ceph-mon[24818]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' 2024-04-14T13:53:56.568 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:53:56 smithi069 ceph-mon[24818]: from='osd.4 [v2:172.21.15.122:6800/3006695914,v1:172.21.15.122:6801/3006695914]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-04-14T13:53:56.568 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:53:56 smithi069 ceph-mon[24818]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-04-14T13:53:56.568 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:53:56 smithi069 ceph-mon[24818]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' 2024-04-14T13:53:56.568 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:53:56 smithi069 ceph-mon[24818]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-14T13:53:56.568 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:53:56 smithi069 ceph-mon[24818]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T13:53:56.568 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:53:56 smithi069 ceph-mon[24818]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-14T13:53:56.568 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:53:56 smithi069 ceph-mon[29472]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' 2024-04-14T13:53:56.568 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:53:56 smithi069 ceph-mon[29472]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' 2024-04-14T13:53:56.568 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:53:56 smithi069 ceph-mon[29472]: from='osd.4 [v2:172.21.15.122:6800/3006695914,v1:172.21.15.122:6801/3006695914]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-04-14T13:53:56.568 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:53:56 smithi069 ceph-mon[29472]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-04-14T13:53:56.568 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:53:56 smithi069 ceph-mon[29472]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' 2024-04-14T13:53:56.568 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:53:56 smithi069 ceph-mon[29472]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-14T13:53:56.568 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:53:56 smithi069 ceph-mon[29472]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T13:53:56.569 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:53:56 smithi069 ceph-mon[29472]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-14T13:53:56.685 DEBUG:teuthology.orchestra.run.smithi122:osd.4> sudo journalctl -f -n 0 -u ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968@osd.4.service 2024-04-14T13:53:56.687 INFO:tasks.cephadm:Deploying osd.5 on smithi122 with /dev/vg_nvme/lv_3... 2024-04-14T13:53:56.687 DEBUG:teuthology.orchestra.run.smithi122:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid dc326fc2-fa65-11ee-bc8e-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-04-14T13:53:57.567 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:53:57 smithi069 ceph-mon[24818]: pgmap v99: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-04-14T13:53:57.567 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:53:57 smithi069 ceph-mon[24818]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-04-14T13:53:57.567 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:53:57 smithi069 ceph-mon[24818]: osdmap e28: 5 total, 4 up, 5 in 2024-04-14T13:53:57.568 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:53:57 smithi069 ceph-mon[24818]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-14T13:53:57.568 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:53:57 smithi069 ceph-mon[24818]: from='osd.4 [v2:172.21.15.122:6800/3006695914,v1:172.21.15.122:6801/3006695914]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi122", "root=default"]}]: dispatch 2024-04-14T13:53:57.568 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:53:57 smithi069 ceph-mon[24818]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi122", "root=default"]}]: dispatch 2024-04-14T13:53:57.568 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:53:57 smithi069 ceph-mon[29472]: pgmap v99: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-04-14T13:53:57.568 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:53:57 smithi069 ceph-mon[29472]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-04-14T13:53:57.568 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:53:57 smithi069 ceph-mon[29472]: osdmap e28: 5 total, 4 up, 5 in 2024-04-14T13:53:57.568 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:53:57 smithi069 ceph-mon[29472]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-14T13:53:57.568 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:53:57 smithi069 ceph-mon[29472]: from='osd.4 [v2:172.21.15.122:6800/3006695914,v1:172.21.15.122:6801/3006695914]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi122", "root=default"]}]: dispatch 2024-04-14T13:53:57.568 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:53:57 smithi069 ceph-mon[29472]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi122", "root=default"]}]: dispatch 2024-04-14T13:53:57.595 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:53:57 smithi122 ceph-mon[25721]: pgmap v99: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-04-14T13:53:57.595 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:53:57 smithi122 ceph-mon[25721]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-04-14T13:53:57.595 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:53:57 smithi122 ceph-mon[25721]: osdmap e28: 5 total, 4 up, 5 in 2024-04-14T13:53:57.595 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:53:57 smithi122 ceph-mon[25721]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-14T13:53:57.595 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:53:57 smithi122 ceph-mon[25721]: from='osd.4 [v2:172.21.15.122:6800/3006695914,v1:172.21.15.122:6801/3006695914]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi122", "root=default"]}]: dispatch 2024-04-14T13:53:57.595 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:53:57 smithi122 ceph-mon[25721]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi122", "root=default"]}]: dispatch 2024-04-14T13:53:57.596 INFO:journalctl@ceph.osd.4.smithi122.stdout:Apr 14 13:53:57 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-osd-4[29170]: 2024-04-14T13:53:57.271+0000 7f1bb07ec700 -1 osd.4 0 waiting for initial osdmap 2024-04-14T13:53:57.596 INFO:journalctl@ceph.osd.4.smithi122.stdout:Apr 14 13:53:57 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-osd-4[29170]: 2024-04-14T13:53:57.275+0000 7f1ba9980700 -1 osd.4 29 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-14T13:53:58.441 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:53:58 smithi122 ceph-mon[25721]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi122", "root=default"]}]': finished 2024-04-14T13:53:58.441 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:53:58 smithi122 ceph-mon[25721]: osdmap e29: 5 total, 4 up, 5 in 2024-04-14T13:53:58.441 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:53:58 smithi122 ceph-mon[25721]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-14T13:53:58.442 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:53:58 smithi122 ceph-mon[25721]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-14T13:53:58.567 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:53:58 smithi069 ceph-mon[24818]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi122", "root=default"]}]': finished 2024-04-14T13:53:58.567 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:53:58 smithi069 ceph-mon[24818]: osdmap e29: 5 total, 4 up, 5 in 2024-04-14T13:53:58.567 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:53:58 smithi069 ceph-mon[24818]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-14T13:53:58.567 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:53:58 smithi069 ceph-mon[24818]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-14T13:53:58.568 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:53:58 smithi069 ceph-mon[29472]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi122", "root=default"]}]': finished 2024-04-14T13:53:58.568 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:53:58 smithi069 ceph-mon[29472]: osdmap e29: 5 total, 4 up, 5 in 2024-04-14T13:53:58.568 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:53:58 smithi069 ceph-mon[29472]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-14T13:53:58.568 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:53:58 smithi069 ceph-mon[29472]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-14T13:53:59.567 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:53:59 smithi069 ceph-mon[24818]: purged_snaps scrub starts 2024-04-14T13:53:59.567 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:53:59 smithi069 ceph-mon[24818]: purged_snaps scrub ok 2024-04-14T13:53:59.567 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:53:59 smithi069 ceph-mon[24818]: pgmap v102: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-04-14T13:53:59.567 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:53:59 smithi069 ceph-mon[24818]: osd.4 [v2:172.21.15.122:6800/3006695914,v1:172.21.15.122:6801/3006695914] boot 2024-04-14T13:53:59.568 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:53:59 smithi069 ceph-mon[24818]: osdmap e30: 5 total, 5 up, 5 in 2024-04-14T13:53:59.568 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:53:59 smithi069 ceph-mon[24818]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-14T13:53:59.568 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:53:59 smithi069 ceph-mon[24818]: osdmap e31: 5 total, 5 up, 5 in 2024-04-14T13:53:59.568 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:53:59 smithi069 ceph-mon[29472]: purged_snaps scrub starts 2024-04-14T13:53:59.568 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:53:59 smithi069 ceph-mon[29472]: purged_snaps scrub ok 2024-04-14T13:53:59.568 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:53:59 smithi069 ceph-mon[29472]: pgmap v102: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-04-14T13:53:59.568 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:53:59 smithi069 ceph-mon[29472]: osd.4 [v2:172.21.15.122:6800/3006695914,v1:172.21.15.122:6801/3006695914] boot 2024-04-14T13:53:59.568 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:53:59 smithi069 ceph-mon[29472]: osdmap e30: 5 total, 5 up, 5 in 2024-04-14T13:53:59.568 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:53:59 smithi069 ceph-mon[29472]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-14T13:53:59.568 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:53:59 smithi069 ceph-mon[29472]: osdmap e31: 5 total, 5 up, 5 in 2024-04-14T13:53:59.595 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:53:59 smithi122 ceph-mon[25721]: purged_snaps scrub starts 2024-04-14T13:53:59.595 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:53:59 smithi122 ceph-mon[25721]: purged_snaps scrub ok 2024-04-14T13:53:59.595 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:53:59 smithi122 ceph-mon[25721]: pgmap v102: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-04-14T13:53:59.595 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:53:59 smithi122 ceph-mon[25721]: osd.4 [v2:172.21.15.122:6800/3006695914,v1:172.21.15.122:6801/3006695914] boot 2024-04-14T13:53:59.595 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:53:59 smithi122 ceph-mon[25721]: osdmap e30: 5 total, 5 up, 5 in 2024-04-14T13:53:59.595 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:53:59 smithi122 ceph-mon[25721]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-14T13:53:59.595 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:53:59 smithi122 ceph-mon[25721]: osdmap e31: 5 total, 5 up, 5 in 2024-04-14T13:54:00.295 INFO:teuthology.orchestra.run.smithi122.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-04-14T13:54:00.295 INFO:teuthology.orchestra.run.smithi122.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-04-14T13:54:00.296 INFO:teuthology.orchestra.run.smithi122.stderr: stderr: 10+0 records in 2024-04-14T13:54:00.296 INFO:teuthology.orchestra.run.smithi122.stderr:10+0 records out 2024-04-14T13:54:00.296 INFO:teuthology.orchestra.run.smithi122.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0151074 s, 694 MB/s 2024-04-14T13:54:00.296 INFO:teuthology.orchestra.run.smithi122.stderr:--> Zapping successful for: 2024-04-14T13:54:00.845 DEBUG:teuthology.orchestra.run.smithi122:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid dc326fc2-fa65-11ee-bc8e-c7b262605968 -- ceph orch daemon add osd smithi122:vg_nvme/lv_3 2024-04-14T13:54:01.467 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:54:01 smithi122 ceph-mon[25721]: pgmap v105: 1 pgs: 1 peering; 577 KiB data, 28 MiB used, 447 GiB / 447 GiB avail 2024-04-14T13:54:01.467 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:54:01 smithi122 ceph-mon[25721]: osdmap e32: 5 total, 5 up, 5 in 2024-04-14T13:54:01.468 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:54:01 smithi122 ceph-mon[25721]: Detected new or changed devices on smithi122 2024-04-14T13:54:01.468 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:54:01 smithi122 ceph-mon[25721]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' 2024-04-14T13:54:01.468 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:54:01 smithi122 ceph-mon[25721]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-14T13:54:01.468 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:54:01 smithi122 ceph-mon[25721]: Adjusting osd_memory_target on smithi122 to 17088M 2024-04-14T13:54:01.468 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:54:01 smithi122 ceph-mon[25721]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' 2024-04-14T13:54:01.468 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:54:01 smithi122 ceph-mon[25721]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' 2024-04-14T13:54:01.567 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:54:01 smithi069 ceph-mon[24818]: pgmap v105: 1 pgs: 1 peering; 577 KiB data, 28 MiB used, 447 GiB / 447 GiB avail 2024-04-14T13:54:01.567 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:54:01 smithi069 ceph-mon[24818]: osdmap e32: 5 total, 5 up, 5 in 2024-04-14T13:54:01.567 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:54:01 smithi069 ceph-mon[24818]: Detected new or changed devices on smithi122 2024-04-14T13:54:01.567 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:54:01 smithi069 ceph-mon[24818]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' 2024-04-14T13:54:01.567 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:54:01 smithi069 ceph-mon[24818]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-14T13:54:01.568 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:54:01 smithi069 ceph-mon[24818]: Adjusting osd_memory_target on smithi122 to 17088M 2024-04-14T13:54:01.568 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:54:01 smithi069 ceph-mon[24818]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' 2024-04-14T13:54:01.568 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:54:01 smithi069 ceph-mon[24818]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' 2024-04-14T13:54:01.568 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:54:01 smithi069 ceph-mon[29472]: pgmap v105: 1 pgs: 1 peering; 577 KiB data, 28 MiB used, 447 GiB / 447 GiB avail 2024-04-14T13:54:01.568 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:54:01 smithi069 ceph-mon[29472]: osdmap e32: 5 total, 5 up, 5 in 2024-04-14T13:54:01.568 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:54:01 smithi069 ceph-mon[29472]: Detected new or changed devices on smithi122 2024-04-14T13:54:01.568 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:54:01 smithi069 ceph-mon[29472]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' 2024-04-14T13:54:01.568 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:54:01 smithi069 ceph-mon[29472]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-14T13:54:01.568 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:54:01 smithi069 ceph-mon[29472]: Adjusting osd_memory_target on smithi122 to 17088M 2024-04-14T13:54:01.568 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:54:01 smithi069 ceph-mon[29472]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' 2024-04-14T13:54:01.568 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:54:01 smithi069 ceph-mon[29472]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' 2024-04-14T13:54:02.491 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:54:02 smithi122 ceph-mon[25721]: osdmap e33: 5 total, 5 up, 5 in 2024-04-14T13:54:02.567 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:54:02 smithi069 ceph-mon[24818]: osdmap e33: 5 total, 5 up, 5 in 2024-04-14T13:54:02.567 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:54:02 smithi069 ceph-mon[29472]: osdmap e33: 5 total, 5 up, 5 in 2024-04-14T13:54:03.345 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:54:03 smithi122 ceph-mon[25721]: pgmap v108: 1 pgs: 1 peering; 577 KiB data, 28 MiB used, 447 GiB / 447 GiB avail 2024-04-14T13:54:03.346 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:54:03 smithi122 ceph-mon[25721]: osdmap e34: 5 total, 5 up, 5 in 2024-04-14T13:54:03.346 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:54:03 smithi122 ceph-mon[25721]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-14T13:54:03.346 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:54:03 smithi122 ceph-mon[25721]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-14T13:54:03.346 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:54:03 smithi122 ceph-mon[25721]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T13:54:03.567 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:54:03 smithi069 ceph-mon[24818]: pgmap v108: 1 pgs: 1 peering; 577 KiB data, 28 MiB used, 447 GiB / 447 GiB avail 2024-04-14T13:54:03.567 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:54:03 smithi069 ceph-mon[24818]: osdmap e34: 5 total, 5 up, 5 in 2024-04-14T13:54:03.567 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:54:03 smithi069 ceph-mon[24818]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-14T13:54:03.568 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:54:03 smithi069 ceph-mon[24818]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-14T13:54:03.568 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:54:03 smithi069 ceph-mon[24818]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T13:54:03.568 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:54:03 smithi069 ceph-mon[29472]: pgmap v108: 1 pgs: 1 peering; 577 KiB data, 28 MiB used, 447 GiB / 447 GiB avail 2024-04-14T13:54:03.568 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:54:03 smithi069 ceph-mon[29472]: osdmap e34: 5 total, 5 up, 5 in 2024-04-14T13:54:03.568 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:54:03 smithi069 ceph-mon[29472]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-14T13:54:03.568 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:54:03 smithi069 ceph-mon[29472]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-14T13:54:03.568 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:54:03 smithi069 ceph-mon[29472]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T13:54:04.326 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:54:04 smithi122 ceph-mon[25721]: from='client.24215 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi122:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T13:54:04.567 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:54:04 smithi069 ceph-mon[24818]: from='client.24215 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi122:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T13:54:04.567 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:54:04 smithi069 ceph-mon[29472]: from='client.24215 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi122:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T13:54:05.567 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:54:05 smithi069 ceph-mon[24818]: pgmap v110: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail; 123 KiB/s, 0 objects/s recovering 2024-04-14T13:54:05.567 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:54:05 smithi069 ceph-mon[24818]: from='client.? 172.21.15.122:0/1813687812' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c127fffe-79af-4bf0-b67e-a204f716ab14"}]: dispatch 2024-04-14T13:54:05.567 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:54:05 smithi069 ceph-mon[24818]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c127fffe-79af-4bf0-b67e-a204f716ab14"}]: dispatch 2024-04-14T13:54:05.568 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:54:05 smithi069 ceph-mon[24818]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "c127fffe-79af-4bf0-b67e-a204f716ab14"}]': finished 2024-04-14T13:54:05.568 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:54:05 smithi069 ceph-mon[24818]: osdmap e35: 6 total, 5 up, 6 in 2024-04-14T13:54:05.568 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:54:05 smithi069 ceph-mon[24818]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-14T13:54:05.568 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:54:05 smithi069 ceph-mon[29472]: pgmap v110: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail; 123 KiB/s, 0 objects/s recovering 2024-04-14T13:54:05.568 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:54:05 smithi069 ceph-mon[29472]: from='client.? 172.21.15.122:0/1813687812' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c127fffe-79af-4bf0-b67e-a204f716ab14"}]: dispatch 2024-04-14T13:54:05.568 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:54:05 smithi069 ceph-mon[29472]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c127fffe-79af-4bf0-b67e-a204f716ab14"}]: dispatch 2024-04-14T13:54:05.568 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:54:05 smithi069 ceph-mon[29472]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "c127fffe-79af-4bf0-b67e-a204f716ab14"}]': finished 2024-04-14T13:54:05.568 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:54:05 smithi069 ceph-mon[29472]: osdmap e35: 6 total, 5 up, 6 in 2024-04-14T13:54:05.568 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:54:05 smithi069 ceph-mon[29472]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-14T13:54:05.595 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:54:05 smithi122 ceph-mon[25721]: pgmap v110: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail; 123 KiB/s, 0 objects/s recovering 2024-04-14T13:54:05.595 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:54:05 smithi122 ceph-mon[25721]: from='client.? 172.21.15.122:0/1813687812' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c127fffe-79af-4bf0-b67e-a204f716ab14"}]: dispatch 2024-04-14T13:54:05.595 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:54:05 smithi122 ceph-mon[25721]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c127fffe-79af-4bf0-b67e-a204f716ab14"}]: dispatch 2024-04-14T13:54:05.595 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:54:05 smithi122 ceph-mon[25721]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "c127fffe-79af-4bf0-b67e-a204f716ab14"}]': finished 2024-04-14T13:54:05.595 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:54:05 smithi122 ceph-mon[25721]: osdmap e35: 6 total, 5 up, 6 in 2024-04-14T13:54:05.595 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:54:05 smithi122 ceph-mon[25721]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-14T13:54:06.567 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:54:06 smithi069 ceph-mon[24818]: from='client.? 172.21.15.122:0/125091564' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-14T13:54:06.567 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:54:06 smithi069 ceph-mon[29472]: from='client.? 172.21.15.122:0/125091564' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-14T13:54:06.595 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:54:06 smithi122 ceph-mon[25721]: from='client.? 172.21.15.122:0/125091564' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-14T13:54:07.567 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:54:07 smithi069 ceph-mon[24818]: pgmap v112: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail; 99 KiB/s, 0 objects/s recovering 2024-04-14T13:54:07.568 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:54:07 smithi069 ceph-mon[29472]: pgmap v112: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail; 99 KiB/s, 0 objects/s recovering 2024-04-14T13:54:07.595 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:54:07 smithi122 ceph-mon[25721]: pgmap v112: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail; 99 KiB/s, 0 objects/s recovering 2024-04-14T13:54:09.537 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:54:09 smithi122 ceph-mon[25721]: pgmap v113: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail; 83 KiB/s, 0 objects/s recovering 2024-04-14T13:54:09.567 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:54:09 smithi069 ceph-mon[24818]: pgmap v113: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail; 83 KiB/s, 0 objects/s recovering 2024-04-14T13:54:09.567 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:54:09 smithi069 ceph-mon[29472]: pgmap v113: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail; 83 KiB/s, 0 objects/s recovering 2024-04-14T13:54:11.430 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:54:11 smithi122 ceph-mon[25721]: pgmap v114: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail; 65 KiB/s, 0 objects/s recovering 2024-04-14T13:54:11.430 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:54:11 smithi122 ceph-mon[25721]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-04-14T13:54:11.430 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:54:11 smithi122 ceph-mon[25721]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T13:54:11.817 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:54:11 smithi069 ceph-mon[24818]: pgmap v114: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail; 65 KiB/s, 0 objects/s recovering 2024-04-14T13:54:11.817 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:54:11 smithi069 ceph-mon[24818]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-04-14T13:54:11.818 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:54:11 smithi069 ceph-mon[24818]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T13:54:11.818 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:54:11 smithi069 ceph-mon[29472]: pgmap v114: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail; 65 KiB/s, 0 objects/s recovering 2024-04-14T13:54:11.818 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:54:11 smithi069 ceph-mon[29472]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-04-14T13:54:11.818 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:54:11 smithi069 ceph-mon[29472]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T13:54:12.595 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:54:12 smithi122 ceph-mon[25721]: Deploying daemon osd.5 on smithi122 2024-04-14T13:54:12.595 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:54:12 smithi122 ceph-mon[25721]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-04-14T13:54:12.595 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:54:12 smithi122 ceph-mon[25721]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-04-14T13:54:12.817 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:54:12 smithi069 ceph-mon[24818]: Deploying daemon osd.5 on smithi122 2024-04-14T13:54:12.817 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:54:12 smithi069 ceph-mon[24818]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-04-14T13:54:12.818 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:54:12 smithi069 ceph-mon[24818]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-04-14T13:54:12.818 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:54:12 smithi069 ceph-mon[29472]: Deploying daemon osd.5 on smithi122 2024-04-14T13:54:12.818 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:54:12 smithi069 ceph-mon[29472]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-04-14T13:54:12.818 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:54:12 smithi069 ceph-mon[29472]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-04-14T13:54:13.595 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:54:13 smithi122 ceph-mon[25721]: pgmap v115: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail; 56 KiB/s, 0 objects/s recovering 2024-04-14T13:54:13.817 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:54:13 smithi069 ceph-mon[24818]: pgmap v115: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail; 56 KiB/s, 0 objects/s recovering 2024-04-14T13:54:13.817 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:54:13 smithi069 ceph-mon[29472]: pgmap v115: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail; 56 KiB/s, 0 objects/s recovering 2024-04-14T13:54:15.565 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:54:15 smithi122 ceph-mon[25721]: pgmap v116: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2024-04-14T13:54:15.817 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:54:15 smithi069 ceph-mon[24818]: pgmap v116: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2024-04-14T13:54:15.817 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:54:15 smithi069 ceph-mon[29472]: pgmap v116: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2024-04-14T13:54:17.345 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:54:17 smithi122 ceph-mon[25721]: pgmap v117: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2024-04-14T13:54:17.345 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:54:17 smithi122 ceph-mon[25721]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' 2024-04-14T13:54:17.345 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:54:17 smithi122 ceph-mon[25721]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-14T13:54:17.345 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:54:17 smithi122 ceph-mon[25721]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T13:54:17.345 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:54:17 smithi122 ceph-mon[25721]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-14T13:54:17.567 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:54:17 smithi069 ceph-mon[24818]: pgmap v117: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2024-04-14T13:54:17.567 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:54:17 smithi069 ceph-mon[24818]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' 2024-04-14T13:54:17.568 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:54:17 smithi069 ceph-mon[24818]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-14T13:54:17.568 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:54:17 smithi069 ceph-mon[24818]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T13:54:17.568 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:54:17 smithi069 ceph-mon[24818]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-14T13:54:17.568 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:54:17 smithi069 ceph-mon[29472]: pgmap v117: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2024-04-14T13:54:17.568 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:54:17 smithi069 ceph-mon[29472]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' 2024-04-14T13:54:17.568 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:54:17 smithi069 ceph-mon[29472]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-14T13:54:17.568 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:54:17 smithi069 ceph-mon[29472]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T13:54:17.568 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:54:17 smithi069 ceph-mon[29472]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-14T13:54:19.082 INFO:teuthology.orchestra.run.smithi122.stdout:Created osd(s) 5 on host 'smithi122' 2024-04-14T13:54:19.345 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:54:19 smithi122 ceph-mon[25721]: pgmap v118: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2024-04-14T13:54:19.345 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:54:19 smithi122 ceph-mon[25721]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' 2024-04-14T13:54:19.567 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:54:19 smithi069 ceph-mon[24818]: pgmap v118: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2024-04-14T13:54:19.567 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:54:19 smithi069 ceph-mon[24818]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' 2024-04-14T13:54:19.567 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:54:19 smithi069 ceph-mon[29472]: pgmap v118: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2024-04-14T13:54:19.568 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:54:19 smithi069 ceph-mon[29472]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' 2024-04-14T13:54:19.730 DEBUG:teuthology.orchestra.run.smithi122:osd.5> sudo journalctl -f -n 0 -u ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968@osd.5.service 2024-04-14T13:54:19.732 INFO:tasks.cephadm:Deploying osd.6 on smithi122 with /dev/vg_nvme/lv_2... 2024-04-14T13:54:19.732 DEBUG:teuthology.orchestra.run.smithi122:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid dc326fc2-fa65-11ee-bc8e-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-04-14T13:54:20.345 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:54:20 smithi122 ceph-mon[25721]: from='osd.5 [v2:172.21.15.122:6808/4131566432,v1:172.21.15.122:6809/4131566432]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-04-14T13:54:20.345 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:54:20 smithi122 ceph-mon[25721]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-04-14T13:54:20.345 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:54:20 smithi122 ceph-mon[25721]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' 2024-04-14T13:54:20.567 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:54:20 smithi069 ceph-mon[24818]: from='osd.5 [v2:172.21.15.122:6808/4131566432,v1:172.21.15.122:6809/4131566432]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-04-14T13:54:20.567 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:54:20 smithi069 ceph-mon[24818]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-04-14T13:54:20.567 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:54:20 smithi069 ceph-mon[24818]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' 2024-04-14T13:54:20.568 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:54:20 smithi069 ceph-mon[29472]: from='osd.5 [v2:172.21.15.122:6808/4131566432,v1:172.21.15.122:6809/4131566432]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-04-14T13:54:20.568 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:54:20 smithi069 ceph-mon[29472]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-04-14T13:54:20.568 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:54:20 smithi069 ceph-mon[29472]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' 2024-04-14T13:54:21.345 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:54:21 smithi122 ceph-mon[25721]: pgmap v119: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2024-04-14T13:54:21.345 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:54:21 smithi122 ceph-mon[25721]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-04-14T13:54:21.345 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:54:21 smithi122 ceph-mon[25721]: osdmap e36: 6 total, 5 up, 6 in 2024-04-14T13:54:21.346 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:54:21 smithi122 ceph-mon[25721]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-14T13:54:21.346 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:54:21 smithi122 ceph-mon[25721]: from='osd.5 [v2:172.21.15.122:6808/4131566432,v1:172.21.15.122:6809/4131566432]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi122", "root=default"]}]: dispatch 2024-04-14T13:54:21.346 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:54:21 smithi122 ceph-mon[25721]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi122", "root=default"]}]: dispatch 2024-04-14T13:54:21.346 INFO:journalctl@ceph.osd.5.smithi122.stdout:Apr 14 13:54:21 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-osd-5[32056]: 2024-04-14T13:54:21.138+0000 7fa75875f700 -1 osd.5 0 waiting for initial osdmap 2024-04-14T13:54:21.346 INFO:journalctl@ceph.osd.5.smithi122.stdout:Apr 14 13:54:21 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-osd-5[32056]: 2024-04-14T13:54:21.144+0000 7fa7530f6700 -1 osd.5 37 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-14T13:54:21.567 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:54:21 smithi069 ceph-mon[24818]: pgmap v119: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2024-04-14T13:54:21.567 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:54:21 smithi069 ceph-mon[24818]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-04-14T13:54:21.567 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:54:21 smithi069 ceph-mon[24818]: osdmap e36: 6 total, 5 up, 6 in 2024-04-14T13:54:21.567 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:54:21 smithi069 ceph-mon[24818]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-14T13:54:21.567 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:54:21 smithi069 ceph-mon[24818]: from='osd.5 [v2:172.21.15.122:6808/4131566432,v1:172.21.15.122:6809/4131566432]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi122", "root=default"]}]: dispatch 2024-04-14T13:54:21.568 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:54:21 smithi069 ceph-mon[24818]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi122", "root=default"]}]: dispatch 2024-04-14T13:54:21.568 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:54:21 smithi069 ceph-mon[29472]: pgmap v119: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2024-04-14T13:54:21.568 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:54:21 smithi069 ceph-mon[29472]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-04-14T13:54:21.568 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:54:21 smithi069 ceph-mon[29472]: osdmap e36: 6 total, 5 up, 6 in 2024-04-14T13:54:21.568 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:54:21 smithi069 ceph-mon[29472]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-14T13:54:21.568 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:54:21 smithi069 ceph-mon[29472]: from='osd.5 [v2:172.21.15.122:6808/4131566432,v1:172.21.15.122:6809/4131566432]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi122", "root=default"]}]: dispatch 2024-04-14T13:54:21.568 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:54:21 smithi069 ceph-mon[29472]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi122", "root=default"]}]: dispatch 2024-04-14T13:54:22.366 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:54:22 smithi122 ceph-mon[25721]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi122", "root=default"]}]': finished 2024-04-14T13:54:22.366 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:54:22 smithi122 ceph-mon[25721]: osdmap e37: 6 total, 5 up, 6 in 2024-04-14T13:54:22.366 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:54:22 smithi122 ceph-mon[25721]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-14T13:54:22.366 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:54:22 smithi122 ceph-mon[25721]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-14T13:54:22.567 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:54:22 smithi069 ceph-mon[24818]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi122", "root=default"]}]': finished 2024-04-14T13:54:22.567 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:54:22 smithi069 ceph-mon[24818]: osdmap e37: 6 total, 5 up, 6 in 2024-04-14T13:54:22.567 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:54:22 smithi069 ceph-mon[24818]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-14T13:54:22.568 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:54:22 smithi069 ceph-mon[24818]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-14T13:54:22.568 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:54:22 smithi069 ceph-mon[29472]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi122", "root=default"]}]': finished 2024-04-14T13:54:22.568 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:54:22 smithi069 ceph-mon[29472]: osdmap e37: 6 total, 5 up, 6 in 2024-04-14T13:54:22.568 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:54:22 smithi069 ceph-mon[29472]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-14T13:54:22.568 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:54:22 smithi069 ceph-mon[29472]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-14T13:54:23.061 INFO:teuthology.orchestra.run.smithi122.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-04-14T13:54:23.061 INFO:teuthology.orchestra.run.smithi122.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-04-14T13:54:23.061 INFO:teuthology.orchestra.run.smithi122.stderr: stderr: 10+0 records in 2024-04-14T13:54:23.061 INFO:teuthology.orchestra.run.smithi122.stderr:10+0 records out 2024-04-14T13:54:23.061 INFO:teuthology.orchestra.run.smithi122.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0118822 s, 882 MB/s 2024-04-14T13:54:23.061 INFO:teuthology.orchestra.run.smithi122.stderr:--> Zapping successful for: 2024-04-14T13:54:23.345 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:54:23 smithi122 ceph-mon[25721]: purged_snaps scrub starts 2024-04-14T13:54:23.345 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:54:23 smithi122 ceph-mon[25721]: purged_snaps scrub ok 2024-04-14T13:54:23.345 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:54:23 smithi122 ceph-mon[25721]: pgmap v122: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2024-04-14T13:54:23.345 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:54:23 smithi122 ceph-mon[25721]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-14T13:54:23.345 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:54:23 smithi122 ceph-mon[25721]: osd.5 [v2:172.21.15.122:6808/4131566432,v1:172.21.15.122:6809/4131566432] boot 2024-04-14T13:54:23.345 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:54:23 smithi122 ceph-mon[25721]: osdmap e38: 6 total, 6 up, 6 in 2024-04-14T13:54:23.345 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:54:23 smithi122 ceph-mon[25721]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-14T13:54:23.567 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:54:23 smithi069 ceph-mon[24818]: purged_snaps scrub starts 2024-04-14T13:54:23.568 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:54:23 smithi069 ceph-mon[24818]: purged_snaps scrub ok 2024-04-14T13:54:23.568 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:54:23 smithi069 ceph-mon[24818]: pgmap v122: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2024-04-14T13:54:23.568 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:54:23 smithi069 ceph-mon[24818]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-14T13:54:23.568 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:54:23 smithi069 ceph-mon[24818]: osd.5 [v2:172.21.15.122:6808/4131566432,v1:172.21.15.122:6809/4131566432] boot 2024-04-14T13:54:23.568 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:54:23 smithi069 ceph-mon[24818]: osdmap e38: 6 total, 6 up, 6 in 2024-04-14T13:54:23.568 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:54:23 smithi069 ceph-mon[24818]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-14T13:54:23.568 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:54:23 smithi069 ceph-mon[29472]: purged_snaps scrub starts 2024-04-14T13:54:23.568 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:54:23 smithi069 ceph-mon[29472]: purged_snaps scrub ok 2024-04-14T13:54:23.568 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:54:23 smithi069 ceph-mon[29472]: pgmap v122: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2024-04-14T13:54:23.568 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:54:23 smithi069 ceph-mon[29472]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-14T13:54:23.568 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:54:23 smithi069 ceph-mon[29472]: osd.5 [v2:172.21.15.122:6808/4131566432,v1:172.21.15.122:6809/4131566432] boot 2024-04-14T13:54:23.568 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:54:23 smithi069 ceph-mon[29472]: osdmap e38: 6 total, 6 up, 6 in 2024-04-14T13:54:23.568 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:54:23 smithi069 ceph-mon[29472]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-14T13:54:23.736 DEBUG:teuthology.orchestra.run.smithi122:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid dc326fc2-fa65-11ee-bc8e-c7b262605968 -- ceph orch daemon add osd smithi122:vg_nvme/lv_2 2024-04-14T13:54:24.292 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:54:24 smithi122 ceph-mon[25721]: osdmap e39: 6 total, 6 up, 6 in 2024-04-14T13:54:24.567 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:54:24 smithi069 ceph-mon[24818]: osdmap e39: 6 total, 6 up, 6 in 2024-04-14T13:54:24.567 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:54:24 smithi069 ceph-mon[29472]: osdmap e39: 6 total, 6 up, 6 in 2024-04-14T13:54:25.432 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:54:25 smithi122 ceph-mon[25721]: pgmap v125: 1 pgs: 1 active+clean; 577 KiB data, 34 MiB used, 536 GiB / 536 GiB avail 2024-04-14T13:54:25.432 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:54:25 smithi122 ceph-mon[25721]: osdmap e40: 6 total, 6 up, 6 in 2024-04-14T13:54:25.432 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:54:25 smithi122 ceph-mon[25721]: Detected new or changed devices on smithi122 2024-04-14T13:54:25.432 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:54:25 smithi122 ceph-mon[25721]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' 2024-04-14T13:54:25.432 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:54:25 smithi122 ceph-mon[25721]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-14T13:54:25.432 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:54:25 smithi122 ceph-mon[25721]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-14T13:54:25.432 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:54:25 smithi122 ceph-mon[25721]: Adjusting osd_memory_target on smithi122 to 8544M 2024-04-14T13:54:25.433 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:54:25 smithi122 ceph-mon[25721]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' 2024-04-14T13:54:25.433 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:54:25 smithi122 ceph-mon[25721]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' 2024-04-14T13:54:25.433 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:54:25 smithi122 ceph-mon[25721]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-14T13:54:25.433 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:54:25 smithi122 ceph-mon[25721]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T13:54:25.433 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:54:25 smithi122 ceph-mon[25721]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-14T13:54:25.433 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:54:25 smithi122 ceph-mon[25721]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' 2024-04-14T13:54:25.567 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:54:25 smithi069 ceph-mon[24818]: pgmap v125: 1 pgs: 1 active+clean; 577 KiB data, 34 MiB used, 536 GiB / 536 GiB avail 2024-04-14T13:54:25.567 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:54:25 smithi069 ceph-mon[24818]: osdmap e40: 6 total, 6 up, 6 in 2024-04-14T13:54:25.568 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:54:25 smithi069 ceph-mon[24818]: Detected new or changed devices on smithi122 2024-04-14T13:54:25.568 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:54:25 smithi069 ceph-mon[24818]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' 2024-04-14T13:54:25.568 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:54:25 smithi069 ceph-mon[24818]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-14T13:54:25.568 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:54:25 smithi069 ceph-mon[24818]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-14T13:54:25.568 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:54:25 smithi069 ceph-mon[24818]: Adjusting osd_memory_target on smithi122 to 8544M 2024-04-14T13:54:25.568 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:54:25 smithi069 ceph-mon[24818]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' 2024-04-14T13:54:25.568 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:54:25 smithi069 ceph-mon[24818]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' 2024-04-14T13:54:25.568 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:54:25 smithi069 ceph-mon[24818]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-14T13:54:25.568 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:54:25 smithi069 ceph-mon[24818]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T13:54:25.568 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:54:25 smithi069 ceph-mon[24818]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-14T13:54:25.568 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:54:25 smithi069 ceph-mon[24818]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' 2024-04-14T13:54:25.568 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:54:25 smithi069 ceph-mon[29472]: pgmap v125: 1 pgs: 1 active+clean; 577 KiB data, 34 MiB used, 536 GiB / 536 GiB avail 2024-04-14T13:54:25.569 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:54:25 smithi069 ceph-mon[29472]: osdmap e40: 6 total, 6 up, 6 in 2024-04-14T13:54:25.569 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:54:25 smithi069 ceph-mon[29472]: Detected new or changed devices on smithi122 2024-04-14T13:54:25.569 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:54:25 smithi069 ceph-mon[29472]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' 2024-04-14T13:54:25.569 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:54:25 smithi069 ceph-mon[29472]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-14T13:54:25.569 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:54:25 smithi069 ceph-mon[29472]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-14T13:54:25.569 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:54:25 smithi069 ceph-mon[29472]: Adjusting osd_memory_target on smithi122 to 8544M 2024-04-14T13:54:25.569 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:54:25 smithi069 ceph-mon[29472]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' 2024-04-14T13:54:25.569 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:54:25 smithi069 ceph-mon[29472]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' 2024-04-14T13:54:25.569 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:54:25 smithi069 ceph-mon[29472]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-14T13:54:25.569 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:54:25 smithi069 ceph-mon[29472]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T13:54:25.569 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:54:25 smithi069 ceph-mon[29472]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-14T13:54:25.569 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:54:25 smithi069 ceph-mon[29472]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' 2024-04-14T13:54:26.455 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:54:26 smithi122 ceph-mon[25721]: osdmap e41: 6 total, 6 up, 6 in 2024-04-14T13:54:26.455 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:54:26 smithi122 ceph-mon[25721]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-14T13:54:26.455 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:54:26 smithi122 ceph-mon[25721]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-14T13:54:26.455 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:54:26 smithi122 ceph-mon[25721]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T13:54:26.567 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:54:26 smithi069 ceph-mon[24818]: osdmap e41: 6 total, 6 up, 6 in 2024-04-14T13:54:26.567 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:54:26 smithi069 ceph-mon[24818]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-14T13:54:26.568 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:54:26 smithi069 ceph-mon[24818]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-14T13:54:26.568 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:54:26 smithi069 ceph-mon[24818]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T13:54:26.568 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:54:26 smithi069 ceph-mon[29472]: osdmap e41: 6 total, 6 up, 6 in 2024-04-14T13:54:26.568 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:54:26 smithi069 ceph-mon[29472]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-14T13:54:26.568 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:54:26 smithi069 ceph-mon[29472]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-14T13:54:26.568 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:54:26 smithi069 ceph-mon[29472]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T13:54:27.489 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:54:27 smithi122 ceph-mon[25721]: pgmap v128: 1 pgs: 1 active+clean; 577 KiB data, 34 MiB used, 536 GiB / 536 GiB avail 2024-04-14T13:54:27.489 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:54:27 smithi122 ceph-mon[25721]: from='client.24242 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi122:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T13:54:27.489 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:54:27 smithi122 ceph-mon[25721]: osdmap e42: 6 total, 6 up, 6 in 2024-04-14T13:54:27.567 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:54:27 smithi069 ceph-mon[24818]: pgmap v128: 1 pgs: 1 active+clean; 577 KiB data, 34 MiB used, 536 GiB / 536 GiB avail 2024-04-14T13:54:27.567 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:54:27 smithi069 ceph-mon[24818]: from='client.24242 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi122:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T13:54:27.567 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:54:27 smithi069 ceph-mon[24818]: osdmap e42: 6 total, 6 up, 6 in 2024-04-14T13:54:27.568 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:54:27 smithi069 ceph-mon[29472]: pgmap v128: 1 pgs: 1 active+clean; 577 KiB data, 34 MiB used, 536 GiB / 536 GiB avail 2024-04-14T13:54:27.568 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:54:27 smithi069 ceph-mon[29472]: from='client.24242 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi122:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T13:54:27.568 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:54:27 smithi069 ceph-mon[29472]: osdmap e42: 6 total, 6 up, 6 in 2024-04-14T13:54:29.567 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:54:29 smithi069 ceph-mon[24818]: pgmap v130: 1 pgs: 1 active+clean; 577 KiB data, 34 MiB used, 536 GiB / 536 GiB avail 2024-04-14T13:54:29.567 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:54:29 smithi069 ceph-mon[24818]: from='client.? 172.21.15.122:0/2896626773' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "6f9f050d-8f2d-4275-8789-46022696eddf"}]: dispatch 2024-04-14T13:54:29.567 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:54:29 smithi069 ceph-mon[24818]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "6f9f050d-8f2d-4275-8789-46022696eddf"}]: dispatch 2024-04-14T13:54:29.568 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:54:29 smithi069 ceph-mon[24818]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "6f9f050d-8f2d-4275-8789-46022696eddf"}]': finished 2024-04-14T13:54:29.568 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:54:29 smithi069 ceph-mon[24818]: osdmap e43: 7 total, 6 up, 7 in 2024-04-14T13:54:29.568 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:54:29 smithi069 ceph-mon[24818]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-14T13:54:29.568 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:54:29 smithi069 ceph-mon[24818]: from='client.? 172.21.15.122:0/4231137365' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-14T13:54:29.568 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:54:29 smithi069 ceph-mon[29472]: pgmap v130: 1 pgs: 1 active+clean; 577 KiB data, 34 MiB used, 536 GiB / 536 GiB avail 2024-04-14T13:54:29.568 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:54:29 smithi069 ceph-mon[29472]: from='client.? 172.21.15.122:0/2896626773' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "6f9f050d-8f2d-4275-8789-46022696eddf"}]: dispatch 2024-04-14T13:54:29.568 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:54:29 smithi069 ceph-mon[29472]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "6f9f050d-8f2d-4275-8789-46022696eddf"}]: dispatch 2024-04-14T13:54:29.568 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:54:29 smithi069 ceph-mon[29472]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "6f9f050d-8f2d-4275-8789-46022696eddf"}]': finished 2024-04-14T13:54:29.568 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:54:29 smithi069 ceph-mon[29472]: osdmap e43: 7 total, 6 up, 7 in 2024-04-14T13:54:29.568 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:54:29 smithi069 ceph-mon[29472]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-14T13:54:29.568 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:54:29 smithi069 ceph-mon[29472]: from='client.? 172.21.15.122:0/4231137365' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-14T13:54:29.595 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:54:29 smithi122 ceph-mon[25721]: pgmap v130: 1 pgs: 1 active+clean; 577 KiB data, 34 MiB used, 536 GiB / 536 GiB avail 2024-04-14T13:54:29.595 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:54:29 smithi122 ceph-mon[25721]: from='client.? 172.21.15.122:0/2896626773' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "6f9f050d-8f2d-4275-8789-46022696eddf"}]: dispatch 2024-04-14T13:54:29.595 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:54:29 smithi122 ceph-mon[25721]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "6f9f050d-8f2d-4275-8789-46022696eddf"}]: dispatch 2024-04-14T13:54:29.595 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:54:29 smithi122 ceph-mon[25721]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "6f9f050d-8f2d-4275-8789-46022696eddf"}]': finished 2024-04-14T13:54:29.595 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:54:29 smithi122 ceph-mon[25721]: osdmap e43: 7 total, 6 up, 7 in 2024-04-14T13:54:29.595 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:54:29 smithi122 ceph-mon[25721]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-14T13:54:29.596 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:54:29 smithi122 ceph-mon[25721]: from='client.? 172.21.15.122:0/4231137365' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-14T13:54:31.466 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:54:31 smithi122 ceph-mon[25721]: pgmap v132: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-04-14T13:54:31.567 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:54:31 smithi069 ceph-mon[24818]: pgmap v132: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-04-14T13:54:31.567 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:54:31 smithi069 ceph-mon[29472]: pgmap v132: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-04-14T13:54:33.295 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:54:33 smithi122 ceph-mon[25721]: pgmap v133: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-04-14T13:54:33.567 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:54:33 smithi069 ceph-mon[24818]: pgmap v133: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-04-14T13:54:33.567 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:54:33 smithi069 ceph-mon[29472]: pgmap v133: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-04-14T13:54:35.486 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:54:35 smithi122 ceph-mon[25721]: pgmap v134: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-04-14T13:54:35.486 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:54:35 smithi122 ceph-mon[25721]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-04-14T13:54:35.486 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:54:35 smithi122 ceph-mon[25721]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T13:54:35.486 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:54:35 smithi122 ceph-mon[25721]: Deploying daemon osd.6 on smithi122 2024-04-14T13:54:35.567 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:54:35 smithi069 ceph-mon[24818]: pgmap v134: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-04-14T13:54:35.567 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:54:35 smithi069 ceph-mon[24818]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-04-14T13:54:35.567 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:54:35 smithi069 ceph-mon[24818]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T13:54:35.568 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:54:35 smithi069 ceph-mon[24818]: Deploying daemon osd.6 on smithi122 2024-04-14T13:54:35.568 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:54:35 smithi069 ceph-mon[29472]: pgmap v134: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-04-14T13:54:35.568 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:54:35 smithi069 ceph-mon[29472]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-04-14T13:54:35.568 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:54:35 smithi069 ceph-mon[29472]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T13:54:35.568 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:54:35 smithi069 ceph-mon[29472]: Deploying daemon osd.6 on smithi122 2024-04-14T13:54:37.548 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:54:37 smithi122 ceph-mon[25721]: pgmap v135: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-04-14T13:54:37.567 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:54:37 smithi069 ceph-mon[24818]: pgmap v135: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-04-14T13:54:37.567 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:54:37 smithi069 ceph-mon[29472]: pgmap v135: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-04-14T13:54:39.448 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:54:39 smithi122 ceph-mon[25721]: pgmap v136: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-04-14T13:54:39.541 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:54:39 smithi069 ceph-mon[24818]: pgmap v136: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-04-14T13:54:39.541 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:54:39 smithi069 ceph-mon[29472]: pgmap v136: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-04-14T13:54:40.817 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:54:40 smithi069 ceph-mon[24818]: pgmap v137: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-04-14T13:54:40.817 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:54:40 smithi069 ceph-mon[24818]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' 2024-04-14T13:54:40.818 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:54:40 smithi069 ceph-mon[24818]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-14T13:54:40.818 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:54:40 smithi069 ceph-mon[24818]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T13:54:40.818 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:54:40 smithi069 ceph-mon[24818]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-14T13:54:40.818 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:54:40 smithi069 ceph-mon[29472]: pgmap v137: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-04-14T13:54:40.818 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:54:40 smithi069 ceph-mon[29472]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' 2024-04-14T13:54:40.818 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:54:40 smithi069 ceph-mon[29472]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-14T13:54:40.818 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:54:40 smithi069 ceph-mon[29472]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T13:54:40.818 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:54:40 smithi069 ceph-mon[29472]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-14T13:54:40.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:54:40 smithi122 ceph-mon[25721]: pgmap v137: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-04-14T13:54:40.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:54:40 smithi122 ceph-mon[25721]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' 2024-04-14T13:54:40.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:54:40 smithi122 ceph-mon[25721]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-14T13:54:40.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:54:40 smithi122 ceph-mon[25721]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T13:54:40.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:54:40 smithi122 ceph-mon[25721]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-14T13:54:42.976 INFO:teuthology.orchestra.run.smithi122.stdout:Created osd(s) 6 on host 'smithi122' 2024-04-14T13:54:43.289 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:54:43 smithi122 ceph-mon[25721]: pgmap v138: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-04-14T13:54:43.290 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:54:43 smithi122 ceph-mon[25721]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' 2024-04-14T13:54:43.290 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:54:43 smithi122 ceph-mon[25721]: from='osd.6 [v2:172.21.15.122:6816/173885830,v1:172.21.15.122:6817/173885830]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-04-14T13:54:43.290 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:54:43 smithi122 ceph-mon[25721]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-04-14T13:54:43.290 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:54:43 smithi122 ceph-mon[25721]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' 2024-04-14T13:54:43.290 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:54:43 smithi122 ceph-mon[25721]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' 2024-04-14T13:54:43.290 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:54:43 smithi122 ceph-mon[25721]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-14T13:54:43.290 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:54:43 smithi122 ceph-mon[25721]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T13:54:43.290 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:54:43 smithi122 ceph-mon[25721]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-14T13:54:43.567 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:54:43 smithi069 ceph-mon[24818]: pgmap v138: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-04-14T13:54:43.567 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:54:43 smithi069 ceph-mon[24818]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' 2024-04-14T13:54:43.567 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:54:43 smithi069 ceph-mon[24818]: from='osd.6 [v2:172.21.15.122:6816/173885830,v1:172.21.15.122:6817/173885830]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-04-14T13:54:43.567 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:54:43 smithi069 ceph-mon[24818]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-04-14T13:54:43.568 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:54:43 smithi069 ceph-mon[24818]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' 2024-04-14T13:54:43.568 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:54:43 smithi069 ceph-mon[24818]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' 2024-04-14T13:54:43.568 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:54:43 smithi069 ceph-mon[24818]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-14T13:54:43.568 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:54:43 smithi069 ceph-mon[24818]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T13:54:43.568 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:54:43 smithi069 ceph-mon[24818]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-14T13:54:43.568 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:54:43 smithi069 ceph-mon[29472]: pgmap v138: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-04-14T13:54:43.568 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:54:43 smithi069 ceph-mon[29472]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' 2024-04-14T13:54:43.568 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:54:43 smithi069 ceph-mon[29472]: from='osd.6 [v2:172.21.15.122:6816/173885830,v1:172.21.15.122:6817/173885830]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-04-14T13:54:43.568 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:54:43 smithi069 ceph-mon[29472]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-04-14T13:54:43.569 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:54:43 smithi069 ceph-mon[29472]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' 2024-04-14T13:54:43.569 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:54:43 smithi069 ceph-mon[29472]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' 2024-04-14T13:54:43.569 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:54:43 smithi069 ceph-mon[29472]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-14T13:54:43.569 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:54:43 smithi069 ceph-mon[29472]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T13:54:43.569 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:54:43 smithi069 ceph-mon[29472]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-14T13:54:43.655 DEBUG:teuthology.orchestra.run.smithi122:osd.6> sudo journalctl -f -n 0 -u ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968@osd.6.service 2024-04-14T13:54:43.658 INFO:tasks.cephadm:Deploying osd.7 on smithi122 with /dev/vg_nvme/lv_1... 2024-04-14T13:54:43.659 DEBUG:teuthology.orchestra.run.smithi122:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid dc326fc2-fa65-11ee-bc8e-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2024-04-14T13:54:44.345 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:54:44 smithi122 ceph-mon[25721]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-04-14T13:54:44.345 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:54:44 smithi122 ceph-mon[25721]: osdmap e44: 7 total, 6 up, 7 in 2024-04-14T13:54:44.345 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:54:44 smithi122 ceph-mon[25721]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-14T13:54:44.345 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:54:44 smithi122 ceph-mon[25721]: from='osd.6 [v2:172.21.15.122:6816/173885830,v1:172.21.15.122:6817/173885830]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi122", "root=default"]}]: dispatch 2024-04-14T13:54:44.346 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:54:44 smithi122 ceph-mon[25721]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi122", "root=default"]}]: dispatch 2024-04-14T13:54:44.346 INFO:journalctl@ceph.osd.6.smithi122.stdout:Apr 14 13:54:44 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-osd-6[34955]: 2024-04-14T13:54:44.111+0000 7f6e60f2c700 -1 osd.6 0 waiting for initial osdmap 2024-04-14T13:54:44.346 INFO:journalctl@ceph.osd.6.smithi122.stdout:Apr 14 13:54:44 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-osd-6[34955]: 2024-04-14T13:54:44.117+0000 7f6e5b8c3700 -1 osd.6 45 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-14T13:54:44.567 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:54:44 smithi069 ceph-mon[24818]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-04-14T13:54:44.567 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:54:44 smithi069 ceph-mon[24818]: osdmap e44: 7 total, 6 up, 7 in 2024-04-14T13:54:44.567 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:54:44 smithi069 ceph-mon[24818]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-14T13:54:44.567 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:54:44 smithi069 ceph-mon[24818]: from='osd.6 [v2:172.21.15.122:6816/173885830,v1:172.21.15.122:6817/173885830]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi122", "root=default"]}]: dispatch 2024-04-14T13:54:44.568 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:54:44 smithi069 ceph-mon[24818]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi122", "root=default"]}]: dispatch 2024-04-14T13:54:44.568 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:54:44 smithi069 ceph-mon[29472]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-04-14T13:54:44.568 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:54:44 smithi069 ceph-mon[29472]: osdmap e44: 7 total, 6 up, 7 in 2024-04-14T13:54:44.568 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:54:44 smithi069 ceph-mon[29472]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-14T13:54:44.568 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:54:44 smithi069 ceph-mon[29472]: from='osd.6 [v2:172.21.15.122:6816/173885830,v1:172.21.15.122:6817/173885830]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi122", "root=default"]}]: dispatch 2024-04-14T13:54:44.568 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:54:44 smithi069 ceph-mon[29472]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi122", "root=default"]}]: dispatch 2024-04-14T13:54:45.425 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:54:45 smithi122 ceph-mon[25721]: pgmap v140: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-04-14T13:54:45.425 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:54:45 smithi122 ceph-mon[25721]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi122", "root=default"]}]': finished 2024-04-14T13:54:45.426 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:54:45 smithi122 ceph-mon[25721]: osdmap e45: 7 total, 6 up, 7 in 2024-04-14T13:54:45.426 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:54:45 smithi122 ceph-mon[25721]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-14T13:54:45.426 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:54:45 smithi122 ceph-mon[25721]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-14T13:54:45.426 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:54:45 smithi122 ceph-mon[25721]: osd.6 [v2:172.21.15.122:6816/173885830,v1:172.21.15.122:6817/173885830] boot 2024-04-14T13:54:45.426 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:54:45 smithi122 ceph-mon[25721]: osdmap e46: 7 total, 7 up, 7 in 2024-04-14T13:54:45.426 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:54:45 smithi122 ceph-mon[25721]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-14T13:54:45.567 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:54:45 smithi069 ceph-mon[24818]: pgmap v140: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-04-14T13:54:45.567 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:54:45 smithi069 ceph-mon[24818]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi122", "root=default"]}]': finished 2024-04-14T13:54:45.567 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:54:45 smithi069 ceph-mon[24818]: osdmap e45: 7 total, 6 up, 7 in 2024-04-14T13:54:45.567 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:54:45 smithi069 ceph-mon[24818]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-14T13:54:45.567 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:54:45 smithi069 ceph-mon[24818]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-14T13:54:45.568 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:54:45 smithi069 ceph-mon[24818]: osd.6 [v2:172.21.15.122:6816/173885830,v1:172.21.15.122:6817/173885830] boot 2024-04-14T13:54:45.568 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:54:45 smithi069 ceph-mon[24818]: osdmap e46: 7 total, 7 up, 7 in 2024-04-14T13:54:45.568 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:54:45 smithi069 ceph-mon[24818]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-14T13:54:45.568 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:54:45 smithi069 ceph-mon[29472]: pgmap v140: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-04-14T13:54:45.568 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:54:45 smithi069 ceph-mon[29472]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi122", "root=default"]}]': finished 2024-04-14T13:54:45.568 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:54:45 smithi069 ceph-mon[29472]: osdmap e45: 7 total, 6 up, 7 in 2024-04-14T13:54:45.568 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:54:45 smithi069 ceph-mon[29472]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-14T13:54:45.568 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:54:45 smithi069 ceph-mon[29472]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-14T13:54:45.568 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:54:45 smithi069 ceph-mon[29472]: osd.6 [v2:172.21.15.122:6816/173885830,v1:172.21.15.122:6817/173885830] boot 2024-04-14T13:54:45.568 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:54:45 smithi069 ceph-mon[29472]: osdmap e46: 7 total, 7 up, 7 in 2024-04-14T13:54:45.568 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:54:45 smithi069 ceph-mon[29472]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-14T13:54:46.345 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:54:46 smithi122 ceph-mon[25721]: purged_snaps scrub starts 2024-04-14T13:54:46.345 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:54:46 smithi122 ceph-mon[25721]: purged_snaps scrub ok 2024-04-14T13:54:46.345 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:54:46 smithi122 ceph-mon[25721]: osdmap e47: 7 total, 7 up, 7 in 2024-04-14T13:54:46.567 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:54:46 smithi069 ceph-mon[24818]: purged_snaps scrub starts 2024-04-14T13:54:46.567 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:54:46 smithi069 ceph-mon[24818]: purged_snaps scrub ok 2024-04-14T13:54:46.567 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:54:46 smithi069 ceph-mon[24818]: osdmap e47: 7 total, 7 up, 7 in 2024-04-14T13:54:46.567 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:54:46 smithi069 ceph-mon[29472]: purged_snaps scrub starts 2024-04-14T13:54:46.568 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:54:46 smithi069 ceph-mon[29472]: purged_snaps scrub ok 2024-04-14T13:54:46.568 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:54:46 smithi069 ceph-mon[29472]: osdmap e47: 7 total, 7 up, 7 in 2024-04-14T13:54:46.585 INFO:teuthology.orchestra.run.smithi122.stderr:--> Zapping: /dev/vg_nvme/lv_1 2024-04-14T13:54:46.585 INFO:teuthology.orchestra.run.smithi122.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2024-04-14T13:54:46.585 INFO:teuthology.orchestra.run.smithi122.stderr: stderr: 10+0 records in 2024-04-14T13:54:46.585 INFO:teuthology.orchestra.run.smithi122.stderr:10+0 records out 2024-04-14T13:54:46.585 INFO:teuthology.orchestra.run.smithi122.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0123192 s, 851 MB/s 2024-04-14T13:54:46.585 INFO:teuthology.orchestra.run.smithi122.stderr:--> Zapping successful for: 2024-04-14T13:54:47.463 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:54:47 smithi122 ceph-mon[25721]: pgmap v144: 1 pgs: 1 remapped+peering; 577 KiB data, 40 MiB used, 626 GiB / 626 GiB avail 2024-04-14T13:54:47.463 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:54:47 smithi122 ceph-mon[25721]: osdmap e48: 7 total, 7 up, 7 in 2024-04-14T13:54:47.485 DEBUG:teuthology.orchestra.run.smithi122:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid dc326fc2-fa65-11ee-bc8e-c7b262605968 -- ceph orch daemon add osd smithi122:vg_nvme/lv_1 2024-04-14T13:54:47.567 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:54:47 smithi069 ceph-mon[24818]: pgmap v144: 1 pgs: 1 remapped+peering; 577 KiB data, 40 MiB used, 626 GiB / 626 GiB avail 2024-04-14T13:54:47.567 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:54:47 smithi069 ceph-mon[24818]: osdmap e48: 7 total, 7 up, 7 in 2024-04-14T13:54:47.567 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:54:47 smithi069 ceph-mon[29472]: pgmap v144: 1 pgs: 1 remapped+peering; 577 KiB data, 40 MiB used, 626 GiB / 626 GiB avail 2024-04-14T13:54:47.567 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:54:47 smithi069 ceph-mon[29472]: osdmap e48: 7 total, 7 up, 7 in 2024-04-14T13:54:48.345 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:54:48 smithi122 ceph-mon[25721]: osdmap e49: 7 total, 7 up, 7 in 2024-04-14T13:54:48.345 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:54:48 smithi122 ceph-mon[25721]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' 2024-04-14T13:54:48.345 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:54:48 smithi122 ceph-mon[25721]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-14T13:54:48.345 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:54:48 smithi122 ceph-mon[25721]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-14T13:54:48.345 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:54:48 smithi122 ceph-mon[25721]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-14T13:54:48.346 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:54:48 smithi122 ceph-mon[25721]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' 2024-04-14T13:54:48.346 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:54:48 smithi122 ceph-mon[25721]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' 2024-04-14T13:54:48.567 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:54:48 smithi069 ceph-mon[24818]: osdmap e49: 7 total, 7 up, 7 in 2024-04-14T13:54:48.567 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:54:48 smithi069 ceph-mon[24818]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' 2024-04-14T13:54:48.568 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:54:48 smithi069 ceph-mon[24818]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-14T13:54:48.568 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:54:48 smithi069 ceph-mon[24818]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-14T13:54:48.568 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:54:48 smithi069 ceph-mon[24818]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-14T13:54:48.568 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:54:48 smithi069 ceph-mon[24818]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' 2024-04-14T13:54:48.568 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:54:48 smithi069 ceph-mon[24818]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' 2024-04-14T13:54:48.568 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:54:48 smithi069 ceph-mon[29472]: osdmap e49: 7 total, 7 up, 7 in 2024-04-14T13:54:48.568 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:54:48 smithi069 ceph-mon[29472]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' 2024-04-14T13:54:48.568 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:54:48 smithi069 ceph-mon[29472]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-14T13:54:48.568 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:54:48 smithi069 ceph-mon[29472]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-14T13:54:48.568 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:54:48 smithi069 ceph-mon[29472]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-14T13:54:48.568 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:54:48 smithi069 ceph-mon[29472]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' 2024-04-14T13:54:48.569 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:54:48 smithi069 ceph-mon[29472]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' 2024-04-14T13:54:49.309 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:54:49 smithi122 ceph-mon[25721]: pgmap v147: 1 pgs: 1 remapped+peering; 577 KiB data, 40 MiB used, 626 GiB / 626 GiB avail 2024-04-14T13:54:49.309 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:54:49 smithi122 ceph-mon[25721]: Detected new or changed devices on smithi122 2024-04-14T13:54:49.309 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:54:49 smithi122 ceph-mon[25721]: Adjusting osd_memory_target on smithi122 to 5696M 2024-04-14T13:54:49.310 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:54:49 smithi122 ceph-mon[25721]: osdmap e50: 7 total, 7 up, 7 in 2024-04-14T13:54:49.567 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:54:49 smithi069 ceph-mon[24818]: pgmap v147: 1 pgs: 1 remapped+peering; 577 KiB data, 40 MiB used, 626 GiB / 626 GiB avail 2024-04-14T13:54:49.567 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:54:49 smithi069 ceph-mon[24818]: Detected new or changed devices on smithi122 2024-04-14T13:54:49.567 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:54:49 smithi069 ceph-mon[24818]: Adjusting osd_memory_target on smithi122 to 5696M 2024-04-14T13:54:49.567 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:54:49 smithi069 ceph-mon[24818]: osdmap e50: 7 total, 7 up, 7 in 2024-04-14T13:54:49.567 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:54:49 smithi069 ceph-mon[29472]: pgmap v147: 1 pgs: 1 remapped+peering; 577 KiB data, 40 MiB used, 626 GiB / 626 GiB avail 2024-04-14T13:54:49.568 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:54:49 smithi069 ceph-mon[29472]: Detected new or changed devices on smithi122 2024-04-14T13:54:49.568 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:54:49 smithi069 ceph-mon[29472]: Adjusting osd_memory_target on smithi122 to 5696M 2024-04-14T13:54:49.568 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:54:49 smithi069 ceph-mon[29472]: osdmap e50: 7 total, 7 up, 7 in 2024-04-14T13:54:50.438 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:54:50 smithi122 ceph-mon[25721]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-14T13:54:50.438 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:54:50 smithi122 ceph-mon[25721]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-14T13:54:50.438 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:54:50 smithi122 ceph-mon[25721]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T13:54:50.567 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:54:50 smithi069 ceph-mon[24818]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-14T13:54:50.567 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:54:50 smithi069 ceph-mon[24818]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-14T13:54:50.567 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:54:50 smithi069 ceph-mon[24818]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T13:54:50.568 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:54:50 smithi069 ceph-mon[29472]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-14T13:54:50.568 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:54:50 smithi069 ceph-mon[29472]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-14T13:54:50.568 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:54:50 smithi069 ceph-mon[29472]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T13:54:51.404 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:54:51 smithi122 ceph-mon[25721]: pgmap v149: 1 pgs: 1 remapped+peering; 577 KiB data, 40 MiB used, 626 GiB / 626 GiB avail 2024-04-14T13:54:51.404 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:54:51 smithi122 ceph-mon[25721]: from='client.24269 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi122:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T13:54:51.567 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:54:51 smithi069 ceph-mon[24818]: pgmap v149: 1 pgs: 1 remapped+peering; 577 KiB data, 40 MiB used, 626 GiB / 626 GiB avail 2024-04-14T13:54:51.567 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:54:51 smithi069 ceph-mon[24818]: from='client.24269 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi122:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T13:54:51.567 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:54:51 smithi069 ceph-mon[29472]: pgmap v149: 1 pgs: 1 remapped+peering; 577 KiB data, 40 MiB used, 626 GiB / 626 GiB avail 2024-04-14T13:54:51.568 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:54:51 smithi069 ceph-mon[29472]: from='client.24269 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi122:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T13:54:52.567 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:54:52 smithi069 ceph-mon[24818]: from='client.? 172.21.15.122:0/1588597709' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "7ee70967-f2b6-4b0d-8973-9134433832d9"}]: dispatch 2024-04-14T13:54:52.567 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:54:52 smithi069 ceph-mon[24818]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "7ee70967-f2b6-4b0d-8973-9134433832d9"}]: dispatch 2024-04-14T13:54:52.567 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:54:52 smithi069 ceph-mon[24818]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "7ee70967-f2b6-4b0d-8973-9134433832d9"}]': finished 2024-04-14T13:54:52.567 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:54:52 smithi069 ceph-mon[24818]: osdmap e51: 8 total, 7 up, 8 in 2024-04-14T13:54:52.568 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:54:52 smithi069 ceph-mon[24818]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-14T13:54:52.568 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:54:52 smithi069 ceph-mon[29472]: from='client.? 172.21.15.122:0/1588597709' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "7ee70967-f2b6-4b0d-8973-9134433832d9"}]: dispatch 2024-04-14T13:54:52.568 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:54:52 smithi069 ceph-mon[29472]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "7ee70967-f2b6-4b0d-8973-9134433832d9"}]: dispatch 2024-04-14T13:54:52.568 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:54:52 smithi069 ceph-mon[29472]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "7ee70967-f2b6-4b0d-8973-9134433832d9"}]': finished 2024-04-14T13:54:52.568 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:54:52 smithi069 ceph-mon[29472]: osdmap e51: 8 total, 7 up, 8 in 2024-04-14T13:54:52.568 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:54:52 smithi069 ceph-mon[29472]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-14T13:54:52.595 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:54:52 smithi122 ceph-mon[25721]: from='client.? 172.21.15.122:0/1588597709' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "7ee70967-f2b6-4b0d-8973-9134433832d9"}]: dispatch 2024-04-14T13:54:52.595 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:54:52 smithi122 ceph-mon[25721]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "7ee70967-f2b6-4b0d-8973-9134433832d9"}]: dispatch 2024-04-14T13:54:52.595 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:54:52 smithi122 ceph-mon[25721]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "7ee70967-f2b6-4b0d-8973-9134433832d9"}]': finished 2024-04-14T13:54:52.595 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:54:52 smithi122 ceph-mon[25721]: osdmap e51: 8 total, 7 up, 8 in 2024-04-14T13:54:52.595 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:54:52 smithi122 ceph-mon[25721]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-14T13:54:53.567 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:54:53 smithi069 ceph-mon[24818]: pgmap v150: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail; 86 KiB/s, 0 objects/s recovering 2024-04-14T13:54:53.567 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:54:53 smithi069 ceph-mon[24818]: from='client.? 172.21.15.122:0/1951639400' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-14T13:54:53.568 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:54:53 smithi069 ceph-mon[29472]: pgmap v150: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail; 86 KiB/s, 0 objects/s recovering 2024-04-14T13:54:53.568 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:54:53 smithi069 ceph-mon[29472]: from='client.? 172.21.15.122:0/1951639400' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-14T13:54:53.595 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:54:53 smithi122 ceph-mon[25721]: pgmap v150: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail; 86 KiB/s, 0 objects/s recovering 2024-04-14T13:54:53.595 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:54:53 smithi122 ceph-mon[25721]: from='client.? 172.21.15.122:0/1951639400' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-14T13:54:55.567 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:54:55 smithi069 ceph-mon[24818]: pgmap v152: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail; 82 KiB/s, 0 objects/s recovering 2024-04-14T13:54:55.567 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:54:55 smithi069 ceph-mon[29472]: pgmap v152: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail; 82 KiB/s, 0 objects/s recovering 2024-04-14T13:54:55.595 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:54:55 smithi122 ceph-mon[25721]: pgmap v152: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail; 82 KiB/s, 0 objects/s recovering 2024-04-14T13:54:57.345 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:54:57 smithi122 ceph-mon[25721]: pgmap v153: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail; 65 KiB/s, 0 objects/s recovering 2024-04-14T13:54:57.567 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:54:57 smithi069 ceph-mon[24818]: pgmap v153: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail; 65 KiB/s, 0 objects/s recovering 2024-04-14T13:54:57.567 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:54:57 smithi069 ceph-mon[29472]: pgmap v153: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail; 65 KiB/s, 0 objects/s recovering 2024-04-14T13:54:59.567 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:54:59 smithi069 ceph-mon[24818]: pgmap v154: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail; 56 KiB/s, 0 objects/s recovering 2024-04-14T13:54:59.567 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:54:59 smithi069 ceph-mon[24818]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-04-14T13:54:59.567 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:54:59 smithi069 ceph-mon[24818]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T13:54:59.567 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:54:59 smithi069 ceph-mon[24818]: Deploying daemon osd.7 on smithi122 2024-04-14T13:54:59.568 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:54:59 smithi069 ceph-mon[29472]: pgmap v154: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail; 56 KiB/s, 0 objects/s recovering 2024-04-14T13:54:59.568 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:54:59 smithi069 ceph-mon[29472]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-04-14T13:54:59.568 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:54:59 smithi069 ceph-mon[29472]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T13:54:59.568 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:54:59 smithi069 ceph-mon[29472]: Deploying daemon osd.7 on smithi122 2024-04-14T13:54:59.595 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:54:59 smithi122 ceph-mon[25721]: pgmap v154: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail; 56 KiB/s, 0 objects/s recovering 2024-04-14T13:54:59.595 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:54:59 smithi122 ceph-mon[25721]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-04-14T13:54:59.595 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:54:59 smithi122 ceph-mon[25721]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T13:54:59.595 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:54:59 smithi122 ceph-mon[25721]: Deploying daemon osd.7 on smithi122 2024-04-14T13:55:01.567 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:55:01 smithi069 ceph-mon[24818]: pgmap v155: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail; 52 KiB/s, 0 objects/s recovering 2024-04-14T13:55:01.567 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:55:01 smithi069 ceph-mon[29472]: pgmap v155: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail; 52 KiB/s, 0 objects/s recovering 2024-04-14T13:55:01.595 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:55:01 smithi122 ceph-mon[25721]: pgmap v155: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail; 52 KiB/s, 0 objects/s recovering 2024-04-14T13:55:03.495 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:55:03 smithi122 ceph-mon[25721]: pgmap v156: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2024-04-14T13:55:03.567 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:55:03 smithi069 ceph-mon[24818]: pgmap v156: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2024-04-14T13:55:03.567 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:55:03 smithi069 ceph-mon[29472]: pgmap v156: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2024-04-14T13:55:04.817 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:55:04 smithi069 ceph-mon[24818]: pgmap v157: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2024-04-14T13:55:04.817 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:55:04 smithi069 ceph-mon[24818]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' 2024-04-14T13:55:04.817 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:55:04 smithi069 ceph-mon[24818]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-14T13:55:04.817 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:55:04 smithi069 ceph-mon[24818]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T13:55:04.817 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:55:04 smithi069 ceph-mon[24818]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-14T13:55:04.818 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:55:04 smithi069 ceph-mon[29472]: pgmap v157: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2024-04-14T13:55:04.818 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:55:04 smithi069 ceph-mon[29472]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' 2024-04-14T13:55:04.818 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:55:04 smithi069 ceph-mon[29472]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-14T13:55:04.818 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:55:04 smithi069 ceph-mon[29472]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T13:55:04.818 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:55:04 smithi069 ceph-mon[29472]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-14T13:55:04.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:55:04 smithi122 ceph-mon[25721]: pgmap v157: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2024-04-14T13:55:04.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:55:04 smithi122 ceph-mon[25721]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' 2024-04-14T13:55:04.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:55:04 smithi122 ceph-mon[25721]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-14T13:55:04.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:55:04 smithi122 ceph-mon[25721]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T13:55:04.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:55:04 smithi122 ceph-mon[25721]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-14T13:55:07.026 INFO:teuthology.orchestra.run.smithi122.stdout:Created osd(s) 7 on host 'smithi122' 2024-04-14T13:55:07.567 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:55:07 smithi069 ceph-mon[24818]: pgmap v158: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2024-04-14T13:55:07.567 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:55:07 smithi069 ceph-mon[24818]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' 2024-04-14T13:55:07.567 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:55:07 smithi069 ceph-mon[24818]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' 2024-04-14T13:55:07.568 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:55:07 smithi069 ceph-mon[24818]: from='osd.7 [v2:172.21.15.122:6824/2953794780,v1:172.21.15.122:6825/2953794780]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-04-14T13:55:07.568 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:55:07 smithi069 ceph-mon[24818]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-04-14T13:55:07.568 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:55:07 smithi069 ceph-mon[24818]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' 2024-04-14T13:55:07.568 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:55:07 smithi069 ceph-mon[24818]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-14T13:55:07.568 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:55:07 smithi069 ceph-mon[24818]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T13:55:07.568 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:55:07 smithi069 ceph-mon[24818]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-14T13:55:07.568 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:55:07 smithi069 ceph-mon[29472]: pgmap v158: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2024-04-14T13:55:07.568 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:55:07 smithi069 ceph-mon[29472]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' 2024-04-14T13:55:07.568 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:55:07 smithi069 ceph-mon[29472]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' 2024-04-14T13:55:07.568 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:55:07 smithi069 ceph-mon[29472]: from='osd.7 [v2:172.21.15.122:6824/2953794780,v1:172.21.15.122:6825/2953794780]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-04-14T13:55:07.568 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:55:07 smithi069 ceph-mon[29472]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-04-14T13:55:07.568 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:55:07 smithi069 ceph-mon[29472]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' 2024-04-14T13:55:07.569 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:55:07 smithi069 ceph-mon[29472]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-14T13:55:07.569 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:55:07 smithi069 ceph-mon[29472]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T13:55:07.569 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:55:07 smithi069 ceph-mon[29472]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-14T13:55:07.591 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:55:07 smithi122 ceph-mon[25721]: pgmap v158: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2024-04-14T13:55:07.591 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:55:07 smithi122 ceph-mon[25721]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' 2024-04-14T13:55:07.591 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:55:07 smithi122 ceph-mon[25721]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' 2024-04-14T13:55:07.591 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:55:07 smithi122 ceph-mon[25721]: from='osd.7 [v2:172.21.15.122:6824/2953794780,v1:172.21.15.122:6825/2953794780]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-04-14T13:55:07.591 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:55:07 smithi122 ceph-mon[25721]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-04-14T13:55:07.591 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:55:07 smithi122 ceph-mon[25721]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' 2024-04-14T13:55:07.591 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:55:07 smithi122 ceph-mon[25721]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-14T13:55:07.592 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:55:07 smithi122 ceph-mon[25721]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T13:55:07.592 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:55:07 smithi122 ceph-mon[25721]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-14T13:55:07.614 DEBUG:teuthology.orchestra.run.smithi122:osd.7> sudo journalctl -f -n 0 -u ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968@osd.7.service 2024-04-14T13:55:07.616 INFO:tasks.cephadm:Waiting for 8 OSDs to come up... 2024-04-14T13:55:07.617 DEBUG:teuthology.orchestra.run.smithi069:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid dc326fc2-fa65-11ee-bc8e-c7b262605968 -- ceph osd stat -f json 2024-04-14T13:55:08.567 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:55:08 smithi069 ceph-mon[24818]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-04-14T13:55:08.567 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:55:08 smithi069 ceph-mon[24818]: osdmap e52: 8 total, 7 up, 8 in 2024-04-14T13:55:08.567 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:55:08 smithi069 ceph-mon[24818]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-14T13:55:08.567 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:55:08 smithi069 ceph-mon[24818]: from='osd.7 [v2:172.21.15.122:6824/2953794780,v1:172.21.15.122:6825/2953794780]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi122", "root=default"]}]: dispatch 2024-04-14T13:55:08.568 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:55:08 smithi069 ceph-mon[24818]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi122", "root=default"]}]: dispatch 2024-04-14T13:55:08.568 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:55:08 smithi069 ceph-mon[29472]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-04-14T13:55:08.568 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:55:08 smithi069 ceph-mon[29472]: osdmap e52: 8 total, 7 up, 8 in 2024-04-14T13:55:08.568 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:55:08 smithi069 ceph-mon[29472]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-14T13:55:08.568 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:55:08 smithi069 ceph-mon[29472]: from='osd.7 [v2:172.21.15.122:6824/2953794780,v1:172.21.15.122:6825/2953794780]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi122", "root=default"]}]: dispatch 2024-04-14T13:55:08.568 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:55:08 smithi069 ceph-mon[29472]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi122", "root=default"]}]: dispatch 2024-04-14T13:55:08.597 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:55:08 smithi122 ceph-mon[25721]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-04-14T13:55:08.598 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:55:08 smithi122 ceph-mon[25721]: osdmap e52: 8 total, 7 up, 8 in 2024-04-14T13:55:08.598 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:55:08 smithi122 ceph-mon[25721]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-14T13:55:08.598 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:55:08 smithi122 ceph-mon[25721]: from='osd.7 [v2:172.21.15.122:6824/2953794780,v1:172.21.15.122:6825/2953794780]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi122", "root=default"]}]: dispatch 2024-04-14T13:55:08.598 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:55:08 smithi122 ceph-mon[25721]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi122", "root=default"]}]: dispatch 2024-04-14T13:55:08.598 INFO:journalctl@ceph.osd.7.smithi122.stdout:Apr 14 13:55:08 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-osd-7[37950]: 2024-04-14T13:55:08.305+0000 7f6197b8f700 -1 osd.7 0 waiting for initial osdmap 2024-04-14T13:55:08.598 INFO:journalctl@ceph.osd.7.smithi122.stdout:Apr 14 13:55:08 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-osd-7[37950]: 2024-04-14T13:55:08.312+0000 7f6192526700 -1 osd.7 53 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-14T13:55:09.462 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-14T13:55:09.463 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:55:09 smithi069 ceph-mon[24818]: pgmap v160: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2024-04-14T13:55:09.463 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:55:09 smithi069 ceph-mon[24818]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi122", "root=default"]}]': finished 2024-04-14T13:55:09.463 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:55:09 smithi069 ceph-mon[24818]: osdmap e53: 8 total, 7 up, 8 in 2024-04-14T13:55:09.463 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:55:09 smithi069 ceph-mon[24818]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-14T13:55:09.463 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:55:09 smithi069 ceph-mon[24818]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-14T13:55:09.463 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:55:09 smithi069 ceph-mon[29472]: pgmap v160: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2024-04-14T13:55:09.463 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:55:09 smithi069 ceph-mon[29472]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi122", "root=default"]}]': finished 2024-04-14T13:55:09.463 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:55:09 smithi069 ceph-mon[29472]: osdmap e53: 8 total, 7 up, 8 in 2024-04-14T13:55:09.463 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:55:09 smithi069 ceph-mon[29472]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-14T13:55:09.463 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:55:09 smithi069 ceph-mon[29472]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-14T13:55:09.595 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:55:09 smithi122 ceph-mon[25721]: pgmap v160: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2024-04-14T13:55:09.595 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:55:09 smithi122 ceph-mon[25721]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi122", "root=default"]}]': finished 2024-04-14T13:55:09.595 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:55:09 smithi122 ceph-mon[25721]: osdmap e53: 8 total, 7 up, 8 in 2024-04-14T13:55:09.596 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:55:09 smithi122 ceph-mon[25721]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-14T13:55:09.596 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:55:09 smithi122 ceph-mon[25721]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-14T13:55:10.196 INFO:teuthology.orchestra.run.smithi069.stdout:{"epoch":54,"num_osds":8,"num_up_osds":8,"osd_up_since":1713102909,"num_in_osds":8,"osd_in_since":1713102892,"num_remapped_pgs":1} 2024-04-14T13:55:10.197 DEBUG:teuthology.orchestra.run.smithi069:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid dc326fc2-fa65-11ee-bc8e-c7b262605968 -- ceph osd dump --format=json 2024-04-14T13:55:10.413 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:55:10 smithi122 ceph-mon[25721]: purged_snaps scrub starts 2024-04-14T13:55:10.413 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:55:10 smithi122 ceph-mon[25721]: purged_snaps scrub ok 2024-04-14T13:55:10.413 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:55:10 smithi122 ceph-mon[25721]: osd.7 [v2:172.21.15.122:6824/2953794780,v1:172.21.15.122:6825/2953794780] boot 2024-04-14T13:55:10.413 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:55:10 smithi122 ceph-mon[25721]: osdmap e54: 8 total, 8 up, 8 in 2024-04-14T13:55:10.413 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:55:10 smithi122 ceph-mon[25721]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-14T13:55:10.413 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:55:10 smithi122 ceph-mon[25721]: from='client.? 172.21.15.69:0/145463381' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-04-14T13:55:10.413 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:55:10 smithi122 ceph-mon[25721]: osdmap e55: 8 total, 8 up, 8 in 2024-04-14T13:55:10.416 INFO:teuthology.orchestra.run.smithi069.stderr:Inferring config /var/lib/ceph/dc326fc2-fa65-11ee-bc8e-c7b262605968/mon.a/config 2024-04-14T13:55:10.536 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:55:10 smithi069 ceph-mon[24818]: purged_snaps scrub starts 2024-04-14T13:55:10.537 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:55:10 smithi069 ceph-mon[24818]: purged_snaps scrub ok 2024-04-14T13:55:10.537 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:55:10 smithi069 ceph-mon[24818]: osd.7 [v2:172.21.15.122:6824/2953794780,v1:172.21.15.122:6825/2953794780] boot 2024-04-14T13:55:10.537 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:55:10 smithi069 ceph-mon[24818]: osdmap e54: 8 total, 8 up, 8 in 2024-04-14T13:55:10.537 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:55:10 smithi069 ceph-mon[24818]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-14T13:55:10.537 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:55:10 smithi069 ceph-mon[24818]: from='client.? 172.21.15.69:0/145463381' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-04-14T13:55:10.537 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:55:10 smithi069 ceph-mon[24818]: osdmap e55: 8 total, 8 up, 8 in 2024-04-14T13:55:10.537 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:55:10 smithi069 ceph-mon[29472]: purged_snaps scrub starts 2024-04-14T13:55:10.538 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:55:10 smithi069 ceph-mon[29472]: purged_snaps scrub ok 2024-04-14T13:55:10.538 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:55:10 smithi069 ceph-mon[29472]: osd.7 [v2:172.21.15.122:6824/2953794780,v1:172.21.15.122:6825/2953794780] boot 2024-04-14T13:55:10.538 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:55:10 smithi069 ceph-mon[29472]: osdmap e54: 8 total, 8 up, 8 in 2024-04-14T13:55:10.538 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:55:10 smithi069 ceph-mon[29472]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-14T13:55:10.538 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:55:10 smithi069 ceph-mon[29472]: from='client.? 172.21.15.69:0/145463381' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-04-14T13:55:10.538 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:55:10 smithi069 ceph-mon[29472]: osdmap e55: 8 total, 8 up, 8 in 2024-04-14T13:55:11.758 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:55:11 smithi069 ceph-mon[24818]: pgmap v163: 1 pgs: 1 active+clean; 577 KiB data, 47 MiB used, 715 GiB / 715 GiB avail 2024-04-14T13:55:11.758 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:55:11 smithi069 ceph-mon[24818]: Detected new or changed devices on smithi122 2024-04-14T13:55:11.758 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:55:11 smithi069 ceph-mon[24818]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' 2024-04-14T13:55:11.758 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:55:11 smithi069 ceph-mon[24818]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-14T13:55:11.758 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:55:11 smithi069 ceph-mon[24818]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-14T13:55:11.759 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:55:11 smithi069 ceph-mon[24818]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-14T13:55:11.759 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:55:11 smithi069 ceph-mon[24818]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-14T13:55:11.759 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:55:11 smithi069 ceph-mon[24818]: Adjusting osd_memory_target on smithi122 to 4272M 2024-04-14T13:55:11.759 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:55:11 smithi069 ceph-mon[24818]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' 2024-04-14T13:55:11.759 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:55:11 smithi069 ceph-mon[24818]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' 2024-04-14T13:55:11.759 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:55:11 smithi069 ceph-mon[24818]: osdmap e56: 8 total, 8 up, 8 in 2024-04-14T13:55:11.759 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:55:11 smithi069 ceph-mon[29472]: pgmap v163: 1 pgs: 1 active+clean; 577 KiB data, 47 MiB used, 715 GiB / 715 GiB avail 2024-04-14T13:55:11.759 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:55:11 smithi069 ceph-mon[29472]: Detected new or changed devices on smithi122 2024-04-14T13:55:11.759 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:55:11 smithi069 ceph-mon[29472]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' 2024-04-14T13:55:11.759 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:55:11 smithi069 ceph-mon[29472]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-14T13:55:11.760 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:55:11 smithi069 ceph-mon[29472]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-14T13:55:11.760 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:55:11 smithi069 ceph-mon[29472]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-14T13:55:11.760 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:55:11 smithi069 ceph-mon[29472]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-14T13:55:11.760 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:55:11 smithi069 ceph-mon[29472]: Adjusting osd_memory_target on smithi122 to 4272M 2024-04-14T13:55:11.760 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:55:11 smithi069 ceph-mon[29472]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' 2024-04-14T13:55:11.760 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:55:11 smithi069 ceph-mon[29472]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' 2024-04-14T13:55:11.760 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:55:11 smithi069 ceph-mon[29472]: osdmap e56: 8 total, 8 up, 8 in 2024-04-14T13:55:11.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:55:11 smithi122 ceph-mon[25721]: pgmap v163: 1 pgs: 1 active+clean; 577 KiB data, 47 MiB used, 715 GiB / 715 GiB avail 2024-04-14T13:55:11.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:55:11 smithi122 ceph-mon[25721]: Detected new or changed devices on smithi122 2024-04-14T13:55:11.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:55:11 smithi122 ceph-mon[25721]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' 2024-04-14T13:55:11.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:55:11 smithi122 ceph-mon[25721]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-14T13:55:11.846 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:55:11 smithi122 ceph-mon[25721]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-14T13:55:11.846 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:55:11 smithi122 ceph-mon[25721]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-14T13:55:11.846 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:55:11 smithi122 ceph-mon[25721]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-14T13:55:11.846 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:55:11 smithi122 ceph-mon[25721]: Adjusting osd_memory_target on smithi122 to 4272M 2024-04-14T13:55:11.846 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:55:11 smithi122 ceph-mon[25721]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' 2024-04-14T13:55:11.846 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:55:11 smithi122 ceph-mon[25721]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' 2024-04-14T13:55:11.846 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:55:11 smithi122 ceph-mon[25721]: osdmap e56: 8 total, 8 up, 8 in 2024-04-14T13:55:12.121 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-14T13:55:12.121 INFO:teuthology.orchestra.run.smithi069.stdout:{"epoch":56,"fsid":"dc326fc2-fa65-11ee-bc8e-c7b262605968","created":"2024-04-14T13:50:21.024738+0000","modified":"2024-04-14T13:55:11.172236+0000","last_up_change":"2024-04-14T13:55:09.296070+0000","last_in_change":"2024-04-14T13:54:52.122409+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":18,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-14T13:53:15.745972+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}}],"osds":[{"osd":0,"uuid":"99aae7c5-9755-49e8-91c7-fceed3ff30fb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6802","nonce":2517323608},{"type":"v1","addr":"172.21.15.69:6803","nonce":2517323608}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6804","nonce":2517323608},{"type":"v1","addr":"172.21.15.69:6805","nonce":2517323608}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6808","nonce":2517323608},{"type":"v1","addr":"172.21.15.69:6809","nonce":2517323608}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6806","nonce":2517323608},{"type":"v1","addr":"172.21.15.69:6807","nonce":2517323608}]},"public_addr":"172.21.15.69:6803/2517323608","cluster_addr":"172.21.15.69:6805/2517323608","heartbeat_back_addr":"172.21.15.69:6809/2517323608","heartbeat_front_addr":"172.21.15.69:6807/2517323608","state":["exists","up"]},{"osd":1,"uuid":"1d99b840-344d-422a-a466-13ed2d956280","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6810","nonce":1377309913},{"type":"v1","addr":"172.21.15.69:6811","nonce":1377309913}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6812","nonce":1377309913},{"type":"v1","addr":"172.21.15.69:6813","nonce":1377309913}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6816","nonce":1377309913},{"type":"v1","addr":"172.21.15.69:6817","nonce":1377309913}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6814","nonce":1377309913},{"type":"v1","addr":"172.21.15.69:6815","nonce":1377309913}]},"public_addr":"172.21.15.69:6811/1377309913","cluster_addr":"172.21.15.69:6813/1377309913","heartbeat_back_addr":"172.21.15.69:6817/1377309913","heartbeat_front_addr":"172.21.15.69:6815/1377309913","state":["exists","up"]},{"osd":2,"uuid":"f4e0dcf9-5003-4a8f-8a51-fa36f58c0ac4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6818","nonce":2090584066},{"type":"v1","addr":"172.21.15.69:6819","nonce":2090584066}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6820","nonce":2090584066},{"type":"v1","addr":"172.21.15.69:6821","nonce":2090584066}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6824","nonce":2090584066},{"type":"v1","addr":"172.21.15.69:6825","nonce":2090584066}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6822","nonce":2090584066},{"type":"v1","addr":"172.21.15.69:6823","nonce":2090584066}]},"public_addr":"172.21.15.69:6819/2090584066","cluster_addr":"172.21.15.69:6821/2090584066","heartbeat_back_addr":"172.21.15.69:6825/2090584066","heartbeat_front_addr":"172.21.15.69:6823/2090584066","state":["exists","up"]},{"osd":3,"uuid":"27e543e2-f5c0-462d-ab23-d439e06c056e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6826","nonce":148592620},{"type":"v1","addr":"172.21.15.69:6827","nonce":148592620}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6828","nonce":148592620},{"type":"v1","addr":"172.21.15.69:6829","nonce":148592620}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6832","nonce":148592620},{"type":"v1","addr":"172.21.15.69:6833","nonce":148592620}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6830","nonce":148592620},{"type":"v1","addr":"172.21.15.69:6831","nonce":148592620}]},"public_addr":"172.21.15.69:6827/148592620","cluster_addr":"172.21.15.69:6829/148592620","heartbeat_back_addr":"172.21.15.69:6833/148592620","heartbeat_front_addr":"172.21.15.69:6831/148592620","state":["exists","up"]},{"osd":4,"uuid":"cc9f056c-1e48-41c6-80db-dd1ce38c11db","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":30,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6800","nonce":3006695914},{"type":"v1","addr":"172.21.15.122:6801","nonce":3006695914}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6802","nonce":3006695914},{"type":"v1","addr":"172.21.15.122:6803","nonce":3006695914}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6806","nonce":3006695914},{"type":"v1","addr":"172.21.15.122:6807","nonce":3006695914}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6804","nonce":3006695914},{"type":"v1","addr":"172.21.15.122:6805","nonce":3006695914}]},"public_addr":"172.21.15.122:6801/3006695914","cluster_addr":"172.21.15.122:6803/3006695914","heartbeat_back_addr":"172.21.15.122:6807/3006695914","heartbeat_front_addr":"172.21.15.122:6805/3006695914","state":["exists","up"]},{"osd":5,"uuid":"c127fffe-79af-4bf0-b67e-a204f716ab14","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":41,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6808","nonce":4131566432},{"type":"v1","addr":"172.21.15.122:6809","nonce":4131566432}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6810","nonce":4131566432},{"type":"v1","addr":"172.21.15.122:6811","nonce":4131566432}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6814","nonce":4131566432},{"type":"v1","addr":"172.21.15.122:6815","nonce":4131566432}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6812","nonce":4131566432},{"type":"v1","addr":"172.21.15.122:6813","nonce":4131566432}]},"public_addr":"172.21.15.122:6809/4131566432","cluster_addr":"172.21.15.122:6811/4131566432","heartbeat_back_addr":"172.21.15.122:6815/4131566432","heartbeat_front_addr":"172.21.15.122:6813/4131566432","state":["exists","up"]},{"osd":6,"uuid":"6f9f050d-8f2d-4275-8789-46022696eddf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":49,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6816","nonce":173885830},{"type":"v1","addr":"172.21.15.122:6817","nonce":173885830}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6818","nonce":173885830},{"type":"v1","addr":"172.21.15.122:6819","nonce":173885830}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6822","nonce":173885830},{"type":"v1","addr":"172.21.15.122:6823","nonce":173885830}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6820","nonce":173885830},{"type":"v1","addr":"172.21.15.122:6821","nonce":173885830}]},"public_addr":"172.21.15.122:6817/173885830","cluster_addr":"172.21.15.122:6819/173885830","heartbeat_back_addr":"172.21.15.122:6823/173885830","heartbeat_front_addr":"172.21.15.122:6821/173885830","state":["exists","up"]},{"osd":7,"uuid":"7ee70967-f2b6-4b0d-8973-9134433832d9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":54,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6824","nonce":2953794780},{"type":"v1","addr":"172.21.15.122:6825","nonce":2953794780}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6826","nonce":2953794780},{"type":"v1","addr":"172.21.15.122:6827","nonce":2953794780}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6830","nonce":2953794780},{"type":"v1","addr":"172.21.15.122:6831","nonce":2953794780}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6828","nonce":2953794780},{"type":"v1","addr":"172.21.15.122:6829","nonce":2953794780}]},"public_addr":"172.21.15.122:6825/2953794780","cluster_addr":"172.21.15.122:6827/2953794780","heartbeat_back_addr":"172.21.15.122:6831/2953794780","heartbeat_front_addr":"172.21.15.122:6829/2953794780","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-04-14T13:52:27.650496+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-04-14T13:52:50.294211+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-04-14T13:53:12.866222+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-04-14T13:53:35.936737+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-04-14T13:53:56.970934+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-04-14T13:54:20.266792+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-04-14T13:54:43.509058+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-04-14T13:55:07.675683+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[{"pgid":"1.0","osds":[0,6]}],"primary_temp":[],"blocklist":{"172.21.15.69:0/3170426090":"2024-04-15T13:51:11.392969+0000","172.21.15.69:0/3397430943":"2024-04-15T13:51:11.392969+0000","172.21.15.69:0/2434136662":"2024-04-15T13:51:11.392969+0000","172.21.15.69:6800/2748095016":"2024-04-15T13:50:49.340545+0000","172.21.15.69:6801/3528824566":"2024-04-15T13:51:11.392969+0000","172.21.15.69:6800/3528824566":"2024-04-15T13:51:11.392969+0000","172.21.15.69:6801/2748095016":"2024-04-15T13:50:49.340545+0000","172.21.15.69:0/3759166769":"2024-04-15T13:50:49.340545+0000","172.21.15.69:0/2144826942":"2024-04-15T13:50:49.340545+0000","172.21.15.69:0/1250951073":"2024-04-15T13:50:49.340545+0000"},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-14T13:55:12.698 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:55:12 smithi069 ceph-mon[24818]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-04-14T13:55:12.698 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:55:12 smithi069 ceph-mon[24818]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-04-14T13:55:12.698 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:55:12 smithi069 ceph-mon[24818]: from='client.? 172.21.15.69:0/3918366976' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-14T13:55:12.698 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:55:12 smithi069 ceph-mon[24818]: osdmap e57: 8 total, 8 up, 8 in 2024-04-14T13:55:12.698 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:55:12 smithi069 ceph-mon[29472]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-04-14T13:55:12.699 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:55:12 smithi069 ceph-mon[29472]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-04-14T13:55:12.699 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:55:12 smithi069 ceph-mon[29472]: from='client.? 172.21.15.69:0/3918366976' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-14T13:55:12.699 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:55:12 smithi069 ceph-mon[29472]: osdmap e57: 8 total, 8 up, 8 in 2024-04-14T13:55:12.721 INFO:tasks.cephadm.ceph_manager.ceph:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-14T13:53:15.745972+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}}] 2024-04-14T13:55:12.721 DEBUG:teuthology.orchestra.run.smithi069:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid dc326fc2-fa65-11ee-bc8e-c7b262605968 -- ceph osd pool get .mgr pg_num 2024-04-14T13:55:12.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:55:12 smithi122 ceph-mon[25721]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-04-14T13:55:12.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:55:12 smithi122 ceph-mon[25721]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-04-14T13:55:12.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:55:12 smithi122 ceph-mon[25721]: from='client.? 172.21.15.69:0/3918366976' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-14T13:55:12.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:55:12 smithi122 ceph-mon[25721]: osdmap e57: 8 total, 8 up, 8 in 2024-04-14T13:55:12.935 INFO:teuthology.orchestra.run.smithi069.stderr:Inferring config /var/lib/ceph/dc326fc2-fa65-11ee-bc8e-c7b262605968/mon.a/config 2024-04-14T13:55:13.817 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:55:13 smithi069 ceph-mon[24818]: pgmap v166: 1 pgs: 1 remapped+peering; 577 KiB data, 47 MiB used, 715 GiB / 715 GiB avail 2024-04-14T13:55:13.817 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:55:13 smithi069 ceph-mon[24818]: osdmap e58: 8 total, 8 up, 8 in 2024-04-14T13:55:13.817 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:55:13 smithi069 ceph-mon[29472]: pgmap v166: 1 pgs: 1 remapped+peering; 577 KiB data, 47 MiB used, 715 GiB / 715 GiB avail 2024-04-14T13:55:13.817 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:55:13 smithi069 ceph-mon[29472]: osdmap e58: 8 total, 8 up, 8 in 2024-04-14T13:55:13.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:55:13 smithi122 ceph-mon[25721]: pgmap v166: 1 pgs: 1 remapped+peering; 577 KiB data, 47 MiB used, 715 GiB / 715 GiB avail 2024-04-14T13:55:13.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:55:13 smithi122 ceph-mon[25721]: osdmap e58: 8 total, 8 up, 8 in 2024-04-14T13:55:14.672 INFO:teuthology.orchestra.run.smithi069.stdout:pg_num: 1 2024-04-14T13:55:15.343 INFO:tasks.cephadm:Adding prometheus.a on smithi122 2024-04-14T13:55:15.344 DEBUG:teuthology.orchestra.run.smithi122:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid dc326fc2-fa65-11ee-bc8e-c7b262605968 -- ceph orch apply prometheus '1;smithi122=a' 2024-04-14T13:55:15.666 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:55:15 smithi122 ceph-mon[25721]: pgmap v169: 1 pgs: 1 remapped+peering; 577 KiB data, 47 MiB used, 715 GiB / 715 GiB avail 2024-04-14T13:55:15.667 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:55:15 smithi122 ceph-mon[25721]: from='client.? 172.21.15.69:0/1706782135' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-14T13:55:15.817 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:55:15 smithi069 ceph-mon[29472]: pgmap v169: 1 pgs: 1 remapped+peering; 577 KiB data, 47 MiB used, 715 GiB / 715 GiB avail 2024-04-14T13:55:15.817 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:55:15 smithi069 ceph-mon[29472]: from='client.? 172.21.15.69:0/1706782135' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-14T13:55:15.818 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:55:15 smithi069 ceph-mon[24818]: pgmap v169: 1 pgs: 1 remapped+peering; 577 KiB data, 47 MiB used, 715 GiB / 715 GiB avail 2024-04-14T13:55:15.818 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:55:15 smithi069 ceph-mon[24818]: from='client.? 172.21.15.69:0/1706782135' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-14T13:55:17.412 INFO:teuthology.orchestra.run.smithi122.stdout:Scheduled prometheus update... 2024-04-14T13:55:17.734 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:55:17 smithi122 ceph-mon[25721]: pgmap v170: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail; 97 KiB/s, 0 objects/s recovering 2024-04-14T13:55:17.734 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:55:17 smithi122 ceph-mon[25721]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' 2024-04-14T13:55:17.734 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:55:17 smithi122 ceph-mon[25721]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-14T13:55:17.734 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:55:17 smithi122 ceph-mon[25721]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T13:55:17.734 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:55:17 smithi122 ceph-mon[25721]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-14T13:55:17.734 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:55:17 smithi122 ceph-mon[25721]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' 2024-04-14T13:55:17.734 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:55:17 smithi122 ceph-mon[25721]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' cmd=[{"prefix": "mgr module enable", "module": "prometheus"}]: dispatch 2024-04-14T13:55:17.817 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:55:17 smithi069 ceph-mon[24818]: pgmap v170: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail; 97 KiB/s, 0 objects/s recovering 2024-04-14T13:55:17.817 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:55:17 smithi069 ceph-mon[24818]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' 2024-04-14T13:55:17.817 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:55:17 smithi069 ceph-mon[24818]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-14T13:55:17.817 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:55:17 smithi069 ceph-mon[24818]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T13:55:17.818 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:55:17 smithi069 ceph-mon[24818]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-14T13:55:17.818 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:55:17 smithi069 ceph-mon[24818]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' 2024-04-14T13:55:17.818 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:55:17 smithi069 ceph-mon[24818]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' cmd=[{"prefix": "mgr module enable", "module": "prometheus"}]: dispatch 2024-04-14T13:55:17.818 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:55:17 smithi069 ceph-mon[29472]: pgmap v170: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail; 97 KiB/s, 0 objects/s recovering 2024-04-14T13:55:17.818 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:55:17 smithi069 ceph-mon[29472]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' 2024-04-14T13:55:17.818 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:55:17 smithi069 ceph-mon[29472]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-14T13:55:17.818 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:55:17 smithi069 ceph-mon[29472]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T13:55:17.818 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:55:17 smithi069 ceph-mon[29472]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-14T13:55:17.818 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:55:17 smithi069 ceph-mon[29472]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' 2024-04-14T13:55:17.818 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:55:17 smithi069 ceph-mon[29472]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' cmd=[{"prefix": "mgr module enable", "module": "prometheus"}]: dispatch 2024-04-14T13:55:18.067 DEBUG:teuthology.orchestra.run.smithi122:prometheus.a> sudo journalctl -f -n 0 -u ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968@prometheus.a.service 2024-04-14T13:55:18.069 INFO:tasks.cephadm:Adding node-exporter.a on smithi069 2024-04-14T13:55:18.069 INFO:tasks.cephadm:Adding node-exporter.b on smithi122 2024-04-14T13:55:18.070 DEBUG:teuthology.orchestra.run.smithi122:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid dc326fc2-fa65-11ee-bc8e-c7b262605968 -- ceph orch apply node-exporter '2;smithi069=a;smithi122=b' 2024-04-14T13:55:18.667 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:55:18 smithi122 ceph-mon[25721]: from='client.24305 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "prometheus", "placement": "1;smithi122=a", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T13:55:18.668 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:55:18 smithi122 ceph-mon[25721]: Saving service prometheus spec with placement smithi122=a;count:1 2024-04-14T13:55:18.668 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:55:18 smithi122 ceph-mon[25721]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' cmd='[{"prefix": "mgr module enable", "module": "prometheus"}]': finished 2024-04-14T13:55:18.668 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:55:18 smithi122 ceph-mon[25721]: mgrmap e16: y(active, since 4m), standbys: x 2024-04-14T13:55:18.668 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Apr 14 13:55:18 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-x[26913]: ignoring --setuser ceph since I am not root 2024-04-14T13:55:18.668 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Apr 14 13:55:18 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-x[26913]: ignoring --setgroup ceph since I am not root 2024-04-14T13:55:18.668 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Apr 14 13:55:18 smithi122 ceph-mgr[26917]: -- 172.21.15.122:0/3539959291 <== mon.2 v2:172.21.15.122:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x56097709e340 con 0x560977077000 2024-04-14T13:55:18.817 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 14 13:55:18 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-y[25054]: ignoring --setuser ceph since I am not root 2024-04-14T13:55:18.817 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 14 13:55:18 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-y[25054]: ignoring --setgroup ceph since I am not root 2024-04-14T13:55:18.817 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 14 13:55:18 smithi069 ceph-mgr[25058]: -- 172.21.15.69:0/179111737 <== mon.0 v2:172.21.15.69:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x560890d60340 con 0x560890d38800 2024-04-14T13:55:18.818 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:55:18 smithi069 ceph-mon[24818]: from='client.24305 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "prometheus", "placement": "1;smithi122=a", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T13:55:18.818 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:55:18 smithi069 ceph-mon[24818]: Saving service prometheus spec with placement smithi122=a;count:1 2024-04-14T13:55:18.818 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:55:18 smithi069 ceph-mon[24818]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' cmd='[{"prefix": "mgr module enable", "module": "prometheus"}]': finished 2024-04-14T13:55:18.818 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:55:18 smithi069 ceph-mon[24818]: mgrmap e16: y(active, since 4m), standbys: x 2024-04-14T13:55:18.818 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:55:18 smithi069 ceph-mon[29472]: from='client.24305 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "prometheus", "placement": "1;smithi122=a", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T13:55:18.818 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:55:18 smithi069 ceph-mon[29472]: Saving service prometheus spec with placement smithi122=a;count:1 2024-04-14T13:55:18.819 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:55:18 smithi069 ceph-mon[29472]: from='mgr.14148 172.21.15.69:0/4039342825' entity='mgr.y' cmd='[{"prefix": "mgr module enable", "module": "prometheus"}]': finished 2024-04-14T13:55:18.819 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:55:18 smithi069 ceph-mon[29472]: mgrmap e16: y(active, since 4m), standbys: x 2024-04-14T13:55:19.317 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 14 13:55:18 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-y[25054]: 2024-04-14T13:55:18.940+0000 7fa8dd1f8000 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-14T13:55:19.317 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 14 13:55:19 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-y[25054]: 2024-04-14T13:55:19.027+0000 7fa8dd1f8000 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-14T13:55:19.345 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Apr 14 13:55:18 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-x[26913]: 2024-04-14T13:55:18.945+0000 7fe62e46d000 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-14T13:55:19.345 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Apr 14 13:55:19 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-x[26913]: 2024-04-14T13:55:19.035+0000 7fe62e46d000 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-14T13:55:19.925 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 14 13:55:19 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-y[25054]: 2024-04-14T13:55:19.606+0000 7fa8dd1f8000 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-14T13:55:19.926 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 14 13:55:19 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-y[25054]: 2024-04-14T13:55:19.693+0000 7fa8dd1f8000 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-14T13:55:19.926 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 14 13:55:19 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-y[25054]: 2024-04-14T13:55:19.775+0000 7fa8dd1f8000 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-14T13:55:19.926 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 14 13:55:19 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-y[25054]: 2024-04-14T13:55:19.923+0000 7fa8dd1f8000 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-14T13:55:19.963 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Apr 14 13:55:19 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-x[26913]: 2024-04-14T13:55:19.628+0000 7fe62e46d000 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-14T13:55:19.963 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Apr 14 13:55:19 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-x[26913]: 2024-04-14T13:55:19.718+0000 7fe62e46d000 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-14T13:55:19.964 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Apr 14 13:55:19 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-x[26913]: 2024-04-14T13:55:19.805+0000 7fe62e46d000 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-14T13:55:19.964 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Apr 14 13:55:19 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-x[26913]: 2024-04-14T13:55:19.960+0000 7fe62e46d000 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-14T13:55:20.195 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 14 13:55:20 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-y[25054]: 2024-04-14T13:55:20.002+0000 7fa8dd1f8000 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-14T13:55:20.195 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 14 13:55:20 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-y[25054]: 2024-04-14T13:55:20.109+0000 7fa8dd1f8000 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-14T13:55:20.236 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Apr 14 13:55:20 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-x[26913]: 2024-04-14T13:55:20.042+0000 7fe62e46d000 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-14T13:55:20.236 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Apr 14 13:55:20 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-x[26913]: 2024-04-14T13:55:20.149+0000 7fe62e46d000 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-14T13:55:20.236 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Apr 14 13:55:20 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-x[26913]: 2024-04-14T13:55:20.233+0000 7fe62e46d000 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-14T13:55:20.567 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 14 13:55:20 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-y[25054]: 2024-04-14T13:55:20.192+0000 7fa8dd1f8000 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-14T13:55:20.567 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 14 13:55:20 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-y[25054]: 2024-04-14T13:55:20.355+0000 7fa8dd1f8000 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-14T13:55:20.567 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 14 13:55:20 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-y[25054]: 2024-04-14T13:55:20.439+0000 7fa8dd1f8000 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-14T13:55:20.595 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Apr 14 13:55:20 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-x[26913]: 2024-04-14T13:55:20.394+0000 7fe62e46d000 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-14T13:55:20.595 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Apr 14 13:55:20 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-x[26913]: 2024-04-14T13:55:20.477+0000 7fe62e46d000 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-14T13:55:21.771 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 14 13:55:21 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-y[25054]: 2024-04-14T13:55:21.448+0000 7fa8dd1f8000 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-14T13:55:21.771 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 14 13:55:21 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-y[25054]: 2024-04-14T13:55:21.577+0000 7fa8dd1f8000 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-14T13:55:21.802 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Apr 14 13:55:21 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-x[26913]: 2024-04-14T13:55:21.481+0000 7fe62e46d000 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-14T13:55:21.802 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Apr 14 13:55:21 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-x[26913]: 2024-04-14T13:55:21.608+0000 7fe62e46d000 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-14T13:55:22.067 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 14 13:55:21 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-y[25054]: 2024-04-14T13:55:21.769+0000 7fa8dd1f8000 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-14T13:55:22.067 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 14 13:55:21 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-y[25054]: 2024-04-14T13:55:21.856+0000 7fa8dd1f8000 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-14T13:55:22.067 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 14 13:55:21 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-y[25054]: 2024-04-14T13:55:21.990+0000 7fa8dd1f8000 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-14T13:55:22.095 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Apr 14 13:55:21 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-x[26913]: 2024-04-14T13:55:21.800+0000 7fe62e46d000 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-14T13:55:22.095 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Apr 14 13:55:21 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-x[26913]: 2024-04-14T13:55:21.886+0000 7fe62e46d000 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-14T13:55:22.095 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Apr 14 13:55:22 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-x[26913]: 2024-04-14T13:55:22.020+0000 7fe62e46d000 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-14T13:55:22.553 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 14 13:55:22 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-y[25054]: 2024-04-14T13:55:22.331+0000 7fa8dd1f8000 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-14T13:55:22.595 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Apr 14 13:55:22 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-x[26913]: 2024-04-14T13:55:22.337+0000 7fe62e46d000 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-14T13:55:22.595 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Apr 14 13:55:22 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-x[26913]: 2024-04-14T13:55:22.557+0000 7fe62e46d000 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-14T13:55:22.806 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 14 13:55:22 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-y[25054]: 2024-04-14T13:55:22.550+0000 7fa8dd1f8000 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-14T13:55:22.806 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 14 13:55:22 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-y[25054]: 2024-04-14T13:55:22.635+0000 7fa8dd1f8000 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-14T13:55:22.806 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 14 13:55:22 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-y[25054]: 2024-04-14T13:55:22.804+0000 7fa8dd1f8000 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-14T13:55:23.095 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Apr 14 13:55:22 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-x[26913]: 2024-04-14T13:55:22.641+0000 7fe62e46d000 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-14T13:55:23.095 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Apr 14 13:55:22 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-x[26913]: 2024-04-14T13:55:22.812+0000 7fe62e46d000 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-14T13:55:24.011 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 14 13:55:23 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-y[25054]: 2024-04-14T13:55:23.715+0000 7fa8dd1f8000 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-14T13:55:24.012 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Apr 14 13:55:23 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-x[26913]: 2024-04-14T13:55:23.717+0000 7fe62e46d000 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-14T13:55:24.317 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 14 13:55:24 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-y[25054]: 2024-04-14T13:55:24.008+0000 7fa8dd1f8000 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-14T13:55:24.317 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 14 13:55:24 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-y[25054]: 2024-04-14T13:55:24.105+0000 7fa8dd1f8000 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-14T13:55:24.317 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 14 13:55:24 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-y[25054]: 2024-04-14T13:55:24.254+0000 7fa8dd1f8000 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-14T13:55:24.318 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Apr 14 13:55:24 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-x[26913]: 2024-04-14T13:55:24.010+0000 7fe62e46d000 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-14T13:55:24.318 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Apr 14 13:55:24 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-x[26913]: 2024-04-14T13:55:24.103+0000 7fe62e46d000 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-14T13:55:24.318 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Apr 14 13:55:24 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-x[26913]: 2024-04-14T13:55:24.253+0000 7fe62e46d000 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-14T13:55:24.318 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Apr 14 13:55:24 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-x[26913]: [14/Apr/2024:13:55:24] ENGINE Bus STARTING 2024-04-14T13:55:24.595 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Apr 14 13:55:24 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-x[26913]: CherryPy Checker: 2024-04-14T13:55:24.596 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Apr 14 13:55:24 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-x[26913]: The Application mounted at '' has an empty config. 2024-04-14T13:55:24.596 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Apr 14 13:55:24 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-x[26913]: 2024-04-14T13:55:24.596 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Apr 14 13:55:24 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-x[26913]: [14/Apr/2024:13:55:24] ENGINE Serving on http://:::9283 2024-04-14T13:55:24.596 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Apr 14 13:55:24 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-x[26913]: [14/Apr/2024:13:55:24] ENGINE Bus STARTED 2024-04-14T13:55:24.596 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:55:24 smithi122 ceph-mon[25721]: Standby manager daemon x restarted 2024-04-14T13:55:24.597 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:55:24 smithi122 ceph-mon[25721]: Standby manager daemon x started 2024-04-14T13:55:24.597 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:55:24 smithi122 ceph-mon[25721]: Active manager daemon y restarted 2024-04-14T13:55:24.597 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:55:24 smithi122 ceph-mon[25721]: Activating manager daemon y 2024-04-14T13:55:24.597 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:55:24 smithi122 ceph-mon[25721]: osdmap e59: 8 total, 8 up, 8 in 2024-04-14T13:55:24.597 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:55:24 smithi122 ceph-mon[25721]: from='mgr.? 172.21.15.122:0/2886473869' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-04-14T13:55:24.597 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:55:24 smithi122 ceph-mon[25721]: from='mgr.? 172.21.15.122:0/2886473869' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-14T13:55:24.597 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:55:24 smithi122 ceph-mon[25721]: from='mgr.? 172.21.15.122:0/2886473869' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-04-14T13:55:24.598 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:55:24 smithi122 ceph-mon[25721]: from='mgr.? 172.21.15.122:0/2886473869' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-14T13:55:24.633 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 14 13:55:24 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-y[25054]: [14/Apr/2024:13:55:24] ENGINE Bus STARTING 2024-04-14T13:55:24.633 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:55:24 smithi069 ceph-mon[24818]: Standby manager daemon x restarted 2024-04-14T13:55:24.633 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:55:24 smithi069 ceph-mon[24818]: Standby manager daemon x started 2024-04-14T13:55:24.633 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:55:24 smithi069 ceph-mon[24818]: Active manager daemon y restarted 2024-04-14T13:55:24.633 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:55:24 smithi069 ceph-mon[24818]: Activating manager daemon y 2024-04-14T13:55:24.633 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:55:24 smithi069 ceph-mon[24818]: osdmap e59: 8 total, 8 up, 8 in 2024-04-14T13:55:24.633 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:55:24 smithi069 ceph-mon[24818]: from='mgr.? 172.21.15.122:0/2886473869' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-04-14T13:55:24.634 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:55:24 smithi069 ceph-mon[24818]: from='mgr.? 172.21.15.122:0/2886473869' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-14T13:55:24.634 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:55:24 smithi069 ceph-mon[24818]: from='mgr.? 172.21.15.122:0/2886473869' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-04-14T13:55:24.634 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:55:24 smithi069 ceph-mon[24818]: from='mgr.? 172.21.15.122:0/2886473869' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-14T13:55:24.634 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:55:24 smithi069 ceph-mon[29472]: Standby manager daemon x restarted 2024-04-14T13:55:24.634 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:55:24 smithi069 ceph-mon[29472]: Standby manager daemon x started 2024-04-14T13:55:24.634 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:55:24 smithi069 ceph-mon[29472]: Active manager daemon y restarted 2024-04-14T13:55:24.634 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:55:24 smithi069 ceph-mon[29472]: Activating manager daemon y 2024-04-14T13:55:24.634 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:55:24 smithi069 ceph-mon[29472]: osdmap e59: 8 total, 8 up, 8 in 2024-04-14T13:55:24.634 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:55:24 smithi069 ceph-mon[29472]: from='mgr.? 172.21.15.122:0/2886473869' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-04-14T13:55:24.634 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:55:24 smithi069 ceph-mon[29472]: from='mgr.? 172.21.15.122:0/2886473869' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-14T13:55:24.635 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:55:24 smithi069 ceph-mon[29472]: from='mgr.? 172.21.15.122:0/2886473869' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-04-14T13:55:24.635 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:55:24 smithi069 ceph-mon[29472]: from='mgr.? 172.21.15.122:0/2886473869' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-14T13:55:25.002 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 14 13:55:24 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-y[25054]: CherryPy Checker: 2024-04-14T13:55:25.003 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 14 13:55:24 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-y[25054]: The Application mounted at '' has an empty config. 2024-04-14T13:55:25.003 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 14 13:55:24 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-y[25054]: 2024-04-14T13:55:25.003 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 14 13:55:24 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-y[25054]: [14/Apr/2024:13:55:24] ENGINE Serving on http://:::9283 2024-04-14T13:55:25.003 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 14 13:55:24 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-y[25054]: [14/Apr/2024:13:55:24] ENGINE Bus STARTED 2024-04-14T13:55:25.317 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 14 13:55:25 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-y[25054]: [14/Apr/2024:13:55:25] ENGINE Bus STARTING 2024-04-14T13:55:25.317 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 14 13:55:25 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-y[25054]: [14/Apr/2024:13:55:25] ENGINE Serving on https://172.21.15.69:7150 2024-04-14T13:55:25.317 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 14 13:55:25 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-y[25054]: [14/Apr/2024:13:55:25] ENGINE Bus STARTED 2024-04-14T13:55:25.344 INFO:teuthology.orchestra.run.smithi122.stdout:Scheduled node-exporter update... 2024-04-14T13:55:25.595 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:55:25 smithi122 ceph-mon[25721]: mgrmap e17: y(active, starting, since 0.0577311s), standbys: x 2024-04-14T13:55:25.595 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:55:25 smithi122 ceph-mon[25721]: from='mgr.14448 172.21.15.69:0/453473650' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-14T13:55:25.595 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:55:25 smithi122 ceph-mon[25721]: from='mgr.14448 172.21.15.69:0/453473650' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-14T13:55:25.596 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:55:25 smithi122 ceph-mon[25721]: from='mgr.14448 172.21.15.69:0/453473650' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-14T13:55:25.596 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:55:25 smithi122 ceph-mon[25721]: from='mgr.14448 172.21.15.69:0/453473650' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-04-14T13:55:25.596 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:55:25 smithi122 ceph-mon[25721]: from='mgr.14448 172.21.15.69:0/453473650' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-04-14T13:55:25.596 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:55:25 smithi122 ceph-mon[25721]: from='mgr.14448 172.21.15.69:0/453473650' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-14T13:55:25.596 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:55:25 smithi122 ceph-mon[25721]: from='mgr.14448 172.21.15.69:0/453473650' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-14T13:55:25.596 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:55:25 smithi122 ceph-mon[25721]: from='mgr.14448 172.21.15.69:0/453473650' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-14T13:55:25.596 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:55:25 smithi122 ceph-mon[25721]: from='mgr.14448 172.21.15.69:0/453473650' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-14T13:55:25.596 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:55:25 smithi122 ceph-mon[25721]: from='mgr.14448 172.21.15.69:0/453473650' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-14T13:55:25.596 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:55:25 smithi122 ceph-mon[25721]: from='mgr.14448 172.21.15.69:0/453473650' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-14T13:55:25.596 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:55:25 smithi122 ceph-mon[25721]: from='mgr.14448 172.21.15.69:0/453473650' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-14T13:55:25.596 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:55:25 smithi122 ceph-mon[25721]: from='mgr.14448 172.21.15.69:0/453473650' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-14T13:55:25.596 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:55:25 smithi122 ceph-mon[25721]: from='mgr.14448 172.21.15.69:0/453473650' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-04-14T13:55:25.596 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:55:25 smithi122 ceph-mon[25721]: from='mgr.14448 172.21.15.69:0/453473650' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-04-14T13:55:25.596 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:55:25 smithi122 ceph-mon[25721]: from='mgr.14448 172.21.15.69:0/453473650' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-04-14T13:55:25.596 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:55:25 smithi122 ceph-mon[25721]: Manager daemon y is now available 2024-04-14T13:55:25.597 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:55:25 smithi122 ceph-mon[25721]: from='mgr.14448 172.21.15.69:0/453473650' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-14T13:55:25.597 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:55:25 smithi122 ceph-mon[25721]: from='mgr.14448 172.21.15.69:0/453473650' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-14T13:55:25.597 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:55:25 smithi122 ceph-mon[25721]: from='mgr.14448 172.21.15.69:0/453473650' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T13:55:25.597 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:55:25 smithi122 ceph-mon[25721]: from='mgr.14448 172.21.15.69:0/453473650' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-14T13:55:25.597 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:55:25 smithi122 ceph-mon[25721]: from='mgr.14448 172.21.15.69:0/453473650' entity='mgr.y' 2024-04-14T13:55:25.597 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:55:25 smithi122 ceph-mon[25721]: from='mgr.14448 172.21.15.69:0/453473650' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-04-14T13:55:25.597 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:55:25 smithi122 ceph-mon[25721]: from='mgr.14448 172.21.15.69:0/453473650' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-04-14T13:55:25.597 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:55:25 smithi122 ceph-mon[25721]: from='mgr.14448 172.21.15.69:0/453473650' entity='mgr.y' 2024-04-14T13:55:25.817 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:55:25 smithi069 ceph-mon[24818]: mgrmap e17: y(active, starting, since 0.0577311s), standbys: x 2024-04-14T13:55:25.817 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:55:25 smithi069 ceph-mon[24818]: from='mgr.14448 172.21.15.69:0/453473650' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-14T13:55:25.817 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:55:25 smithi069 ceph-mon[24818]: from='mgr.14448 172.21.15.69:0/453473650' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-14T13:55:25.818 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:55:25 smithi069 ceph-mon[24818]: from='mgr.14448 172.21.15.69:0/453473650' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-14T13:55:25.818 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:55:25 smithi069 ceph-mon[24818]: from='mgr.14448 172.21.15.69:0/453473650' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-04-14T13:55:25.818 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:55:25 smithi069 ceph-mon[24818]: from='mgr.14448 172.21.15.69:0/453473650' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-04-14T13:55:25.818 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:55:25 smithi069 ceph-mon[24818]: from='mgr.14448 172.21.15.69:0/453473650' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-14T13:55:25.818 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:55:25 smithi069 ceph-mon[24818]: from='mgr.14448 172.21.15.69:0/453473650' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-14T13:55:25.818 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:55:25 smithi069 ceph-mon[24818]: from='mgr.14448 172.21.15.69:0/453473650' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-14T13:55:25.818 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:55:25 smithi069 ceph-mon[24818]: from='mgr.14448 172.21.15.69:0/453473650' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-14T13:55:25.818 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:55:25 smithi069 ceph-mon[24818]: from='mgr.14448 172.21.15.69:0/453473650' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-14T13:55:25.818 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:55:25 smithi069 ceph-mon[24818]: from='mgr.14448 172.21.15.69:0/453473650' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-14T13:55:25.818 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:55:25 smithi069 ceph-mon[24818]: from='mgr.14448 172.21.15.69:0/453473650' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-14T13:55:25.818 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:55:25 smithi069 ceph-mon[24818]: from='mgr.14448 172.21.15.69:0/453473650' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-14T13:55:25.818 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:55:25 smithi069 ceph-mon[24818]: from='mgr.14448 172.21.15.69:0/453473650' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-04-14T13:55:25.818 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:55:25 smithi069 ceph-mon[24818]: from='mgr.14448 172.21.15.69:0/453473650' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-04-14T13:55:25.818 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:55:25 smithi069 ceph-mon[24818]: from='mgr.14448 172.21.15.69:0/453473650' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-04-14T13:55:25.818 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:55:25 smithi069 ceph-mon[24818]: Manager daemon y is now available 2024-04-14T13:55:25.819 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:55:25 smithi069 ceph-mon[24818]: from='mgr.14448 172.21.15.69:0/453473650' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-14T13:55:25.819 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:55:25 smithi069 ceph-mon[24818]: from='mgr.14448 172.21.15.69:0/453473650' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-14T13:55:25.819 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:55:25 smithi069 ceph-mon[24818]: from='mgr.14448 172.21.15.69:0/453473650' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T13:55:25.819 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:55:25 smithi069 ceph-mon[24818]: from='mgr.14448 172.21.15.69:0/453473650' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-14T13:55:25.819 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:55:25 smithi069 ceph-mon[24818]: from='mgr.14448 172.21.15.69:0/453473650' entity='mgr.y' 2024-04-14T13:55:25.819 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:55:25 smithi069 ceph-mon[24818]: from='mgr.14448 172.21.15.69:0/453473650' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-04-14T13:55:25.819 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:55:25 smithi069 ceph-mon[24818]: from='mgr.14448 172.21.15.69:0/453473650' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-04-14T13:55:25.819 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:55:25 smithi069 ceph-mon[24818]: from='mgr.14448 172.21.15.69:0/453473650' entity='mgr.y' 2024-04-14T13:55:25.819 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:55:25 smithi069 ceph-mon[29472]: mgrmap e17: y(active, starting, since 0.0577311s), standbys: x 2024-04-14T13:55:25.819 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:55:25 smithi069 ceph-mon[29472]: from='mgr.14448 172.21.15.69:0/453473650' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-14T13:55:25.819 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:55:25 smithi069 ceph-mon[29472]: from='mgr.14448 172.21.15.69:0/453473650' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-14T13:55:25.819 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:55:25 smithi069 ceph-mon[29472]: from='mgr.14448 172.21.15.69:0/453473650' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-14T13:55:25.820 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:55:25 smithi069 ceph-mon[29472]: from='mgr.14448 172.21.15.69:0/453473650' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-04-14T13:55:25.820 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:55:25 smithi069 ceph-mon[29472]: from='mgr.14448 172.21.15.69:0/453473650' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-04-14T13:55:25.820 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:55:25 smithi069 ceph-mon[29472]: from='mgr.14448 172.21.15.69:0/453473650' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-14T13:55:25.820 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:55:25 smithi069 ceph-mon[29472]: from='mgr.14448 172.21.15.69:0/453473650' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-14T13:55:25.820 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:55:25 smithi069 ceph-mon[29472]: from='mgr.14448 172.21.15.69:0/453473650' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-14T13:55:25.820 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:55:25 smithi069 ceph-mon[29472]: from='mgr.14448 172.21.15.69:0/453473650' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-14T13:55:25.820 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:55:25 smithi069 ceph-mon[29472]: from='mgr.14448 172.21.15.69:0/453473650' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-14T13:55:25.820 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:55:25 smithi069 ceph-mon[29472]: from='mgr.14448 172.21.15.69:0/453473650' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-14T13:55:25.820 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:55:25 smithi069 ceph-mon[29472]: from='mgr.14448 172.21.15.69:0/453473650' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-14T13:55:25.820 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:55:25 smithi069 ceph-mon[29472]: from='mgr.14448 172.21.15.69:0/453473650' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-14T13:55:25.820 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:55:25 smithi069 ceph-mon[29472]: from='mgr.14448 172.21.15.69:0/453473650' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-04-14T13:55:25.820 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:55:25 smithi069 ceph-mon[29472]: from='mgr.14448 172.21.15.69:0/453473650' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-04-14T13:55:25.820 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:55:25 smithi069 ceph-mon[29472]: from='mgr.14448 172.21.15.69:0/453473650' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-04-14T13:55:25.820 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:55:25 smithi069 ceph-mon[29472]: Manager daemon y is now available 2024-04-14T13:55:25.820 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:55:25 smithi069 ceph-mon[29472]: from='mgr.14448 172.21.15.69:0/453473650' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-14T13:55:25.821 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:55:25 smithi069 ceph-mon[29472]: from='mgr.14448 172.21.15.69:0/453473650' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-14T13:55:25.821 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:55:25 smithi069 ceph-mon[29472]: from='mgr.14448 172.21.15.69:0/453473650' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T13:55:25.821 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:55:25 smithi069 ceph-mon[29472]: from='mgr.14448 172.21.15.69:0/453473650' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-14T13:55:25.821 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:55:25 smithi069 ceph-mon[29472]: from='mgr.14448 172.21.15.69:0/453473650' entity='mgr.y' 2024-04-14T13:55:25.821 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:55:25 smithi069 ceph-mon[29472]: from='mgr.14448 172.21.15.69:0/453473650' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-04-14T13:55:25.821 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:55:25 smithi069 ceph-mon[29472]: from='mgr.14448 172.21.15.69:0/453473650' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-04-14T13:55:25.821 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:55:25 smithi069 ceph-mon[29472]: from='mgr.14448 172.21.15.69:0/453473650' entity='mgr.y' 2024-04-14T13:55:26.382 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:55:26 smithi122 ceph-mon[25721]: [14/Apr/2024:13:55:25] ENGINE Bus STARTING 2024-04-14T13:55:26.382 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:55:26 smithi122 ceph-mon[25721]: [14/Apr/2024:13:55:25] ENGINE Serving on https://172.21.15.69:7150 2024-04-14T13:55:26.382 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:55:26 smithi122 ceph-mon[25721]: [14/Apr/2024:13:55:25] ENGINE Bus STARTED 2024-04-14T13:55:26.382 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:55:26 smithi122 ceph-mon[25721]: mgrmap e18: y(active, since 1.07175s), standbys: x 2024-04-14T13:55:26.382 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:55:26 smithi122 ceph-mon[25721]: from='client.24323 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "node-exporter", "placement": "2;smithi069=a;smithi122=b", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T13:55:26.382 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:55:26 smithi122 ceph-mon[25721]: pgmap v3: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-14T13:55:26.382 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:55:26 smithi122 ceph-mon[25721]: Saving service node-exporter spec with placement smithi069=a;smithi122=b;count:2 2024-04-14T13:55:26.382 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:55:26 smithi122 ceph-mon[25721]: from='mgr.14448 172.21.15.69:0/453473650' entity='mgr.y' 2024-04-14T13:55:26.383 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:55:26 smithi122 ceph-mon[25721]: from='mgr.14448 172.21.15.69:0/453473650' entity='mgr.y' 2024-04-14T13:55:26.399 DEBUG:teuthology.orchestra.run.smithi069:node-exporter.a> sudo journalctl -f -n 0 -u ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968@node-exporter.a.service 2024-04-14T13:55:26.402 DEBUG:teuthology.orchestra.run.smithi122:node-exporter.b> sudo journalctl -f -n 0 -u ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968@node-exporter.b.service 2024-04-14T13:55:26.404 INFO:tasks.cephadm:Adding alertmanager.a on smithi069 2024-04-14T13:55:26.404 DEBUG:teuthology.orchestra.run.smithi122:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid dc326fc2-fa65-11ee-bc8e-c7b262605968 -- ceph orch apply alertmanager '1;smithi069=a' 2024-04-14T13:55:26.562 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:55:26 smithi069 ceph-mon[24818]: [14/Apr/2024:13:55:25] ENGINE Bus STARTING 2024-04-14T13:55:26.563 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:55:26 smithi069 ceph-mon[24818]: [14/Apr/2024:13:55:25] ENGINE Serving on https://172.21.15.69:7150 2024-04-14T13:55:26.563 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:55:26 smithi069 ceph-mon[24818]: [14/Apr/2024:13:55:25] ENGINE Bus STARTED 2024-04-14T13:55:26.563 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:55:26 smithi069 ceph-mon[24818]: mgrmap e18: y(active, since 1.07175s), standbys: x 2024-04-14T13:55:26.563 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:55:26 smithi069 ceph-mon[24818]: from='client.24323 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "node-exporter", "placement": "2;smithi069=a;smithi122=b", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T13:55:26.563 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:55:26 smithi069 ceph-mon[24818]: pgmap v3: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-14T13:55:26.563 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:55:26 smithi069 ceph-mon[24818]: Saving service node-exporter spec with placement smithi069=a;smithi122=b;count:2 2024-04-14T13:55:26.563 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:55:26 smithi069 ceph-mon[24818]: from='mgr.14448 172.21.15.69:0/453473650' entity='mgr.y' 2024-04-14T13:55:26.564 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:55:26 smithi069 ceph-mon[24818]: from='mgr.14448 172.21.15.69:0/453473650' entity='mgr.y' 2024-04-14T13:55:26.564 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:55:26 smithi069 ceph-mon[29472]: [14/Apr/2024:13:55:25] ENGINE Bus STARTING 2024-04-14T13:55:26.564 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:55:26 smithi069 ceph-mon[29472]: [14/Apr/2024:13:55:25] ENGINE Serving on https://172.21.15.69:7150 2024-04-14T13:55:26.564 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:55:26 smithi069 ceph-mon[29472]: [14/Apr/2024:13:55:25] ENGINE Bus STARTED 2024-04-14T13:55:26.564 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:55:26 smithi069 ceph-mon[29472]: mgrmap e18: y(active, since 1.07175s), standbys: x 2024-04-14T13:55:26.564 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:55:26 smithi069 ceph-mon[29472]: from='client.24323 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "node-exporter", "placement": "2;smithi069=a;smithi122=b", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T13:55:26.564 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:55:26 smithi069 ceph-mon[29472]: pgmap v3: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-14T13:55:26.564 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:55:26 smithi069 ceph-mon[29472]: Saving service node-exporter spec with placement smithi069=a;smithi122=b;count:2 2024-04-14T13:55:26.564 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:55:26 smithi069 ceph-mon[29472]: from='mgr.14448 172.21.15.69:0/453473650' entity='mgr.y' 2024-04-14T13:55:26.564 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:55:26 smithi069 ceph-mon[29472]: from='mgr.14448 172.21.15.69:0/453473650' entity='mgr.y' 2024-04-14T13:55:27.595 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:55:27 smithi122 ceph-mon[25721]: pgmap v4: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-14T13:55:27.595 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:55:27 smithi122 ceph-mon[25721]: from='mgr.14448 172.21.15.69:0/453473650' entity='mgr.y' 2024-04-14T13:55:27.595 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:55:27 smithi122 ceph-mon[25721]: from='mgr.14448 172.21.15.69:0/453473650' entity='mgr.y' 2024-04-14T13:55:27.595 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:55:27 smithi122 ceph-mon[25721]: mgrmap e19: y(active, since 2s), standbys: x 2024-04-14T13:55:27.595 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:55:27 smithi122 ceph-mon[25721]: from='mgr.14448 172.21.15.69:0/453473650' entity='mgr.y' 2024-04-14T13:55:27.817 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:55:27 smithi069 ceph-mon[24818]: pgmap v4: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-14T13:55:27.817 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:55:27 smithi069 ceph-mon[24818]: from='mgr.14448 172.21.15.69:0/453473650' entity='mgr.y' 2024-04-14T13:55:27.817 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:55:27 smithi069 ceph-mon[24818]: from='mgr.14448 172.21.15.69:0/453473650' entity='mgr.y' 2024-04-14T13:55:27.817 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:55:27 smithi069 ceph-mon[24818]: mgrmap e19: y(active, since 2s), standbys: x 2024-04-14T13:55:27.817 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:55:27 smithi069 ceph-mon[24818]: from='mgr.14448 172.21.15.69:0/453473650' entity='mgr.y' 2024-04-14T13:55:27.817 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:55:27 smithi069 ceph-mon[29472]: pgmap v4: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-14T13:55:27.818 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:55:27 smithi069 ceph-mon[29472]: from='mgr.14448 172.21.15.69:0/453473650' entity='mgr.y' 2024-04-14T13:55:27.818 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:55:27 smithi069 ceph-mon[29472]: from='mgr.14448 172.21.15.69:0/453473650' entity='mgr.y' 2024-04-14T13:55:27.818 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:55:27 smithi069 ceph-mon[29472]: mgrmap e19: y(active, since 2s), standbys: x 2024-04-14T13:55:27.818 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:55:27 smithi069 ceph-mon[29472]: from='mgr.14448 172.21.15.69:0/453473650' entity='mgr.y' 2024-04-14T13:55:28.576 INFO:teuthology.orchestra.run.smithi122.stdout:Scheduled alertmanager update... 2024-04-14T13:55:28.817 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:55:28 smithi069 ceph-mon[24818]: Updating smithi069:/etc/ceph/ceph.conf 2024-04-14T13:55:28.817 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:55:28 smithi069 ceph-mon[24818]: Updating smithi069:/etc/ceph/ceph.client.admin.keyring 2024-04-14T13:55:28.817 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:55:28 smithi069 ceph-mon[24818]: from='mgr.14448 172.21.15.69:0/453473650' entity='mgr.y' 2024-04-14T13:55:28.817 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:55:28 smithi069 ceph-mon[24818]: Updating smithi122:/etc/ceph/ceph.conf 2024-04-14T13:55:28.818 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:55:28 smithi069 ceph-mon[24818]: from='mgr.14448 172.21.15.69:0/453473650' entity='mgr.y' 2024-04-14T13:55:28.818 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:55:28 smithi069 ceph-mon[24818]: from='mgr.14448 172.21.15.69:0/453473650' entity='mgr.y' 2024-04-14T13:55:28.818 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:55:28 smithi069 ceph-mon[29472]: Updating smithi069:/etc/ceph/ceph.conf 2024-04-14T13:55:28.818 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:55:28 smithi069 ceph-mon[29472]: Updating smithi069:/etc/ceph/ceph.client.admin.keyring 2024-04-14T13:55:28.818 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:55:28 smithi069 ceph-mon[29472]: from='mgr.14448 172.21.15.69:0/453473650' entity='mgr.y' 2024-04-14T13:55:28.818 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:55:28 smithi069 ceph-mon[29472]: Updating smithi122:/etc/ceph/ceph.conf 2024-04-14T13:55:28.818 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:55:28 smithi069 ceph-mon[29472]: from='mgr.14448 172.21.15.69:0/453473650' entity='mgr.y' 2024-04-14T13:55:28.818 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:55:28 smithi069 ceph-mon[29472]: from='mgr.14448 172.21.15.69:0/453473650' entity='mgr.y' 2024-04-14T13:55:28.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:55:28 smithi122 ceph-mon[25721]: Updating smithi069:/etc/ceph/ceph.conf 2024-04-14T13:55:28.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:55:28 smithi122 ceph-mon[25721]: Updating smithi069:/etc/ceph/ceph.client.admin.keyring 2024-04-14T13:55:28.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:55:28 smithi122 ceph-mon[25721]: from='mgr.14448 172.21.15.69:0/453473650' entity='mgr.y' 2024-04-14T13:55:28.846 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:55:28 smithi122 ceph-mon[25721]: Updating smithi122:/etc/ceph/ceph.conf 2024-04-14T13:55:28.846 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:55:28 smithi122 ceph-mon[25721]: from='mgr.14448 172.21.15.69:0/453473650' entity='mgr.y' 2024-04-14T13:55:28.846 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:55:28 smithi122 ceph-mon[25721]: from='mgr.14448 172.21.15.69:0/453473650' entity='mgr.y' 2024-04-14T13:55:29.243 DEBUG:teuthology.orchestra.run.smithi069:alertmanager.a> sudo journalctl -f -n 0 -u ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968@alertmanager.a.service 2024-04-14T13:55:29.246 INFO:tasks.cephadm:Adding grafana.a on smithi122 2024-04-14T13:55:29.246 DEBUG:teuthology.orchestra.run.smithi122:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid dc326fc2-fa65-11ee-bc8e-c7b262605968 -- ceph orch apply grafana '1;smithi122=a' 2024-04-14T13:55:29.555 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:55:29 smithi122 ceph-mon[25721]: Updating smithi122:/etc/ceph/ceph.client.admin.keyring 2024-04-14T13:55:29.555 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:55:29 smithi122 ceph-mon[25721]: Deploying daemon prometheus.a on smithi122 2024-04-14T13:55:29.555 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:55:29 smithi122 ceph-mon[25721]: pgmap v5: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-14T13:55:29.555 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:55:29 smithi122 ceph-mon[25721]: from='mgr.14448 172.21.15.69:0/453473650' entity='mgr.y' 2024-04-14T13:55:29.567 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:55:29 smithi069 ceph-mon[24818]: Updating smithi122:/etc/ceph/ceph.client.admin.keyring 2024-04-14T13:55:29.567 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:55:29 smithi069 ceph-mon[24818]: Deploying daemon prometheus.a on smithi122 2024-04-14T13:55:29.567 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:55:29 smithi069 ceph-mon[24818]: pgmap v5: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-14T13:55:29.567 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:55:29 smithi069 ceph-mon[24818]: from='mgr.14448 172.21.15.69:0/453473650' entity='mgr.y' 2024-04-14T13:55:29.568 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:55:29 smithi069 ceph-mon[29472]: Updating smithi122:/etc/ceph/ceph.client.admin.keyring 2024-04-14T13:55:29.568 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:55:29 smithi069 ceph-mon[29472]: Deploying daemon prometheus.a on smithi122 2024-04-14T13:55:29.568 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:55:29 smithi069 ceph-mon[29472]: pgmap v5: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-14T13:55:29.568 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:55:29 smithi069 ceph-mon[29472]: from='mgr.14448 172.21.15.69:0/453473650' entity='mgr.y' 2024-04-14T13:55:30.595 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:55:30 smithi122 ceph-mon[25721]: from='client.24347 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "alertmanager", "placement": "1;smithi069=a", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T13:55:30.595 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:55:30 smithi122 ceph-mon[25721]: Saving service alertmanager spec with placement smithi069=a;count:1 2024-04-14T13:55:30.817 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:55:30 smithi069 ceph-mon[24818]: from='client.24347 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "alertmanager", "placement": "1;smithi069=a", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T13:55:30.817 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:55:30 smithi069 ceph-mon[24818]: Saving service alertmanager spec with placement smithi069=a;count:1 2024-04-14T13:55:30.817 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:55:30 smithi069 ceph-mon[29472]: from='client.24347 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "alertmanager", "placement": "1;smithi069=a", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T13:55:30.818 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:55:30 smithi069 ceph-mon[29472]: Saving service alertmanager spec with placement smithi069=a;count:1 2024-04-14T13:55:31.643 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:55:31 smithi122 ceph-mon[25721]: pgmap v6: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-14T13:55:31.817 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:55:31 smithi069 ceph-mon[24818]: pgmap v6: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-14T13:55:31.817 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:55:31 smithi069 ceph-mon[29472]: pgmap v6: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-14T13:55:31.939 INFO:teuthology.orchestra.run.smithi122.stdout:Scheduled grafana update... 2024-04-14T13:55:32.957 DEBUG:teuthology.orchestra.run.smithi122:grafana.a> sudo journalctl -f -n 0 -u ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968@grafana.a.service 2024-04-14T13:55:32.959 INFO:tasks.cephadm:Setting up client nodes... 2024-04-14T13:55:32.963 DEBUG:teuthology.orchestra.run.smithi069:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid dc326fc2-fa65-11ee-bc8e-c7b262605968 -- ceph auth get-or-create client.0 mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2024-04-14T13:55:33.272 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:55:32 smithi069 ceph-mon[24818]: from='client.24353 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "grafana", "placement": "1;smithi122=a", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T13:55:33.272 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:55:32 smithi069 ceph-mon[24818]: Saving service grafana spec with placement smithi122=a;count:1 2024-04-14T13:55:33.272 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:55:32 smithi069 ceph-mon[24818]: from='mgr.14448 172.21.15.69:0/453473650' entity='mgr.y' 2024-04-14T13:55:33.272 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:55:32 smithi069 ceph-mon[24818]: pgmap v7: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-14T13:55:33.273 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:55:32 smithi069 ceph-mon[29472]: from='client.24353 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "grafana", "placement": "1;smithi122=a", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T13:55:33.273 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:55:32 smithi069 ceph-mon[29472]: Saving service grafana spec with placement smithi122=a;count:1 2024-04-14T13:55:33.273 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:55:32 smithi069 ceph-mon[29472]: from='mgr.14448 172.21.15.69:0/453473650' entity='mgr.y' 2024-04-14T13:55:33.273 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:55:32 smithi069 ceph-mon[29472]: pgmap v7: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-14T13:55:33.345 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:55:32 smithi122 ceph-mon[25721]: from='client.24353 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "grafana", "placement": "1;smithi122=a", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T13:55:33.345 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:55:32 smithi122 ceph-mon[25721]: Saving service grafana spec with placement smithi122=a;count:1 2024-04-14T13:55:33.345 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:55:32 smithi122 ceph-mon[25721]: from='mgr.14448 172.21.15.69:0/453473650' entity='mgr.y' 2024-04-14T13:55:33.345 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:55:32 smithi122 ceph-mon[25721]: pgmap v7: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-14T13:55:34.883 INFO:teuthology.orchestra.run.smithi069.stdout:[client.0] 2024-04-14T13:55:34.884 INFO:teuthology.orchestra.run.smithi069.stdout: key = AQBW4BtmcZdBNBAASAD2NA3sv3kmwomznKrydw== 2024-04-14T13:55:35.585 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:55:35 smithi069 ceph-mon[29472]: pgmap v8: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-14T13:55:35.586 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:55:35 smithi069 ceph-mon[29472]: from='client.? 172.21.15.69:0/3195018175' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-04-14T13:55:35.586 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:55:35 smithi069 ceph-mon[29472]: from='client.? 172.21.15.69:0/3195018175' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-04-14T13:55:35.586 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:55:35 smithi069 ceph-mon[24818]: pgmap v8: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-14T13:55:35.586 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:55:35 smithi069 ceph-mon[24818]: from='client.? 172.21.15.69:0/3195018175' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-04-14T13:55:35.586 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:55:35 smithi069 ceph-mon[24818]: from='client.? 172.21.15.69:0/3195018175' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-04-14T13:55:35.610 DEBUG:teuthology.orchestra.run.smithi069:> set -ex 2024-04-14T13:55:35.610 DEBUG:teuthology.orchestra.run.smithi069:> sudo dd of=/etc/ceph/ceph.client.0.keyring 2024-04-14T13:55:35.610 DEBUG:teuthology.orchestra.run.smithi069:> sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-04-14T13:55:35.654 DEBUG:teuthology.orchestra.run.smithi122:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid dc326fc2-fa65-11ee-bc8e-c7b262605968 -- ceph auth get-or-create client.1 mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2024-04-14T13:55:35.682 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:55:35 smithi122 ceph-mon[25721]: pgmap v8: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-14T13:55:35.682 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:55:35 smithi122 ceph-mon[25721]: from='client.? 172.21.15.69:0/3195018175' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-04-14T13:55:35.682 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:55:35 smithi122 ceph-mon[25721]: from='client.? 172.21.15.69:0/3195018175' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-04-14T13:55:37.754 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:55:37 smithi122 ceph-mon[25721]: pgmap v9: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-14T13:55:37.817 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:55:37 smithi069 ceph-mon[24818]: pgmap v9: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-14T13:55:37.817 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:55:37 smithi069 ceph-mon[29472]: pgmap v9: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-14T13:55:38.903 INFO:teuthology.orchestra.run.smithi122.stdout:[client.1] 2024-04-14T13:55:38.904 INFO:teuthology.orchestra.run.smithi122.stdout: key = AQBa4Btmml+RNRAAudWmfifxPP1ALM7zChBkrA== 2024-04-14T13:55:39.443 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:55:39 smithi122 ceph-mon[25721]: pgmap v10: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-14T13:55:39.443 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:55:39 smithi122 ceph-mon[25721]: from='client.? 172.21.15.122:0/3817443020' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-04-14T13:55:39.443 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:55:39 smithi122 ceph-mon[25721]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-04-14T13:55:39.443 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:55:39 smithi122 ceph-mon[25721]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-04-14T13:55:39.741 DEBUG:teuthology.orchestra.run.smithi122:> set -ex 2024-04-14T13:55:39.742 DEBUG:teuthology.orchestra.run.smithi122:> sudo dd of=/etc/ceph/ceph.client.1.keyring 2024-04-14T13:55:39.742 DEBUG:teuthology.orchestra.run.smithi122:> sudo chmod 0644 /etc/ceph/ceph.client.1.keyring 2024-04-14T13:55:39.817 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:55:39 smithi069 ceph-mon[24818]: pgmap v10: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-14T13:55:39.817 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:55:39 smithi069 ceph-mon[24818]: from='client.? 172.21.15.122:0/3817443020' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-04-14T13:55:39.817 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:55:39 smithi069 ceph-mon[24818]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-04-14T13:55:39.817 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:55:39 smithi069 ceph-mon[24818]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-04-14T13:55:39.817 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:55:39 smithi069 ceph-mon[29472]: pgmap v10: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-14T13:55:39.818 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:55:39 smithi069 ceph-mon[29472]: from='client.? 172.21.15.122:0/3817443020' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-04-14T13:55:39.818 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:55:39 smithi069 ceph-mon[29472]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-04-14T13:55:39.818 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:55:39 smithi069 ceph-mon[29472]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-04-14T13:55:39.826 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2024-04-14T13:55:39.827 INFO:tasks.cephadm.ceph_manager.ceph:waiting for mgr available 2024-04-14T13:55:39.827 DEBUG:teuthology.orchestra.run.smithi069:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid dc326fc2-fa65-11ee-bc8e-c7b262605968 -- ceph mgr dump --format=json 2024-04-14T13:55:40.046 INFO:teuthology.orchestra.run.smithi069.stderr:Inferring config /var/lib/ceph/dc326fc2-fa65-11ee-bc8e-c7b262605968/mon.a/config 2024-04-14T13:55:41.270 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:55:41 smithi069 ceph-mon[29472]: pgmap v11: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-14T13:55:41.270 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:55:41 smithi069 ceph-mon[24818]: pgmap v11: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-14T13:55:41.595 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:55:41 smithi122 ceph-mon[25721]: pgmap v11: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-14T13:55:41.722 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-14T13:55:42.424 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:55:42 smithi069 ceph-mon[24818]: from='client.? 172.21.15.69:0/4060842397' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-04-14T13:55:42.424 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:55:42 smithi069 ceph-mon[29472]: from='client.? 172.21.15.69:0/4060842397' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-04-14T13:55:42.450 INFO:teuthology.orchestra.run.smithi069.stdout:{"epoch":19,"active_gid":14448,"active_name":"y","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6800","nonce":3122031884},{"type":"v1","addr":"172.21.15.69:6801","nonce":3122031884}]},"active_addr":"172.21.15.69:6801/3122031884","active_change":"2024-04-14T13:55:24.256729+0000","active_mgr_features":4540138303579357183,"available":true,"standbys":[{"gid":24311,"name":"x","mgr_features":4540138303579357183,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2400","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"7","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.23.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:8.3.5","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"docker.io/library/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"docker.io/arcts/keepalived","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.3.1","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.33.4","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"noautoscale":{"name":"noautoscale","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"global autoscale flag","long_desc":"Option to turn on/off the autoscaler for all pools","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"drive_group_interval":{"name":"drive_group_interval","type":"float","level":"advanced","flags":0,"default_value":"300.0","min":"","max":"","enum_allowed":[],"desc":"interval in seconds between re-application of applied drive_groups","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]}],"modules":["cephadm","dashboard","iostat","nfs","prometheus","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2400","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"7","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.23.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:8.3.5","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"docker.io/library/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"docker.io/arcts/keepalived","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.3.1","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.33.4","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"noautoscale":{"name":"noautoscale","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"global autoscale flag","long_desc":"Option to turn on/off the autoscaler for all pools","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"drive_group_interval":{"name":"drive_group_interval","type":"float","level":"advanced","flags":0,"default_value":"300.0","min":"","max":"","enum_allowed":[],"desc":"interval in seconds between re-application of applied drive_groups","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{"dashboard":"https://172.21.15.69:8443/","prometheus":"http://172.21.15.69:9283/"},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"last_failure_osd_epoch":59,"active_clients":[{"addrvec":[{"type":"v2","addr":"172.21.15.69:0","nonce":1711970283}]},{"addrvec":[{"type":"v2","addr":"172.21.15.69:0","nonce":1033838063}]},{"addrvec":[{"type":"v2","addr":"172.21.15.69:0","nonce":4037048393}]},{"addrvec":[{"type":"v2","addr":"172.21.15.69:0","nonce":3960312281}]}]}} 2024-04-14T13:55:42.454 INFO:tasks.cephadm.ceph_manager.ceph:mgr available! 2024-04-14T13:55:42.454 INFO:tasks.cephadm.ceph_manager.ceph:waiting for all up 2024-04-14T13:55:42.455 DEBUG:teuthology.orchestra.run.smithi069:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid dc326fc2-fa65-11ee-bc8e-c7b262605968 -- ceph osd dump --format=json 2024-04-14T13:55:42.583 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:55:42 smithi122 ceph-mon[25721]: from='client.? 172.21.15.69:0/4060842397' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-04-14T13:55:42.675 INFO:teuthology.orchestra.run.smithi069.stderr:Inferring config /var/lib/ceph/dc326fc2-fa65-11ee-bc8e-c7b262605968/mon.a/config 2024-04-14T13:55:43.317 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:55:43 smithi069 ceph-mon[24818]: pgmap v12: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-14T13:55:43.317 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:55:43 smithi069 ceph-mon[29472]: pgmap v12: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-14T13:55:43.534 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:55:43 smithi122 ceph-mon[25721]: pgmap v12: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-14T13:55:43.535 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Apr 14 13:55:43 smithi122 systemd[1]: Starting Ceph prometheus.a for dc326fc2-fa65-11ee-bc8e-c7b262605968... 2024-04-14T13:55:43.795 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Apr 14 13:55:43 smithi122 podman[40865]: 2024-04-14 13:55:43.436758071 +0000 UTC m=+0.017854669 image pull 514e6a882f6e74806a5856468489eeff8d7106095557578da96935e4d0ba4d9d quay.io/prometheus/prometheus:v2.33.4 2024-04-14T13:55:43.796 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Apr 14 13:55:43 smithi122 podman[40865]: 2024-04-14 13:55:43.543193735 +0000 UTC m=+0.124290322 container create 18895e3c9f451c7e9d8116de15c26d864460545ae2cc8defadfdcf28fdd1a894 (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-prometheus-a, maintainer=The Prometheus Authors ) 2024-04-14T13:55:43.796 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Apr 14 13:55:43 smithi122 podman[40865]: 2024-04-14 13:55:43.746199746 +0000 UTC m=+0.327296336 container init 18895e3c9f451c7e9d8116de15c26d864460545ae2cc8defadfdcf28fdd1a894 (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-prometheus-a, maintainer=The Prometheus Authors ) 2024-04-14T13:55:43.796 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Apr 14 13:55:43 smithi122 podman[40865]: 2024-04-14 13:55:43.749061409 +0000 UTC m=+0.330157999 container start 18895e3c9f451c7e9d8116de15c26d864460545ae2cc8defadfdcf28fdd1a894 (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-prometheus-a, maintainer=The Prometheus Authors ) 2024-04-14T13:55:43.796 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Apr 14 13:55:43 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-prometheus-a[40879]: ts=2024-04-14T13:55:43.772Z caller=main.go:475 level=info msg="No time or size retention was set so using the default time retention" duration=15d 2024-04-14T13:55:43.796 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Apr 14 13:55:43 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-prometheus-a[40879]: ts=2024-04-14T13:55:43.772Z caller=main.go:512 level=info msg="Starting Prometheus" version="(version=2.33.4, branch=HEAD, revision=83032011a5d3e6102624fe58241a374a7201fee8)" 2024-04-14T13:55:43.796 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Apr 14 13:55:43 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-prometheus-a[40879]: ts=2024-04-14T13:55:43.772Z caller=main.go:517 level=info build_context="(go=go1.17.7, user=root@d13bf69e7be8, date=20220222-16:51:28)" 2024-04-14T13:55:43.796 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Apr 14 13:55:43 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-prometheus-a[40879]: ts=2024-04-14T13:55:43.772Z caller=main.go:518 level=info host_details="(Linux 5.14.0-435.el9.x86_64 #1 SMP PREEMPT_DYNAMIC Fri Mar 29 18:46:37 UTC 2024 x86_64 smithi122 (none))" 2024-04-14T13:55:43.796 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Apr 14 13:55:43 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-prometheus-a[40879]: ts=2024-04-14T13:55:43.772Z caller=main.go:519 level=info fd_limits="(soft=1048576, hard=1048576)" 2024-04-14T13:55:43.796 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Apr 14 13:55:43 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-prometheus-a[40879]: ts=2024-04-14T13:55:43.772Z caller=main.go:520 level=info vm_limits="(soft=unlimited, hard=unlimited)" 2024-04-14T13:55:43.796 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Apr 14 13:55:43 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-prometheus-a[40879]: ts=2024-04-14T13:55:43.773Z caller=web.go:570 level=info component=web msg="Start listening for connections" address=:9095 2024-04-14T13:55:43.796 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Apr 14 13:55:43 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-prometheus-a[40879]: ts=2024-04-14T13:55:43.774Z caller=main.go:923 level=info msg="Starting TSDB ..." 2024-04-14T13:55:43.796 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Apr 14 13:55:43 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-prometheus-a[40879]: ts=2024-04-14T13:55:43.775Z caller=tls_config.go:195 level=info component=web msg="TLS is disabled." http2=false 2024-04-14T13:55:43.797 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Apr 14 13:55:43 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-prometheus-a[40879]: ts=2024-04-14T13:55:43.777Z caller=head.go:493 level=info component=tsdb msg="Replaying on-disk memory mappable chunks if any" 2024-04-14T13:55:43.797 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Apr 14 13:55:43 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-prometheus-a[40879]: ts=2024-04-14T13:55:43.777Z caller=head.go:527 level=info component=tsdb msg="On-disk memory mappable chunks replay completed" duration=1.592µs 2024-04-14T13:55:43.797 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Apr 14 13:55:43 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-prometheus-a[40879]: ts=2024-04-14T13:55:43.777Z caller=head.go:533 level=info component=tsdb msg="Replaying WAL, this may take a while" 2024-04-14T13:55:43.797 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Apr 14 13:55:43 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-prometheus-a[40879]: ts=2024-04-14T13:55:43.778Z caller=head.go:604 level=info component=tsdb msg="WAL segment loaded" segment=0 maxSegment=0 2024-04-14T13:55:43.797 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Apr 14 13:55:43 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-prometheus-a[40879]: ts=2024-04-14T13:55:43.778Z caller=head.go:610 level=info component=tsdb msg="WAL replay completed" checkpoint_replay_duration=20.902µs wal_replay_duration=175.099µs total_replay_duration=210.847µs 2024-04-14T13:55:43.797 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Apr 14 13:55:43 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-prometheus-a[40879]: ts=2024-04-14T13:55:43.778Z caller=main.go:944 level=info fs_type=XFS_SUPER_MAGIC 2024-04-14T13:55:43.797 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Apr 14 13:55:43 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-prometheus-a[40879]: ts=2024-04-14T13:55:43.778Z caller=main.go:947 level=info msg="TSDB started" 2024-04-14T13:55:43.797 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Apr 14 13:55:43 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-prometheus-a[40879]: ts=2024-04-14T13:55:43.778Z caller=main.go:1128 level=info msg="Loading configuration file" filename=/etc/prometheus/prometheus.yml 2024-04-14T13:55:43.797 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Apr 14 13:55:43 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-prometheus-a[40879]: ts=2024-04-14T13:55:43.793Z caller=main.go:1165 level=info msg="Completed loading of configuration file" filename=/etc/prometheus/prometheus.yml totalDuration=15.178534ms db_storage=1.029µs remote_storage=1.075µs web_handler=269ns query_engine=526ns scrape=391.953µs scrape_sd=16.291µs notify=1.38µs notify_sd=2.015µs rules=14.429557ms 2024-04-14T13:55:43.797 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Apr 14 13:55:43 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-prometheus-a[40879]: ts=2024-04-14T13:55:43.793Z caller=main.go:896 level=info msg="Server is ready to receive web requests." 2024-04-14T13:55:44.095 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Apr 14 13:55:43 smithi122 bash[40865]: 18895e3c9f451c7e9d8116de15c26d864460545ae2cc8defadfdcf28fdd1a894 2024-04-14T13:55:44.095 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Apr 14 13:55:43 smithi122 systemd[1]: Started Ceph prometheus.a for dc326fc2-fa65-11ee-bc8e-c7b262605968. 2024-04-14T13:55:44.416 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-14T13:55:44.417 INFO:teuthology.orchestra.run.smithi069.stdout:{"epoch":59,"fsid":"dc326fc2-fa65-11ee-bc8e-c7b262605968","created":"2024-04-14T13:50:21.024738+0000","modified":"2024-04-14T13:55:24.256417+0000","last_up_change":"2024-04-14T13:55:09.296070+0000","last_in_change":"2024-04-14T13:54:52.122409+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":18,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-14T13:53:15.745972+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}}],"osds":[{"osd":0,"uuid":"99aae7c5-9755-49e8-91c7-fceed3ff30fb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6802","nonce":2517323608},{"type":"v1","addr":"172.21.15.69:6803","nonce":2517323608}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6804","nonce":2517323608},{"type":"v1","addr":"172.21.15.69:6805","nonce":2517323608}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6808","nonce":2517323608},{"type":"v1","addr":"172.21.15.69:6809","nonce":2517323608}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6806","nonce":2517323608},{"type":"v1","addr":"172.21.15.69:6807","nonce":2517323608}]},"public_addr":"172.21.15.69:6803/2517323608","cluster_addr":"172.21.15.69:6805/2517323608","heartbeat_back_addr":"172.21.15.69:6809/2517323608","heartbeat_front_addr":"172.21.15.69:6807/2517323608","state":["exists","up"]},{"osd":1,"uuid":"1d99b840-344d-422a-a466-13ed2d956280","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6810","nonce":1377309913},{"type":"v1","addr":"172.21.15.69:6811","nonce":1377309913}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6812","nonce":1377309913},{"type":"v1","addr":"172.21.15.69:6813","nonce":1377309913}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6816","nonce":1377309913},{"type":"v1","addr":"172.21.15.69:6817","nonce":1377309913}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6814","nonce":1377309913},{"type":"v1","addr":"172.21.15.69:6815","nonce":1377309913}]},"public_addr":"172.21.15.69:6811/1377309913","cluster_addr":"172.21.15.69:6813/1377309913","heartbeat_back_addr":"172.21.15.69:6817/1377309913","heartbeat_front_addr":"172.21.15.69:6815/1377309913","state":["exists","up"]},{"osd":2,"uuid":"f4e0dcf9-5003-4a8f-8a51-fa36f58c0ac4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6818","nonce":2090584066},{"type":"v1","addr":"172.21.15.69:6819","nonce":2090584066}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6820","nonce":2090584066},{"type":"v1","addr":"172.21.15.69:6821","nonce":2090584066}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6824","nonce":2090584066},{"type":"v1","addr":"172.21.15.69:6825","nonce":2090584066}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6822","nonce":2090584066},{"type":"v1","addr":"172.21.15.69:6823","nonce":2090584066}]},"public_addr":"172.21.15.69:6819/2090584066","cluster_addr":"172.21.15.69:6821/2090584066","heartbeat_back_addr":"172.21.15.69:6825/2090584066","heartbeat_front_addr":"172.21.15.69:6823/2090584066","state":["exists","up"]},{"osd":3,"uuid":"27e543e2-f5c0-462d-ab23-d439e06c056e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6826","nonce":148592620},{"type":"v1","addr":"172.21.15.69:6827","nonce":148592620}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6828","nonce":148592620},{"type":"v1","addr":"172.21.15.69:6829","nonce":148592620}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6832","nonce":148592620},{"type":"v1","addr":"172.21.15.69:6833","nonce":148592620}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6830","nonce":148592620},{"type":"v1","addr":"172.21.15.69:6831","nonce":148592620}]},"public_addr":"172.21.15.69:6827/148592620","cluster_addr":"172.21.15.69:6829/148592620","heartbeat_back_addr":"172.21.15.69:6833/148592620","heartbeat_front_addr":"172.21.15.69:6831/148592620","state":["exists","up"]},{"osd":4,"uuid":"cc9f056c-1e48-41c6-80db-dd1ce38c11db","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":30,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6800","nonce":3006695914},{"type":"v1","addr":"172.21.15.122:6801","nonce":3006695914}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6802","nonce":3006695914},{"type":"v1","addr":"172.21.15.122:6803","nonce":3006695914}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6806","nonce":3006695914},{"type":"v1","addr":"172.21.15.122:6807","nonce":3006695914}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6804","nonce":3006695914},{"type":"v1","addr":"172.21.15.122:6805","nonce":3006695914}]},"public_addr":"172.21.15.122:6801/3006695914","cluster_addr":"172.21.15.122:6803/3006695914","heartbeat_back_addr":"172.21.15.122:6807/3006695914","heartbeat_front_addr":"172.21.15.122:6805/3006695914","state":["exists","up"]},{"osd":5,"uuid":"c127fffe-79af-4bf0-b67e-a204f716ab14","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":41,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6808","nonce":4131566432},{"type":"v1","addr":"172.21.15.122:6809","nonce":4131566432}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6810","nonce":4131566432},{"type":"v1","addr":"172.21.15.122:6811","nonce":4131566432}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6814","nonce":4131566432},{"type":"v1","addr":"172.21.15.122:6815","nonce":4131566432}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6812","nonce":4131566432},{"type":"v1","addr":"172.21.15.122:6813","nonce":4131566432}]},"public_addr":"172.21.15.122:6809/4131566432","cluster_addr":"172.21.15.122:6811/4131566432","heartbeat_back_addr":"172.21.15.122:6815/4131566432","heartbeat_front_addr":"172.21.15.122:6813/4131566432","state":["exists","up"]},{"osd":6,"uuid":"6f9f050d-8f2d-4275-8789-46022696eddf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":49,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6816","nonce":173885830},{"type":"v1","addr":"172.21.15.122:6817","nonce":173885830}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6818","nonce":173885830},{"type":"v1","addr":"172.21.15.122:6819","nonce":173885830}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6822","nonce":173885830},{"type":"v1","addr":"172.21.15.122:6823","nonce":173885830}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6820","nonce":173885830},{"type":"v1","addr":"172.21.15.122:6821","nonce":173885830}]},"public_addr":"172.21.15.122:6817/173885830","cluster_addr":"172.21.15.122:6819/173885830","heartbeat_back_addr":"172.21.15.122:6823/173885830","heartbeat_front_addr":"172.21.15.122:6821/173885830","state":["exists","up"]},{"osd":7,"uuid":"7ee70967-f2b6-4b0d-8973-9134433832d9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":54,"up_thru":57,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6824","nonce":2953794780},{"type":"v1","addr":"172.21.15.122:6825","nonce":2953794780}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6826","nonce":2953794780},{"type":"v1","addr":"172.21.15.122:6827","nonce":2953794780}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6830","nonce":2953794780},{"type":"v1","addr":"172.21.15.122:6831","nonce":2953794780}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6828","nonce":2953794780},{"type":"v1","addr":"172.21.15.122:6829","nonce":2953794780}]},"public_addr":"172.21.15.122:6825/2953794780","cluster_addr":"172.21.15.122:6827/2953794780","heartbeat_back_addr":"172.21.15.122:6831/2953794780","heartbeat_front_addr":"172.21.15.122:6829/2953794780","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-04-14T13:52:27.650496+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-04-14T13:52:50.294211+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-04-14T13:53:12.866222+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-04-14T13:53:35.936737+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-04-14T13:53:56.970934+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-04-14T13:54:20.266792+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-04-14T13:54:43.509058+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-04-14T13:55:07.675683+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.69:6800/2212179823":"2024-04-15T13:55:24.256376+0000","172.21.15.69:0/1453816126":"2024-04-15T13:55:24.256376+0000","172.21.15.69:0/3639638985":"2024-04-15T13:55:24.256376+0000","172.21.15.69:0/3170426090":"2024-04-15T13:51:11.392969+0000","172.21.15.69:0/3397430943":"2024-04-15T13:51:11.392969+0000","172.21.15.69:0/226283202":"2024-04-15T13:55:24.256376+0000","172.21.15.69:6801/2212179823":"2024-04-15T13:55:24.256376+0000","172.21.15.69:0/2434136662":"2024-04-15T13:51:11.392969+0000","172.21.15.69:6800/2748095016":"2024-04-15T13:50:49.340545+0000","172.21.15.69:6801/3528824566":"2024-04-15T13:51:11.392969+0000","172.21.15.69:6800/3528824566":"2024-04-15T13:51:11.392969+0000","172.21.15.69:6801/2748095016":"2024-04-15T13:50:49.340545+0000","172.21.15.69:0/3759166769":"2024-04-15T13:50:49.340545+0000","172.21.15.69:0/2144826942":"2024-04-15T13:50:49.340545+0000","172.21.15.69:0/3270155078":"2024-04-15T13:55:24.256376+0000","172.21.15.69:0/1250951073":"2024-04-15T13:50:49.340545+0000"},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-14T13:55:44.771 INFO:journalctl@ceph.node-exporter.a.smithi069.stdout:Apr 14 13:55:44 smithi069 systemd[1]: Starting Ceph node-exporter.a for dc326fc2-fa65-11ee-bc8e-c7b262605968... 2024-04-14T13:55:45.067 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:55:44 smithi069 ceph-mon[24818]: from='mgr.14448 172.21.15.69:0/453473650' entity='mgr.y' 2024-04-14T13:55:45.067 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:55:44 smithi069 ceph-mon[24818]: Deploying daemon node-exporter.a on smithi069 2024-04-14T13:55:45.067 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:55:44 smithi069 ceph-mon[24818]: pgmap v13: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-14T13:55:45.067 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:55:44 smithi069 ceph-mon[24818]: from='mgr.14448 172.21.15.69:0/453473650' entity='mgr.y' 2024-04-14T13:55:45.067 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:55:44 smithi069 ceph-mon[24818]: from='client.? 172.21.15.69:0/3821149243' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-14T13:55:45.068 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:55:44 smithi069 ceph-mon[29472]: from='mgr.14448 172.21.15.69:0/453473650' entity='mgr.y' 2024-04-14T13:55:45.068 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:55:44 smithi069 ceph-mon[29472]: Deploying daemon node-exporter.a on smithi069 2024-04-14T13:55:45.068 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:55:44 smithi069 ceph-mon[29472]: pgmap v13: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-14T13:55:45.068 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:55:44 smithi069 ceph-mon[29472]: from='mgr.14448 172.21.15.69:0/453473650' entity='mgr.y' 2024-04-14T13:55:45.068 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:55:44 smithi069 ceph-mon[29472]: from='client.? 172.21.15.69:0/3821149243' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-14T13:55:45.127 INFO:tasks.cephadm.ceph_manager.ceph:all up! 2024-04-14T13:55:45.127 DEBUG:teuthology.orchestra.run.smithi069:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid dc326fc2-fa65-11ee-bc8e-c7b262605968 -- ceph osd dump --format=json 2024-04-14T13:55:45.345 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:55:44 smithi122 ceph-mon[25721]: from='mgr.14448 172.21.15.69:0/453473650' entity='mgr.y' 2024-04-14T13:55:45.345 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:55:44 smithi122 ceph-mon[25721]: Deploying daemon node-exporter.a on smithi069 2024-04-14T13:55:45.345 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:55:44 smithi122 ceph-mon[25721]: pgmap v13: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-14T13:55:45.345 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:55:44 smithi122 ceph-mon[25721]: from='mgr.14448 172.21.15.69:0/453473650' entity='mgr.y' 2024-04-14T13:55:45.345 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:55:44 smithi122 ceph-mon[25721]: from='client.? 172.21.15.69:0/3821149243' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-14T13:55:45.358 INFO:teuthology.orchestra.run.smithi069.stderr:Inferring config /var/lib/ceph/dc326fc2-fa65-11ee-bc8e-c7b262605968/mon.a/config 2024-04-14T13:55:45.480 INFO:journalctl@ceph.node-exporter.a.smithi069.stdout:Apr 14 13:55:45 smithi069 bash[44370]: Trying to pull quay.io/prometheus/node-exporter:v1.3.1... 2024-04-14T13:55:45.799 INFO:journalctl@ceph.node-exporter.a.smithi069.stdout:Apr 14 13:55:45 smithi069 bash[44370]: Getting image source signatures 2024-04-14T13:55:45.799 INFO:journalctl@ceph.node-exporter.a.smithi069.stdout:Apr 14 13:55:45 smithi069 bash[44370]: Copying blob sha256:b45d31ee2d7f9f452678a85b0c837c29e12089f31ee8dbac6c8c24dfa4054a30 2024-04-14T13:55:45.799 INFO:journalctl@ceph.node-exporter.a.smithi069.stdout:Apr 14 13:55:45 smithi069 bash[44370]: Copying blob sha256:b5db1e299295edf3005515ab7879c1df64a33c185d3a7a23aa4dcaa17d26f7b3 2024-04-14T13:55:45.800 INFO:journalctl@ceph.node-exporter.a.smithi069.stdout:Apr 14 13:55:45 smithi069 bash[44370]: Copying blob sha256:aa2a8d90b84cb2a9c422e7005cd166a008ccf22ef5d7d4f07128478585ce35ea 2024-04-14T13:55:47.646 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-14T13:55:47.647 INFO:teuthology.orchestra.run.smithi069.stdout:{"epoch":59,"fsid":"dc326fc2-fa65-11ee-bc8e-c7b262605968","created":"2024-04-14T13:50:21.024738+0000","modified":"2024-04-14T13:55:24.256417+0000","last_up_change":"2024-04-14T13:55:09.296070+0000","last_in_change":"2024-04-14T13:54:52.122409+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":18,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-14T13:53:15.745972+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}}],"osds":[{"osd":0,"uuid":"99aae7c5-9755-49e8-91c7-fceed3ff30fb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6802","nonce":2517323608},{"type":"v1","addr":"172.21.15.69:6803","nonce":2517323608}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6804","nonce":2517323608},{"type":"v1","addr":"172.21.15.69:6805","nonce":2517323608}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6808","nonce":2517323608},{"type":"v1","addr":"172.21.15.69:6809","nonce":2517323608}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6806","nonce":2517323608},{"type":"v1","addr":"172.21.15.69:6807","nonce":2517323608}]},"public_addr":"172.21.15.69:6803/2517323608","cluster_addr":"172.21.15.69:6805/2517323608","heartbeat_back_addr":"172.21.15.69:6809/2517323608","heartbeat_front_addr":"172.21.15.69:6807/2517323608","state":["exists","up"]},{"osd":1,"uuid":"1d99b840-344d-422a-a466-13ed2d956280","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6810","nonce":1377309913},{"type":"v1","addr":"172.21.15.69:6811","nonce":1377309913}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6812","nonce":1377309913},{"type":"v1","addr":"172.21.15.69:6813","nonce":1377309913}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6816","nonce":1377309913},{"type":"v1","addr":"172.21.15.69:6817","nonce":1377309913}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6814","nonce":1377309913},{"type":"v1","addr":"172.21.15.69:6815","nonce":1377309913}]},"public_addr":"172.21.15.69:6811/1377309913","cluster_addr":"172.21.15.69:6813/1377309913","heartbeat_back_addr":"172.21.15.69:6817/1377309913","heartbeat_front_addr":"172.21.15.69:6815/1377309913","state":["exists","up"]},{"osd":2,"uuid":"f4e0dcf9-5003-4a8f-8a51-fa36f58c0ac4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6818","nonce":2090584066},{"type":"v1","addr":"172.21.15.69:6819","nonce":2090584066}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6820","nonce":2090584066},{"type":"v1","addr":"172.21.15.69:6821","nonce":2090584066}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6824","nonce":2090584066},{"type":"v1","addr":"172.21.15.69:6825","nonce":2090584066}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6822","nonce":2090584066},{"type":"v1","addr":"172.21.15.69:6823","nonce":2090584066}]},"public_addr":"172.21.15.69:6819/2090584066","cluster_addr":"172.21.15.69:6821/2090584066","heartbeat_back_addr":"172.21.15.69:6825/2090584066","heartbeat_front_addr":"172.21.15.69:6823/2090584066","state":["exists","up"]},{"osd":3,"uuid":"27e543e2-f5c0-462d-ab23-d439e06c056e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6826","nonce":148592620},{"type":"v1","addr":"172.21.15.69:6827","nonce":148592620}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6828","nonce":148592620},{"type":"v1","addr":"172.21.15.69:6829","nonce":148592620}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6832","nonce":148592620},{"type":"v1","addr":"172.21.15.69:6833","nonce":148592620}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6830","nonce":148592620},{"type":"v1","addr":"172.21.15.69:6831","nonce":148592620}]},"public_addr":"172.21.15.69:6827/148592620","cluster_addr":"172.21.15.69:6829/148592620","heartbeat_back_addr":"172.21.15.69:6833/148592620","heartbeat_front_addr":"172.21.15.69:6831/148592620","state":["exists","up"]},{"osd":4,"uuid":"cc9f056c-1e48-41c6-80db-dd1ce38c11db","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":30,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6800","nonce":3006695914},{"type":"v1","addr":"172.21.15.122:6801","nonce":3006695914}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6802","nonce":3006695914},{"type":"v1","addr":"172.21.15.122:6803","nonce":3006695914}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6806","nonce":3006695914},{"type":"v1","addr":"172.21.15.122:6807","nonce":3006695914}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6804","nonce":3006695914},{"type":"v1","addr":"172.21.15.122:6805","nonce":3006695914}]},"public_addr":"172.21.15.122:6801/3006695914","cluster_addr":"172.21.15.122:6803/3006695914","heartbeat_back_addr":"172.21.15.122:6807/3006695914","heartbeat_front_addr":"172.21.15.122:6805/3006695914","state":["exists","up"]},{"osd":5,"uuid":"c127fffe-79af-4bf0-b67e-a204f716ab14","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":41,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6808","nonce":4131566432},{"type":"v1","addr":"172.21.15.122:6809","nonce":4131566432}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6810","nonce":4131566432},{"type":"v1","addr":"172.21.15.122:6811","nonce":4131566432}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6814","nonce":4131566432},{"type":"v1","addr":"172.21.15.122:6815","nonce":4131566432}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6812","nonce":4131566432},{"type":"v1","addr":"172.21.15.122:6813","nonce":4131566432}]},"public_addr":"172.21.15.122:6809/4131566432","cluster_addr":"172.21.15.122:6811/4131566432","heartbeat_back_addr":"172.21.15.122:6815/4131566432","heartbeat_front_addr":"172.21.15.122:6813/4131566432","state":["exists","up"]},{"osd":6,"uuid":"6f9f050d-8f2d-4275-8789-46022696eddf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":49,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6816","nonce":173885830},{"type":"v1","addr":"172.21.15.122:6817","nonce":173885830}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6818","nonce":173885830},{"type":"v1","addr":"172.21.15.122:6819","nonce":173885830}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6822","nonce":173885830},{"type":"v1","addr":"172.21.15.122:6823","nonce":173885830}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6820","nonce":173885830},{"type":"v1","addr":"172.21.15.122:6821","nonce":173885830}]},"public_addr":"172.21.15.122:6817/173885830","cluster_addr":"172.21.15.122:6819/173885830","heartbeat_back_addr":"172.21.15.122:6823/173885830","heartbeat_front_addr":"172.21.15.122:6821/173885830","state":["exists","up"]},{"osd":7,"uuid":"7ee70967-f2b6-4b0d-8973-9134433832d9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":54,"up_thru":57,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6824","nonce":2953794780},{"type":"v1","addr":"172.21.15.122:6825","nonce":2953794780}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6826","nonce":2953794780},{"type":"v1","addr":"172.21.15.122:6827","nonce":2953794780}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6830","nonce":2953794780},{"type":"v1","addr":"172.21.15.122:6831","nonce":2953794780}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6828","nonce":2953794780},{"type":"v1","addr":"172.21.15.122:6829","nonce":2953794780}]},"public_addr":"172.21.15.122:6825/2953794780","cluster_addr":"172.21.15.122:6827/2953794780","heartbeat_back_addr":"172.21.15.122:6831/2953794780","heartbeat_front_addr":"172.21.15.122:6829/2953794780","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-04-14T13:52:27.650496+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-04-14T13:52:50.294211+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-04-14T13:53:12.866222+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-04-14T13:53:35.936737+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-04-14T13:53:56.970934+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-04-14T13:54:20.266792+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-04-14T13:54:43.509058+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-04-14T13:55:07.675683+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.69:6800/2212179823":"2024-04-15T13:55:24.256376+0000","172.21.15.69:0/1453816126":"2024-04-15T13:55:24.256376+0000","172.21.15.69:0/3639638985":"2024-04-15T13:55:24.256376+0000","172.21.15.69:0/3170426090":"2024-04-15T13:51:11.392969+0000","172.21.15.69:0/3397430943":"2024-04-15T13:51:11.392969+0000","172.21.15.69:0/226283202":"2024-04-15T13:55:24.256376+0000","172.21.15.69:6801/2212179823":"2024-04-15T13:55:24.256376+0000","172.21.15.69:0/2434136662":"2024-04-15T13:51:11.392969+0000","172.21.15.69:6800/2748095016":"2024-04-15T13:50:49.340545+0000","172.21.15.69:6801/3528824566":"2024-04-15T13:51:11.392969+0000","172.21.15.69:6800/3528824566":"2024-04-15T13:51:11.392969+0000","172.21.15.69:6801/2748095016":"2024-04-15T13:50:49.340545+0000","172.21.15.69:0/3759166769":"2024-04-15T13:50:49.340545+0000","172.21.15.69:0/2144826942":"2024-04-15T13:50:49.340545+0000","172.21.15.69:0/3270155078":"2024-04-15T13:55:24.256376+0000","172.21.15.69:0/1250951073":"2024-04-15T13:50:49.340545+0000"},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-14T13:55:47.647 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:55:47 smithi069 ceph-mon[24818]: pgmap v14: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-14T13:55:47.647 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:55:47 smithi069 ceph-mon[29472]: pgmap v14: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-14T13:55:47.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:55:47 smithi122 ceph-mon[25721]: pgmap v14: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-14T13:55:48.372 INFO:journalctl@ceph.node-exporter.a.smithi069.stdout:Apr 14 13:55:48 smithi069 bash[44370]: Copying config sha256:1dbe0e931976487e20e5cfb272087e08a9779c88fd5e9617ed7042dd9751ec26 2024-04-14T13:55:48.372 INFO:journalctl@ceph.node-exporter.a.smithi069.stdout:Apr 14 13:55:48 smithi069 bash[44370]: Writing manifest to image destination 2024-04-14T13:55:48.642 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:55:48 smithi069 ceph-mon[24818]: from='client.? 172.21.15.69:0/1026350855' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-14T13:55:48.642 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:55:48 smithi069 ceph-mon[29472]: from='client.? 172.21.15.69:0/1026350855' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-14T13:55:48.771 DEBUG:teuthology.orchestra.run.smithi069:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid dc326fc2-fa65-11ee-bc8e-c7b262605968 -- ceph tell osd.0 flush_pg_stats 2024-04-14T13:55:48.771 DEBUG:teuthology.orchestra.run.smithi069:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid dc326fc2-fa65-11ee-bc8e-c7b262605968 -- ceph tell osd.1 flush_pg_stats 2024-04-14T13:55:48.772 DEBUG:teuthology.orchestra.run.smithi069:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid dc326fc2-fa65-11ee-bc8e-c7b262605968 -- ceph tell osd.2 flush_pg_stats 2024-04-14T13:55:48.772 DEBUG:teuthology.orchestra.run.smithi069:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid dc326fc2-fa65-11ee-bc8e-c7b262605968 -- ceph tell osd.3 flush_pg_stats 2024-04-14T13:55:48.772 DEBUG:teuthology.orchestra.run.smithi069:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid dc326fc2-fa65-11ee-bc8e-c7b262605968 -- ceph tell osd.4 flush_pg_stats 2024-04-14T13:55:48.773 DEBUG:teuthology.orchestra.run.smithi069:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid dc326fc2-fa65-11ee-bc8e-c7b262605968 -- ceph tell osd.5 flush_pg_stats 2024-04-14T13:55:48.773 DEBUG:teuthology.orchestra.run.smithi069:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid dc326fc2-fa65-11ee-bc8e-c7b262605968 -- ceph tell osd.6 flush_pg_stats 2024-04-14T13:55:48.774 DEBUG:teuthology.orchestra.run.smithi069:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid dc326fc2-fa65-11ee-bc8e-c7b262605968 -- ceph tell osd.7 flush_pg_stats 2024-04-14T13:55:48.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:55:48 smithi122 ceph-mon[25721]: from='client.? 172.21.15.69:0/1026350855' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-14T13:55:49.003 INFO:journalctl@ceph.node-exporter.a.smithi069.stdout:Apr 14 13:55:48 smithi069 podman[44370]: 2024-04-14 13:55:48.619405977 +0000 UTC m=+3.408345793 image pull 1dbe0e931976487e20e5cfb272087e08a9779c88fd5e9617ed7042dd9751ec26 quay.io/prometheus/node-exporter:v1.3.1 2024-04-14T13:55:49.003 INFO:journalctl@ceph.node-exporter.a.smithi069.stdout:Apr 14 13:55:48 smithi069 podman[44370]: 2024-04-14 13:55:48.876435873 +0000 UTC m=+3.665375704 container create 73e900b404dfe98aa7d271cd50d2ebbb14c7206183ed360fe056923b67536bee (image=quay.io/prometheus/node-exporter:v1.3.1, name=ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-node-exporter-a, maintainer=The Prometheus Authors ) 2024-04-14T13:55:49.241 INFO:teuthology.orchestra.run.smithi069.stderr:Inferring config /var/lib/ceph/dc326fc2-fa65-11ee-bc8e-c7b262605968/mon.a/config 2024-04-14T13:55:49.250 INFO:teuthology.orchestra.run.smithi069.stderr:Inferring config /var/lib/ceph/dc326fc2-fa65-11ee-bc8e-c7b262605968/mon.a/config 2024-04-14T13:55:49.252 INFO:teuthology.orchestra.run.smithi069.stderr:Inferring config /var/lib/ceph/dc326fc2-fa65-11ee-bc8e-c7b262605968/mon.a/config 2024-04-14T13:55:49.264 INFO:teuthology.orchestra.run.smithi069.stderr:Inferring config /var/lib/ceph/dc326fc2-fa65-11ee-bc8e-c7b262605968/mon.a/config 2024-04-14T13:55:49.280 INFO:journalctl@ceph.node-exporter.a.smithi069.stdout:Apr 14 13:55:49 smithi069 podman[44370]: 2024-04-14 13:55:49.095409211 +0000 UTC m=+3.884349043 container init 73e900b404dfe98aa7d271cd50d2ebbb14c7206183ed360fe056923b67536bee (image=quay.io/prometheus/node-exporter:v1.3.1, name=ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-node-exporter-a, maintainer=The Prometheus Authors ) 2024-04-14T13:55:49.280 INFO:journalctl@ceph.node-exporter.a.smithi069.stdout:Apr 14 13:55:49 smithi069 podman[44370]: 2024-04-14 13:55:49.099681087 +0000 UTC m=+3.888620911 container start 73e900b404dfe98aa7d271cd50d2ebbb14c7206183ed360fe056923b67536bee (image=quay.io/prometheus/node-exporter:v1.3.1, name=ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-node-exporter-a, maintainer=The Prometheus Authors ) 2024-04-14T13:55:49.280 INFO:journalctl@ceph.node-exporter.a.smithi069.stdout:Apr 14 13:55:49 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-node-exporter-a[44689]: ts=2024-04-14T13:55:49.108Z caller=node_exporter.go:182 level=info msg="Starting node_exporter" version="(version=1.3.1, branch=HEAD, revision=a2321e7b940ddcff26873612bccdf7cd4c42b6b6)" 2024-04-14T13:55:49.280 INFO:journalctl@ceph.node-exporter.a.smithi069.stdout:Apr 14 13:55:49 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-node-exporter-a[44689]: ts=2024-04-14T13:55:49.109Z caller=node_exporter.go:183 level=info msg="Build context" build_context="(go=go1.17.3, user=root@243aafa5525c, date=20211205-11:09:49)" 2024-04-14T13:55:49.280 INFO:journalctl@ceph.node-exporter.a.smithi069.stdout:Apr 14 13:55:49 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-node-exporter-a[44689]: ts=2024-04-14T13:55:49.110Z caller=filesystem_common.go:111 level=info collector=filesystem msg="Parsed flag --collector.filesystem.mount-points-exclude" flag=^/(dev|proc|run/credentials/.+|sys|var/lib/docker/.+)($|/) 2024-04-14T13:55:49.280 INFO:journalctl@ceph.node-exporter.a.smithi069.stdout:Apr 14 13:55:49 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-node-exporter-a[44689]: ts=2024-04-14T13:55:49.110Z caller=filesystem_common.go:113 level=info collector=filesystem msg="Parsed flag --collector.filesystem.fs-types-exclude" flag=^(autofs|binfmt_misc|bpf|cgroup2?|configfs|debugfs|devpts|devtmpfs|fusectl|hugetlbfs|iso9660|mqueue|nsfs|overlay|proc|procfs|pstore|rpc_pipefs|securityfs|selinuxfs|squashfs|sysfs|tracefs)$ 2024-04-14T13:55:49.281 INFO:journalctl@ceph.node-exporter.a.smithi069.stdout:Apr 14 13:55:49 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-node-exporter-a[44689]: ts=2024-04-14T13:55:49.111Z caller=node_exporter.go:108 level=info msg="Enabled collectors" 2024-04-14T13:55:49.281 INFO:journalctl@ceph.node-exporter.a.smithi069.stdout:Apr 14 13:55:49 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-node-exporter-a[44689]: ts=2024-04-14T13:55:49.111Z caller=node_exporter.go:115 level=info collector=arp 2024-04-14T13:55:49.281 INFO:journalctl@ceph.node-exporter.a.smithi069.stdout:Apr 14 13:55:49 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-node-exporter-a[44689]: ts=2024-04-14T13:55:49.111Z caller=node_exporter.go:115 level=info collector=bcache 2024-04-14T13:55:49.281 INFO:journalctl@ceph.node-exporter.a.smithi069.stdout:Apr 14 13:55:49 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-node-exporter-a[44689]: ts=2024-04-14T13:55:49.111Z caller=node_exporter.go:115 level=info collector=bonding 2024-04-14T13:55:49.281 INFO:journalctl@ceph.node-exporter.a.smithi069.stdout:Apr 14 13:55:49 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-node-exporter-a[44689]: ts=2024-04-14T13:55:49.111Z caller=node_exporter.go:115 level=info collector=btrfs 2024-04-14T13:55:49.281 INFO:journalctl@ceph.node-exporter.a.smithi069.stdout:Apr 14 13:55:49 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-node-exporter-a[44689]: ts=2024-04-14T13:55:49.111Z caller=node_exporter.go:115 level=info collector=conntrack 2024-04-14T13:55:49.281 INFO:journalctl@ceph.node-exporter.a.smithi069.stdout:Apr 14 13:55:49 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-node-exporter-a[44689]: ts=2024-04-14T13:55:49.111Z caller=node_exporter.go:115 level=info collector=cpu 2024-04-14T13:55:49.281 INFO:journalctl@ceph.node-exporter.a.smithi069.stdout:Apr 14 13:55:49 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-node-exporter-a[44689]: ts=2024-04-14T13:55:49.111Z caller=node_exporter.go:115 level=info collector=cpufreq 2024-04-14T13:55:49.281 INFO:journalctl@ceph.node-exporter.a.smithi069.stdout:Apr 14 13:55:49 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-node-exporter-a[44689]: ts=2024-04-14T13:55:49.112Z caller=node_exporter.go:115 level=info collector=diskstats 2024-04-14T13:55:49.281 INFO:journalctl@ceph.node-exporter.a.smithi069.stdout:Apr 14 13:55:49 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-node-exporter-a[44689]: ts=2024-04-14T13:55:49.112Z caller=node_exporter.go:115 level=info collector=dmi 2024-04-14T13:55:49.281 INFO:journalctl@ceph.node-exporter.a.smithi069.stdout:Apr 14 13:55:49 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-node-exporter-a[44689]: ts=2024-04-14T13:55:49.112Z caller=node_exporter.go:115 level=info collector=edac 2024-04-14T13:55:49.281 INFO:journalctl@ceph.node-exporter.a.smithi069.stdout:Apr 14 13:55:49 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-node-exporter-a[44689]: ts=2024-04-14T13:55:49.112Z caller=node_exporter.go:115 level=info collector=entropy 2024-04-14T13:55:49.281 INFO:journalctl@ceph.node-exporter.a.smithi069.stdout:Apr 14 13:55:49 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-node-exporter-a[44689]: ts=2024-04-14T13:55:49.112Z caller=node_exporter.go:115 level=info collector=fibrechannel 2024-04-14T13:55:49.282 INFO:journalctl@ceph.node-exporter.a.smithi069.stdout:Apr 14 13:55:49 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-node-exporter-a[44689]: ts=2024-04-14T13:55:49.112Z caller=node_exporter.go:115 level=info collector=filefd 2024-04-14T13:55:49.282 INFO:journalctl@ceph.node-exporter.a.smithi069.stdout:Apr 14 13:55:49 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-node-exporter-a[44689]: ts=2024-04-14T13:55:49.112Z caller=node_exporter.go:115 level=info collector=filesystem 2024-04-14T13:55:49.282 INFO:journalctl@ceph.node-exporter.a.smithi069.stdout:Apr 14 13:55:49 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-node-exporter-a[44689]: ts=2024-04-14T13:55:49.112Z caller=node_exporter.go:115 level=info collector=hwmon 2024-04-14T13:55:49.282 INFO:journalctl@ceph.node-exporter.a.smithi069.stdout:Apr 14 13:55:49 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-node-exporter-a[44689]: ts=2024-04-14T13:55:49.112Z caller=node_exporter.go:115 level=info collector=infiniband 2024-04-14T13:55:49.282 INFO:journalctl@ceph.node-exporter.a.smithi069.stdout:Apr 14 13:55:49 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-node-exporter-a[44689]: ts=2024-04-14T13:55:49.112Z caller=node_exporter.go:115 level=info collector=ipvs 2024-04-14T13:55:49.282 INFO:journalctl@ceph.node-exporter.a.smithi069.stdout:Apr 14 13:55:49 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-node-exporter-a[44689]: ts=2024-04-14T13:55:49.112Z caller=node_exporter.go:115 level=info collector=loadavg 2024-04-14T13:55:49.282 INFO:journalctl@ceph.node-exporter.a.smithi069.stdout:Apr 14 13:55:49 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-node-exporter-a[44689]: ts=2024-04-14T13:55:49.112Z caller=node_exporter.go:115 level=info collector=mdadm 2024-04-14T13:55:49.282 INFO:journalctl@ceph.node-exporter.a.smithi069.stdout:Apr 14 13:55:49 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-node-exporter-a[44689]: ts=2024-04-14T13:55:49.112Z caller=node_exporter.go:115 level=info collector=meminfo 2024-04-14T13:55:49.282 INFO:journalctl@ceph.node-exporter.a.smithi069.stdout:Apr 14 13:55:49 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-node-exporter-a[44689]: ts=2024-04-14T13:55:49.112Z caller=node_exporter.go:115 level=info collector=netclass 2024-04-14T13:55:49.282 INFO:journalctl@ceph.node-exporter.a.smithi069.stdout:Apr 14 13:55:49 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-node-exporter-a[44689]: ts=2024-04-14T13:55:49.112Z caller=node_exporter.go:115 level=info collector=netdev 2024-04-14T13:55:49.282 INFO:journalctl@ceph.node-exporter.a.smithi069.stdout:Apr 14 13:55:49 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-node-exporter-a[44689]: ts=2024-04-14T13:55:49.112Z caller=node_exporter.go:115 level=info collector=netstat 2024-04-14T13:55:49.282 INFO:journalctl@ceph.node-exporter.a.smithi069.stdout:Apr 14 13:55:49 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-node-exporter-a[44689]: ts=2024-04-14T13:55:49.112Z caller=node_exporter.go:115 level=info collector=nfs 2024-04-14T13:55:49.282 INFO:journalctl@ceph.node-exporter.a.smithi069.stdout:Apr 14 13:55:49 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-node-exporter-a[44689]: ts=2024-04-14T13:55:49.112Z caller=node_exporter.go:115 level=info collector=nfsd 2024-04-14T13:55:49.282 INFO:journalctl@ceph.node-exporter.a.smithi069.stdout:Apr 14 13:55:49 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-node-exporter-a[44689]: ts=2024-04-14T13:55:49.112Z caller=node_exporter.go:115 level=info collector=nvme 2024-04-14T13:55:49.283 INFO:journalctl@ceph.node-exporter.a.smithi069.stdout:Apr 14 13:55:49 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-node-exporter-a[44689]: ts=2024-04-14T13:55:49.112Z caller=node_exporter.go:115 level=info collector=os 2024-04-14T13:55:49.283 INFO:journalctl@ceph.node-exporter.a.smithi069.stdout:Apr 14 13:55:49 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-node-exporter-a[44689]: ts=2024-04-14T13:55:49.112Z caller=node_exporter.go:115 level=info collector=powersupplyclass 2024-04-14T13:55:49.283 INFO:journalctl@ceph.node-exporter.a.smithi069.stdout:Apr 14 13:55:49 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-node-exporter-a[44689]: ts=2024-04-14T13:55:49.112Z caller=node_exporter.go:115 level=info collector=pressure 2024-04-14T13:55:49.283 INFO:journalctl@ceph.node-exporter.a.smithi069.stdout:Apr 14 13:55:49 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-node-exporter-a[44689]: ts=2024-04-14T13:55:49.112Z caller=node_exporter.go:115 level=info collector=rapl 2024-04-14T13:55:49.283 INFO:journalctl@ceph.node-exporter.a.smithi069.stdout:Apr 14 13:55:49 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-node-exporter-a[44689]: ts=2024-04-14T13:55:49.112Z caller=node_exporter.go:115 level=info collector=schedstat 2024-04-14T13:55:49.283 INFO:journalctl@ceph.node-exporter.a.smithi069.stdout:Apr 14 13:55:49 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-node-exporter-a[44689]: ts=2024-04-14T13:55:49.112Z caller=node_exporter.go:115 level=info collector=sockstat 2024-04-14T13:55:49.283 INFO:journalctl@ceph.node-exporter.a.smithi069.stdout:Apr 14 13:55:49 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-node-exporter-a[44689]: ts=2024-04-14T13:55:49.112Z caller=node_exporter.go:115 level=info collector=softnet 2024-04-14T13:55:49.283 INFO:journalctl@ceph.node-exporter.a.smithi069.stdout:Apr 14 13:55:49 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-node-exporter-a[44689]: ts=2024-04-14T13:55:49.112Z caller=node_exporter.go:115 level=info collector=stat 2024-04-14T13:55:49.283 INFO:journalctl@ceph.node-exporter.a.smithi069.stdout:Apr 14 13:55:49 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-node-exporter-a[44689]: ts=2024-04-14T13:55:49.112Z caller=node_exporter.go:115 level=info collector=tapestats 2024-04-14T13:55:49.283 INFO:journalctl@ceph.node-exporter.a.smithi069.stdout:Apr 14 13:55:49 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-node-exporter-a[44689]: ts=2024-04-14T13:55:49.112Z caller=node_exporter.go:115 level=info collector=textfile 2024-04-14T13:55:49.283 INFO:journalctl@ceph.node-exporter.a.smithi069.stdout:Apr 14 13:55:49 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-node-exporter-a[44689]: ts=2024-04-14T13:55:49.112Z caller=node_exporter.go:115 level=info collector=thermal_zone 2024-04-14T13:55:49.283 INFO:journalctl@ceph.node-exporter.a.smithi069.stdout:Apr 14 13:55:49 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-node-exporter-a[44689]: ts=2024-04-14T13:55:49.112Z caller=node_exporter.go:115 level=info collector=time 2024-04-14T13:55:49.283 INFO:journalctl@ceph.node-exporter.a.smithi069.stdout:Apr 14 13:55:49 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-node-exporter-a[44689]: ts=2024-04-14T13:55:49.112Z caller=node_exporter.go:115 level=info collector=udp_queues 2024-04-14T13:55:49.284 INFO:journalctl@ceph.node-exporter.a.smithi069.stdout:Apr 14 13:55:49 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-node-exporter-a[44689]: ts=2024-04-14T13:55:49.112Z caller=node_exporter.go:115 level=info collector=uname 2024-04-14T13:55:49.284 INFO:journalctl@ceph.node-exporter.a.smithi069.stdout:Apr 14 13:55:49 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-node-exporter-a[44689]: ts=2024-04-14T13:55:49.112Z caller=node_exporter.go:115 level=info collector=vmstat 2024-04-14T13:55:49.284 INFO:journalctl@ceph.node-exporter.a.smithi069.stdout:Apr 14 13:55:49 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-node-exporter-a[44689]: ts=2024-04-14T13:55:49.112Z caller=node_exporter.go:115 level=info collector=xfs 2024-04-14T13:55:49.284 INFO:journalctl@ceph.node-exporter.a.smithi069.stdout:Apr 14 13:55:49 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-node-exporter-a[44689]: ts=2024-04-14T13:55:49.112Z caller=node_exporter.go:115 level=info collector=zfs 2024-04-14T13:55:49.284 INFO:journalctl@ceph.node-exporter.a.smithi069.stdout:Apr 14 13:55:49 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-node-exporter-a[44689]: ts=2024-04-14T13:55:49.112Z caller=node_exporter.go:199 level=info msg="Listening on" address=:9100 2024-04-14T13:55:49.284 INFO:journalctl@ceph.node-exporter.a.smithi069.stdout:Apr 14 13:55:49 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-node-exporter-a[44689]: ts=2024-04-14T13:55:49.112Z caller=tls_config.go:195 level=info msg="TLS is disabled." http2=false 2024-04-14T13:55:49.284 INFO:journalctl@ceph.node-exporter.a.smithi069.stdout:Apr 14 13:55:49 smithi069 bash[44370]: 73e900b404dfe98aa7d271cd50d2ebbb14c7206183ed360fe056923b67536bee 2024-04-14T13:55:49.284 INFO:journalctl@ceph.node-exporter.a.smithi069.stdout:Apr 14 13:55:49 smithi069 systemd[1]: Started Ceph node-exporter.a for dc326fc2-fa65-11ee-bc8e-c7b262605968. 2024-04-14T13:55:49.300 INFO:teuthology.orchestra.run.smithi069.stderr:Inferring config /var/lib/ceph/dc326fc2-fa65-11ee-bc8e-c7b262605968/mon.a/config 2024-04-14T13:55:49.304 INFO:teuthology.orchestra.run.smithi069.stderr:Inferring config /var/lib/ceph/dc326fc2-fa65-11ee-bc8e-c7b262605968/mon.a/config 2024-04-14T13:55:49.304 INFO:teuthology.orchestra.run.smithi069.stderr:Inferring config /var/lib/ceph/dc326fc2-fa65-11ee-bc8e-c7b262605968/mon.a/config 2024-04-14T13:55:49.305 INFO:teuthology.orchestra.run.smithi069.stderr:Inferring config /var/lib/ceph/dc326fc2-fa65-11ee-bc8e-c7b262605968/mon.a/config 2024-04-14T13:55:49.534 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:55:49 smithi069 ceph-mon[24818]: pgmap v15: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-14T13:55:49.534 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:55:49 smithi069 ceph-mon[24818]: from='mgr.14448 172.21.15.69:0/453473650' entity='mgr.y' 2024-04-14T13:55:49.535 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:55:49 smithi069 ceph-mon[29472]: pgmap v15: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-14T13:55:49.535 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:55:49 smithi069 ceph-mon[29472]: from='mgr.14448 172.21.15.69:0/453473650' entity='mgr.y' 2024-04-14T13:55:49.595 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:55:49 smithi122 ceph-mon[25721]: pgmap v15: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-14T13:55:49.595 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:55:49 smithi122 ceph-mon[25721]: from='mgr.14448 172.21.15.69:0/453473650' entity='mgr.y' 2024-04-14T13:55:49.802 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 14 13:55:49 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-y[25054]: ::ffff:172.21.15.122 - - [14/Apr/2024:13:55:49] "GET /metrics HTTP/1.1" 200 191529 "" "Prometheus/2.33.4" 2024-04-14T13:55:49.969 INFO:journalctl@ceph.node-exporter.b.smithi122.stdout:Apr 14 13:55:49 smithi122 systemd[1]: Starting Ceph node-exporter.b for dc326fc2-fa65-11ee-bc8e-c7b262605968... 2024-04-14T13:55:50.345 INFO:journalctl@ceph.node-exporter.b.smithi122.stdout:Apr 14 13:55:50 smithi122 bash[41105]: Trying to pull quay.io/prometheus/node-exporter:v1.3.1... 2024-04-14T13:55:50.567 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:55:50 smithi069 ceph-mon[24818]: Deploying daemon node-exporter.b on smithi122 2024-04-14T13:55:50.567 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:55:50 smithi069 ceph-mon[29472]: Deploying daemon node-exporter.b on smithi122 2024-04-14T13:55:50.624 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:55:50 smithi122 ceph-mon[25721]: Deploying daemon node-exporter.b on smithi122 2024-04-14T13:55:51.095 INFO:journalctl@ceph.node-exporter.b.smithi122.stdout:Apr 14 13:55:50 smithi122 bash[41105]: Getting image source signatures 2024-04-14T13:55:51.095 INFO:journalctl@ceph.node-exporter.b.smithi122.stdout:Apr 14 13:55:50 smithi122 bash[41105]: Copying blob sha256:b5db1e299295edf3005515ab7879c1df64a33c185d3a7a23aa4dcaa17d26f7b3 2024-04-14T13:55:51.095 INFO:journalctl@ceph.node-exporter.b.smithi122.stdout:Apr 14 13:55:50 smithi122 bash[41105]: Copying blob sha256:aa2a8d90b84cb2a9c422e7005cd166a008ccf22ef5d7d4f07128478585ce35ea 2024-04-14T13:55:51.095 INFO:journalctl@ceph.node-exporter.b.smithi122.stdout:Apr 14 13:55:50 smithi122 bash[41105]: Copying blob sha256:b45d31ee2d7f9f452678a85b0c837c29e12089f31ee8dbac6c8c24dfa4054a30 2024-04-14T13:55:51.567 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:55:51 smithi069 ceph-mon[24818]: pgmap v16: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-14T13:55:51.567 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:55:51 smithi069 ceph-mon[29472]: pgmap v16: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-14T13:55:51.613 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:55:51 smithi122 ceph-mon[25721]: pgmap v16: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-14T13:55:53.595 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:55:53 smithi122 ceph-mon[25721]: pgmap v17: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-14T13:55:53.595 INFO:journalctl@ceph.node-exporter.b.smithi122.stdout:Apr 14 13:55:53 smithi122 bash[41105]: Copying config sha256:1dbe0e931976487e20e5cfb272087e08a9779c88fd5e9617ed7042dd9751ec26 2024-04-14T13:55:53.595 INFO:journalctl@ceph.node-exporter.b.smithi122.stdout:Apr 14 13:55:53 smithi122 bash[41105]: Writing manifest to image destination 2024-04-14T13:55:53.696 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:55:53 smithi069 ceph-mon[24818]: pgmap v17: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-14T13:55:53.697 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:55:53 smithi069 ceph-mon[29472]: pgmap v17: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-14T13:55:53.897 INFO:journalctl@ceph.node-exporter.b.smithi122.stdout:Apr 14 13:55:53 smithi122 podman[41105]: 2024-04-14 13:55:53.50144555 +0000 UTC m=+3.434397609 image pull 1dbe0e931976487e20e5cfb272087e08a9779c88fd5e9617ed7042dd9751ec26 quay.io/prometheus/node-exporter:v1.3.1 2024-04-14T13:55:53.897 INFO:journalctl@ceph.node-exporter.b.smithi122.stdout:Apr 14 13:55:53 smithi122 podman[41105]: 2024-04-14 13:55:53.600344357 +0000 UTC m=+3.533296466 container create e57f3d5931fbce91c3bf4d2422435c6b3ea9b89efe9f0b019b7925d0cac49251 (image=quay.io/prometheus/node-exporter:v1.3.1, name=ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-node-exporter-b, maintainer=The Prometheus Authors ) 2024-04-14T13:55:53.897 INFO:journalctl@ceph.node-exporter.b.smithi122.stdout:Apr 14 13:55:53 smithi122 podman[41105]: 2024-04-14 13:55:53.827866104 +0000 UTC m=+3.760818161 container init e57f3d5931fbce91c3bf4d2422435c6b3ea9b89efe9f0b019b7925d0cac49251 (image=quay.io/prometheus/node-exporter:v1.3.1, name=ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-node-exporter-b, maintainer=The Prometheus Authors ) 2024-04-14T13:55:53.897 INFO:journalctl@ceph.node-exporter.b.smithi122.stdout:Apr 14 13:55:53 smithi122 podman[41105]: 2024-04-14 13:55:53.830750834 +0000 UTC m=+3.763702883 container start e57f3d5931fbce91c3bf4d2422435c6b3ea9b89efe9f0b019b7925d0cac49251 (image=quay.io/prometheus/node-exporter:v1.3.1, name=ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-node-exporter-b, maintainer=The Prometheus Authors ) 2024-04-14T13:55:53.897 INFO:journalctl@ceph.node-exporter.b.smithi122.stdout:Apr 14 13:55:53 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-node-exporter-b[41167]: ts=2024-04-14T13:55:53.834Z caller=node_exporter.go:182 level=info msg="Starting node_exporter" version="(version=1.3.1, branch=HEAD, revision=a2321e7b940ddcff26873612bccdf7cd4c42b6b6)" 2024-04-14T13:55:53.897 INFO:journalctl@ceph.node-exporter.b.smithi122.stdout:Apr 14 13:55:53 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-node-exporter-b[41167]: ts=2024-04-14T13:55:53.835Z caller=node_exporter.go:183 level=info msg="Build context" build_context="(go=go1.17.3, user=root@243aafa5525c, date=20211205-11:09:49)" 2024-04-14T13:55:53.897 INFO:journalctl@ceph.node-exporter.b.smithi122.stdout:Apr 14 13:55:53 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-node-exporter-b[41167]: ts=2024-04-14T13:55:53.835Z caller=filesystem_common.go:111 level=info collector=filesystem msg="Parsed flag --collector.filesystem.mount-points-exclude" flag=^/(dev|proc|run/credentials/.+|sys|var/lib/docker/.+)($|/) 2024-04-14T13:55:53.897 INFO:journalctl@ceph.node-exporter.b.smithi122.stdout:Apr 14 13:55:53 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-node-exporter-b[41167]: ts=2024-04-14T13:55:53.835Z caller=filesystem_common.go:113 level=info collector=filesystem msg="Parsed flag --collector.filesystem.fs-types-exclude" flag=^(autofs|binfmt_misc|bpf|cgroup2?|configfs|debugfs|devpts|devtmpfs|fusectl|hugetlbfs|iso9660|mqueue|nsfs|overlay|proc|procfs|pstore|rpc_pipefs|securityfs|selinuxfs|squashfs|sysfs|tracefs)$ 2024-04-14T13:55:53.897 INFO:journalctl@ceph.node-exporter.b.smithi122.stdout:Apr 14 13:55:53 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-node-exporter-b[41167]: ts=2024-04-14T13:55:53.836Z caller=node_exporter.go:108 level=info msg="Enabled collectors" 2024-04-14T13:55:53.897 INFO:journalctl@ceph.node-exporter.b.smithi122.stdout:Apr 14 13:55:53 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-node-exporter-b[41167]: ts=2024-04-14T13:55:53.836Z caller=node_exporter.go:115 level=info collector=arp 2024-04-14T13:55:53.897 INFO:journalctl@ceph.node-exporter.b.smithi122.stdout:Apr 14 13:55:53 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-node-exporter-b[41167]: ts=2024-04-14T13:55:53.836Z caller=node_exporter.go:115 level=info collector=bcache 2024-04-14T13:55:53.898 INFO:journalctl@ceph.node-exporter.b.smithi122.stdout:Apr 14 13:55:53 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-node-exporter-b[41167]: ts=2024-04-14T13:55:53.836Z caller=node_exporter.go:115 level=info collector=bonding 2024-04-14T13:55:53.898 INFO:journalctl@ceph.node-exporter.b.smithi122.stdout:Apr 14 13:55:53 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-node-exporter-b[41167]: ts=2024-04-14T13:55:53.836Z caller=node_exporter.go:115 level=info collector=btrfs 2024-04-14T13:55:53.898 INFO:journalctl@ceph.node-exporter.b.smithi122.stdout:Apr 14 13:55:53 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-node-exporter-b[41167]: ts=2024-04-14T13:55:53.836Z caller=node_exporter.go:115 level=info collector=conntrack 2024-04-14T13:55:53.898 INFO:journalctl@ceph.node-exporter.b.smithi122.stdout:Apr 14 13:55:53 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-node-exporter-b[41167]: ts=2024-04-14T13:55:53.836Z caller=node_exporter.go:115 level=info collector=cpu 2024-04-14T13:55:53.898 INFO:journalctl@ceph.node-exporter.b.smithi122.stdout:Apr 14 13:55:53 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-node-exporter-b[41167]: ts=2024-04-14T13:55:53.836Z caller=node_exporter.go:115 level=info collector=cpufreq 2024-04-14T13:55:53.898 INFO:journalctl@ceph.node-exporter.b.smithi122.stdout:Apr 14 13:55:53 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-node-exporter-b[41167]: ts=2024-04-14T13:55:53.836Z caller=node_exporter.go:115 level=info collector=diskstats 2024-04-14T13:55:53.898 INFO:journalctl@ceph.node-exporter.b.smithi122.stdout:Apr 14 13:55:53 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-node-exporter-b[41167]: ts=2024-04-14T13:55:53.836Z caller=node_exporter.go:115 level=info collector=dmi 2024-04-14T13:55:53.898 INFO:journalctl@ceph.node-exporter.b.smithi122.stdout:Apr 14 13:55:53 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-node-exporter-b[41167]: ts=2024-04-14T13:55:53.836Z caller=node_exporter.go:115 level=info collector=edac 2024-04-14T13:55:53.898 INFO:journalctl@ceph.node-exporter.b.smithi122.stdout:Apr 14 13:55:53 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-node-exporter-b[41167]: ts=2024-04-14T13:55:53.836Z caller=node_exporter.go:115 level=info collector=entropy 2024-04-14T13:55:53.898 INFO:journalctl@ceph.node-exporter.b.smithi122.stdout:Apr 14 13:55:53 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-node-exporter-b[41167]: ts=2024-04-14T13:55:53.836Z caller=node_exporter.go:115 level=info collector=fibrechannel 2024-04-14T13:55:53.898 INFO:journalctl@ceph.node-exporter.b.smithi122.stdout:Apr 14 13:55:53 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-node-exporter-b[41167]: ts=2024-04-14T13:55:53.836Z caller=node_exporter.go:115 level=info collector=filefd 2024-04-14T13:55:53.898 INFO:journalctl@ceph.node-exporter.b.smithi122.stdout:Apr 14 13:55:53 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-node-exporter-b[41167]: ts=2024-04-14T13:55:53.836Z caller=node_exporter.go:115 level=info collector=filesystem 2024-04-14T13:55:53.899 INFO:journalctl@ceph.node-exporter.b.smithi122.stdout:Apr 14 13:55:53 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-node-exporter-b[41167]: ts=2024-04-14T13:55:53.836Z caller=node_exporter.go:115 level=info collector=hwmon 2024-04-14T13:55:53.899 INFO:journalctl@ceph.node-exporter.b.smithi122.stdout:Apr 14 13:55:53 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-node-exporter-b[41167]: ts=2024-04-14T13:55:53.836Z caller=node_exporter.go:115 level=info collector=infiniband 2024-04-14T13:55:53.899 INFO:journalctl@ceph.node-exporter.b.smithi122.stdout:Apr 14 13:55:53 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-node-exporter-b[41167]: ts=2024-04-14T13:55:53.836Z caller=node_exporter.go:115 level=info collector=ipvs 2024-04-14T13:55:53.899 INFO:journalctl@ceph.node-exporter.b.smithi122.stdout:Apr 14 13:55:53 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-node-exporter-b[41167]: ts=2024-04-14T13:55:53.836Z caller=node_exporter.go:115 level=info collector=loadavg 2024-04-14T13:55:53.899 INFO:journalctl@ceph.node-exporter.b.smithi122.stdout:Apr 14 13:55:53 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-node-exporter-b[41167]: ts=2024-04-14T13:55:53.836Z caller=node_exporter.go:115 level=info collector=mdadm 2024-04-14T13:55:53.899 INFO:journalctl@ceph.node-exporter.b.smithi122.stdout:Apr 14 13:55:53 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-node-exporter-b[41167]: ts=2024-04-14T13:55:53.836Z caller=node_exporter.go:115 level=info collector=meminfo 2024-04-14T13:55:53.899 INFO:journalctl@ceph.node-exporter.b.smithi122.stdout:Apr 14 13:55:53 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-node-exporter-b[41167]: ts=2024-04-14T13:55:53.836Z caller=node_exporter.go:115 level=info collector=netclass 2024-04-14T13:55:53.899 INFO:journalctl@ceph.node-exporter.b.smithi122.stdout:Apr 14 13:55:53 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-node-exporter-b[41167]: ts=2024-04-14T13:55:53.836Z caller=node_exporter.go:115 level=info collector=netdev 2024-04-14T13:55:53.899 INFO:journalctl@ceph.node-exporter.b.smithi122.stdout:Apr 14 13:55:53 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-node-exporter-b[41167]: ts=2024-04-14T13:55:53.836Z caller=node_exporter.go:115 level=info collector=netstat 2024-04-14T13:55:53.899 INFO:journalctl@ceph.node-exporter.b.smithi122.stdout:Apr 14 13:55:53 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-node-exporter-b[41167]: ts=2024-04-14T13:55:53.836Z caller=node_exporter.go:115 level=info collector=nfs 2024-04-14T13:55:53.899 INFO:journalctl@ceph.node-exporter.b.smithi122.stdout:Apr 14 13:55:53 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-node-exporter-b[41167]: ts=2024-04-14T13:55:53.836Z caller=node_exporter.go:115 level=info collector=nfsd 2024-04-14T13:55:53.899 INFO:journalctl@ceph.node-exporter.b.smithi122.stdout:Apr 14 13:55:53 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-node-exporter-b[41167]: ts=2024-04-14T13:55:53.836Z caller=node_exporter.go:115 level=info collector=nvme 2024-04-14T13:55:53.899 INFO:journalctl@ceph.node-exporter.b.smithi122.stdout:Apr 14 13:55:53 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-node-exporter-b[41167]: ts=2024-04-14T13:55:53.836Z caller=node_exporter.go:115 level=info collector=os 2024-04-14T13:55:53.899 INFO:journalctl@ceph.node-exporter.b.smithi122.stdout:Apr 14 13:55:53 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-node-exporter-b[41167]: ts=2024-04-14T13:55:53.836Z caller=node_exporter.go:115 level=info collector=powersupplyclass 2024-04-14T13:55:53.900 INFO:journalctl@ceph.node-exporter.b.smithi122.stdout:Apr 14 13:55:53 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-node-exporter-b[41167]: ts=2024-04-14T13:55:53.836Z caller=node_exporter.go:115 level=info collector=pressure 2024-04-14T13:55:53.900 INFO:journalctl@ceph.node-exporter.b.smithi122.stdout:Apr 14 13:55:53 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-node-exporter-b[41167]: ts=2024-04-14T13:55:53.836Z caller=node_exporter.go:115 level=info collector=rapl 2024-04-14T13:55:53.900 INFO:journalctl@ceph.node-exporter.b.smithi122.stdout:Apr 14 13:55:53 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-node-exporter-b[41167]: ts=2024-04-14T13:55:53.836Z caller=node_exporter.go:115 level=info collector=schedstat 2024-04-14T13:55:53.900 INFO:journalctl@ceph.node-exporter.b.smithi122.stdout:Apr 14 13:55:53 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-node-exporter-b[41167]: ts=2024-04-14T13:55:53.836Z caller=node_exporter.go:115 level=info collector=sockstat 2024-04-14T13:55:53.900 INFO:journalctl@ceph.node-exporter.b.smithi122.stdout:Apr 14 13:55:53 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-node-exporter-b[41167]: ts=2024-04-14T13:55:53.836Z caller=node_exporter.go:115 level=info collector=softnet 2024-04-14T13:55:53.900 INFO:journalctl@ceph.node-exporter.b.smithi122.stdout:Apr 14 13:55:53 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-node-exporter-b[41167]: ts=2024-04-14T13:55:53.836Z caller=node_exporter.go:115 level=info collector=stat 2024-04-14T13:55:53.900 INFO:journalctl@ceph.node-exporter.b.smithi122.stdout:Apr 14 13:55:53 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-node-exporter-b[41167]: ts=2024-04-14T13:55:53.836Z caller=node_exporter.go:115 level=info collector=tapestats 2024-04-14T13:55:53.900 INFO:journalctl@ceph.node-exporter.b.smithi122.stdout:Apr 14 13:55:53 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-node-exporter-b[41167]: ts=2024-04-14T13:55:53.836Z caller=node_exporter.go:115 level=info collector=textfile 2024-04-14T13:55:53.900 INFO:journalctl@ceph.node-exporter.b.smithi122.stdout:Apr 14 13:55:53 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-node-exporter-b[41167]: ts=2024-04-14T13:55:53.836Z caller=node_exporter.go:115 level=info collector=thermal_zone 2024-04-14T13:55:53.900 INFO:journalctl@ceph.node-exporter.b.smithi122.stdout:Apr 14 13:55:53 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-node-exporter-b[41167]: ts=2024-04-14T13:55:53.836Z caller=node_exporter.go:115 level=info collector=time 2024-04-14T13:55:53.900 INFO:journalctl@ceph.node-exporter.b.smithi122.stdout:Apr 14 13:55:53 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-node-exporter-b[41167]: ts=2024-04-14T13:55:53.836Z caller=node_exporter.go:115 level=info collector=udp_queues 2024-04-14T13:55:53.900 INFO:journalctl@ceph.node-exporter.b.smithi122.stdout:Apr 14 13:55:53 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-node-exporter-b[41167]: ts=2024-04-14T13:55:53.836Z caller=node_exporter.go:115 level=info collector=uname 2024-04-14T13:55:53.900 INFO:journalctl@ceph.node-exporter.b.smithi122.stdout:Apr 14 13:55:53 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-node-exporter-b[41167]: ts=2024-04-14T13:55:53.836Z caller=node_exporter.go:115 level=info collector=vmstat 2024-04-14T13:55:53.901 INFO:journalctl@ceph.node-exporter.b.smithi122.stdout:Apr 14 13:55:53 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-node-exporter-b[41167]: ts=2024-04-14T13:55:53.836Z caller=node_exporter.go:115 level=info collector=xfs 2024-04-14T13:55:53.901 INFO:journalctl@ceph.node-exporter.b.smithi122.stdout:Apr 14 13:55:53 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-node-exporter-b[41167]: ts=2024-04-14T13:55:53.836Z caller=node_exporter.go:115 level=info collector=zfs 2024-04-14T13:55:53.901 INFO:journalctl@ceph.node-exporter.b.smithi122.stdout:Apr 14 13:55:53 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-node-exporter-b[41167]: ts=2024-04-14T13:55:53.836Z caller=node_exporter.go:199 level=info msg="Listening on" address=:9100 2024-04-14T13:55:53.901 INFO:journalctl@ceph.node-exporter.b.smithi122.stdout:Apr 14 13:55:53 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-node-exporter-b[41167]: ts=2024-04-14T13:55:53.836Z caller=tls_config.go:195 level=info msg="TLS is disabled." http2=false 2024-04-14T13:55:53.901 INFO:journalctl@ceph.node-exporter.b.smithi122.stdout:Apr 14 13:55:53 smithi122 bash[41105]: e57f3d5931fbce91c3bf4d2422435c6b3ea9b89efe9f0b019b7925d0cac49251 2024-04-14T13:55:54.345 INFO:journalctl@ceph.node-exporter.b.smithi122.stdout:Apr 14 13:55:53 smithi122 systemd[1]: Started Ceph node-exporter.b for dc326fc2-fa65-11ee-bc8e-c7b262605968. 2024-04-14T13:55:55.269 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:55:54 smithi069 ceph-mon[24818]: from='mgr.14448 172.21.15.69:0/453473650' entity='mgr.y' 2024-04-14T13:55:55.270 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:55:54 smithi069 ceph-mon[24818]: from='mgr.14448 172.21.15.69:0/453473650' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-14T13:55:55.270 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:55:54 smithi069 ceph-mon[24818]: from='mgr.14448 172.21.15.69:0/453473650' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T13:55:55.270 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:55:54 smithi069 ceph-mon[24818]: from='mgr.14448 172.21.15.69:0/453473650' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-14T13:55:55.270 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:55:54 smithi069 ceph-mon[24818]: pgmap v18: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-14T13:55:55.270 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:55:54 smithi069 ceph-mon[24818]: from='mgr.14448 172.21.15.69:0/453473650' entity='mgr.y' 2024-04-14T13:55:55.270 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:55:54 smithi069 ceph-mon[29472]: from='mgr.14448 172.21.15.69:0/453473650' entity='mgr.y' 2024-04-14T13:55:55.270 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:55:54 smithi069 ceph-mon[29472]: from='mgr.14448 172.21.15.69:0/453473650' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-14T13:55:55.270 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:55:54 smithi069 ceph-mon[29472]: from='mgr.14448 172.21.15.69:0/453473650' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T13:55:55.270 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:55:54 smithi069 ceph-mon[29472]: from='mgr.14448 172.21.15.69:0/453473650' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-14T13:55:55.271 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:55:54 smithi069 ceph-mon[29472]: pgmap v18: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-14T13:55:55.271 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:55:54 smithi069 ceph-mon[29472]: from='mgr.14448 172.21.15.69:0/453473650' entity='mgr.y' 2024-04-14T13:55:55.345 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:55:54 smithi122 ceph-mon[25721]: from='mgr.14448 172.21.15.69:0/453473650' entity='mgr.y' 2024-04-14T13:55:55.345 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:55:54 smithi122 ceph-mon[25721]: from='mgr.14448 172.21.15.69:0/453473650' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-14T13:55:55.345 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:55:54 smithi122 ceph-mon[25721]: from='mgr.14448 172.21.15.69:0/453473650' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T13:55:55.346 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:55:54 smithi122 ceph-mon[25721]: from='mgr.14448 172.21.15.69:0/453473650' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-14T13:55:55.346 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:55:54 smithi122 ceph-mon[25721]: pgmap v18: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-14T13:55:55.346 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:55:54 smithi122 ceph-mon[25721]: from='mgr.14448 172.21.15.69:0/453473650' entity='mgr.y' 2024-04-14T13:55:57.017 INFO:teuthology.orchestra.run.smithi069.stdout:128849018905 2024-04-14T13:55:57.017 DEBUG:teuthology.orchestra.run.smithi069:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid dc326fc2-fa65-11ee-bc8e-c7b262605968 -- ceph osd last-stat-seq osd.4 2024-04-14T13:55:57.551 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:55:57 smithi069 ceph-mon[24818]: pgmap v19: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-14T13:55:57.551 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:55:57 smithi069 ceph-mon[24818]: from='mgr.14448 172.21.15.69:0/453473650' entity='mgr.y' 2024-04-14T13:55:57.551 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:55:57 smithi069 ceph-mon[29472]: pgmap v19: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-14T13:55:57.551 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:55:57 smithi069 ceph-mon[29472]: from='mgr.14448 172.21.15.69:0/453473650' entity='mgr.y' 2024-04-14T13:55:57.595 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:55:57 smithi122 ceph-mon[25721]: pgmap v19: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-14T13:55:57.595 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:55:57 smithi122 ceph-mon[25721]: from='mgr.14448 172.21.15.69:0/453473650' entity='mgr.y' 2024-04-14T13:55:57.595 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Apr 14 13:55:57 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-x[26913]: ::ffff:172.21.15.122 - - [14/Apr/2024:13:55:57] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-04-14T13:55:58.761 INFO:teuthology.orchestra.run.smithi069.stdout:55834574888 2024-04-14T13:55:58.762 DEBUG:teuthology.orchestra.run.smithi069:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid dc326fc2-fa65-11ee-bc8e-c7b262605968 -- ceph osd last-stat-seq osd.1 2024-04-14T13:55:59.744 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:55:59 smithi069 ceph-mon[24818]: pgmap v20: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-14T13:55:59.744 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 14 13:55:59 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-y[25054]: ::ffff:172.21.15.122 - - [14/Apr/2024:13:55:59] "GET /metrics HTTP/1.1" 200 191500 "" "Prometheus/2.33.4" 2024-04-14T13:55:59.744 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:55:59 smithi069 ceph-mon[29472]: pgmap v20: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-14T13:55:59.846 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:55:59 smithi122 ceph-mon[25721]: pgmap v20: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-14T13:56:00.698 INFO:teuthology.orchestra.run.smithi069.stdout:163208757270 2024-04-14T13:56:00.698 DEBUG:teuthology.orchestra.run.smithi069:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid dc326fc2-fa65-11ee-bc8e-c7b262605968 -- ceph osd last-stat-seq osd.5 2024-04-14T13:56:01.567 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:56:01 smithi069 ceph-mon[24818]: pgmap v21: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-14T13:56:01.567 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:56:01 smithi069 ceph-mon[29472]: pgmap v21: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-14T13:56:01.595 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:56:01 smithi122 ceph-mon[25721]: pgmap v21: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-14T13:56:02.921 INFO:teuthology.orchestra.run.smithi069.stderr:Inferring config /var/lib/ceph/dc326fc2-fa65-11ee-bc8e-c7b262605968/mon.a/config 2024-04-14T13:56:03.112 INFO:teuthology.orchestra.run.smithi069.stderr:Inferring config /var/lib/ceph/dc326fc2-fa65-11ee-bc8e-c7b262605968/mon.a/config 2024-04-14T13:56:03.114 INFO:teuthology.orchestra.run.smithi069.stderr:Inferring config /var/lib/ceph/dc326fc2-fa65-11ee-bc8e-c7b262605968/mon.a/config 2024-04-14T13:56:03.567 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:56:03 smithi069 ceph-mon[24818]: pgmap v22: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-14T13:56:03.567 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:56:03 smithi069 ceph-mon[29472]: pgmap v22: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-14T13:56:03.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:56:03 smithi122 ceph-mon[25721]: pgmap v22: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-14T13:56:05.484 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:56:05 smithi069 ceph-mon[24818]: pgmap v23: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-14T13:56:05.484 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:56:05 smithi069 ceph-mon[29472]: pgmap v23: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-14T13:56:05.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:56:05 smithi122 ceph-mon[25721]: pgmap v23: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-14T13:56:06.354 INFO:teuthology.orchestra.run.smithi069.stdout:107374182430 2024-04-14T13:56:06.355 DEBUG:teuthology.orchestra.run.smithi069:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid dc326fc2-fa65-11ee-bc8e-c7b262605968 -- ceph osd last-stat-seq osd.3 2024-04-14T13:56:07.145 INFO:teuthology.orchestra.run.smithi069.stdout:231928233996 2024-04-14T13:56:07.146 DEBUG:teuthology.orchestra.run.smithi069:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid dc326fc2-fa65-11ee-bc8e-c7b262605968 -- ceph osd last-stat-seq osd.7 2024-04-14T13:56:07.595 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:56:07 smithi122 ceph-mon[25721]: pgmap v24: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-14T13:56:07.596 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Apr 14 13:56:07 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-x[26913]: ::ffff:172.21.15.122 - - [14/Apr/2024:13:56:07] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-04-14T13:56:07.683 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:56:07 smithi069 ceph-mon[24818]: pgmap v24: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-14T13:56:07.683 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:56:07 smithi069 ceph-mon[29472]: pgmap v24: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-14T13:56:07.704 INFO:teuthology.orchestra.run.smithi069.stdout:77309411362 2024-04-14T13:56:07.704 DEBUG:teuthology.orchestra.run.smithi069:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid dc326fc2-fa65-11ee-bc8e-c7b262605968 -- ceph osd last-stat-seq osd.2 2024-04-14T13:56:09.817 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:56:09 smithi069 ceph-mon[24818]: pgmap v25: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-14T13:56:09.817 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 14 13:56:09 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-y[25054]: ::ffff:172.21.15.122 - - [14/Apr/2024:13:56:09] "GET /metrics HTTP/1.1" 200 191497 "" "Prometheus/2.33.4" 2024-04-14T13:56:09.818 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:56:09 smithi069 ceph-mon[29472]: pgmap v25: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-14T13:56:09.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:56:09 smithi122 ceph-mon[25721]: pgmap v25: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-14T13:56:10.078 INFO:teuthology.orchestra.run.smithi069.stdout:34359738413 2024-04-14T13:56:10.078 DEBUG:teuthology.orchestra.run.smithi069:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid dc326fc2-fa65-11ee-bc8e-c7b262605968 -- ceph osd last-stat-seq osd.0 2024-04-14T13:56:10.127 INFO:teuthology.orchestra.run.smithi069.stdout:163208757272 2024-04-14T13:56:10.725 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:56:10 smithi069 ceph-mon[24818]: from='client.? 172.21.15.69:0/4265694552' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-04-14T13:56:10.725 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:56:10 smithi069 ceph-mon[29472]: from='client.? 172.21.15.69:0/4265694552' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-04-14T13:56:10.822 INFO:teuthology.orchestra.run.smithi069.stdout:197568495632 2024-04-14T13:56:10.822 DEBUG:teuthology.orchestra.run.smithi069:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid dc326fc2-fa65-11ee-bc8e-c7b262605968 -- ceph osd last-stat-seq osd.6 2024-04-14T13:56:10.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:56:10 smithi122 ceph-mon[25721]: from='client.? 172.21.15.69:0/4265694552' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-04-14T13:56:11.567 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:56:11 smithi069 ceph-mon[24818]: pgmap v26: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-14T13:56:11.568 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:56:11 smithi069 ceph-mon[29472]: pgmap v26: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-14T13:56:11.785 INFO:tasks.cephadm.ceph_manager.ceph:need seq 163208757270 got 163208757272 for osd.5 2024-04-14T13:56:11.786 DEBUG:teuthology.parallel:result is None 2024-04-14T13:56:11.791 INFO:teuthology.orchestra.run.smithi069.stderr:Inferring config /var/lib/ceph/dc326fc2-fa65-11ee-bc8e-c7b262605968/mon.a/config 2024-04-14T13:56:11.792 INFO:teuthology.orchestra.run.smithi069.stderr:Inferring config /var/lib/ceph/dc326fc2-fa65-11ee-bc8e-c7b262605968/mon.a/config 2024-04-14T13:56:11.795 INFO:teuthology.orchestra.run.smithi069.stderr:Inferring config /var/lib/ceph/dc326fc2-fa65-11ee-bc8e-c7b262605968/mon.a/config 2024-04-14T13:56:11.795 INFO:teuthology.orchestra.run.smithi069.stderr:Inferring config /var/lib/ceph/dc326fc2-fa65-11ee-bc8e-c7b262605968/mon.a/config 2024-04-14T13:56:11.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:56:11 smithi122 ceph-mon[25721]: pgmap v26: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-14T13:56:13.725 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:56:13 smithi069 ceph-mon[24818]: pgmap v27: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-14T13:56:13.725 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:56:13 smithi069 ceph-mon[29472]: pgmap v27: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-14T13:56:13.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:56:13 smithi122 ceph-mon[25721]: pgmap v27: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-14T13:56:14.741 INFO:teuthology.orchestra.run.smithi069.stdout:128849018909 2024-04-14T13:56:15.725 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:56:15 smithi069 ceph-mon[24818]: pgmap v28: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-14T13:56:15.726 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:56:15 smithi069 ceph-mon[24818]: from='client.? 172.21.15.69:0/1503776624' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-04-14T13:56:15.726 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:56:15 smithi069 ceph-mon[24818]: from='client.? 172.21.15.69:0/1568292493' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-04-14T13:56:15.726 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:56:15 smithi069 ceph-mon[29472]: pgmap v28: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-14T13:56:15.726 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:56:15 smithi069 ceph-mon[29472]: from='client.? 172.21.15.69:0/1503776624' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-04-14T13:56:15.726 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:56:15 smithi069 ceph-mon[29472]: from='client.? 172.21.15.69:0/1568292493' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-04-14T13:56:15.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:56:15 smithi122 ceph-mon[25721]: pgmap v28: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-14T13:56:15.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:56:15 smithi122 ceph-mon[25721]: from='client.? 172.21.15.69:0/1503776624' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-04-14T13:56:15.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:56:15 smithi122 ceph-mon[25721]: from='client.? 172.21.15.69:0/1568292493' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-04-14T13:56:16.899 INFO:tasks.cephadm.ceph_manager.ceph:need seq 128849018905 got 128849018909 for osd.4 2024-04-14T13:56:16.900 DEBUG:teuthology.parallel:result is None 2024-04-14T13:56:17.533 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:56:17 smithi069 ceph-mon[24818]: pgmap v29: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-14T13:56:17.533 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:56:17 smithi069 ceph-mon[29472]: pgmap v29: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-14T13:56:17.595 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:56:17 smithi122 ceph-mon[25721]: pgmap v29: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-14T13:56:17.596 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Apr 14 13:56:17 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-x[26913]: ::ffff:172.21.15.122 - - [14/Apr/2024:13:56:17] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-04-14T13:56:17.615 INFO:teuthology.orchestra.run.smithi069.stdout:107374182433 2024-04-14T13:56:18.160 INFO:teuthology.orchestra.run.smithi069.stdout:231928233999 2024-04-14T13:56:18.465 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:56:18 smithi069 ceph-mon[29472]: from='client.? 172.21.15.69:0/3238951172' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-04-14T13:56:18.465 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:56:18 smithi069 ceph-mon[29472]: from='client.? 172.21.15.69:0/3992414231' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-04-14T13:56:18.465 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:56:18 smithi069 ceph-mon[24818]: from='client.? 172.21.15.69:0/3238951172' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-04-14T13:56:18.465 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:56:18 smithi069 ceph-mon[24818]: from='client.? 172.21.15.69:0/3992414231' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-04-14T13:56:18.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:56:18 smithi122 ceph-mon[25721]: from='client.? 172.21.15.69:0/3238951172' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-04-14T13:56:18.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:56:18 smithi122 ceph-mon[25721]: from='client.? 172.21.15.69:0/3992414231' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-04-14T13:56:18.883 INFO:tasks.cephadm.ceph_manager.ceph:need seq 107374182430 got 107374182433 for osd.3 2024-04-14T13:56:18.883 DEBUG:teuthology.parallel:result is None 2024-04-14T13:56:18.950 INFO:teuthology.orchestra.run.smithi069.stdout:55834574891 2024-04-14T13:56:19.107 INFO:teuthology.orchestra.run.smithi069.stderr:Inferring config /var/lib/ceph/dc326fc2-fa65-11ee-bc8e-c7b262605968/mon.a/config 2024-04-14T13:56:19.513 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:56:19 smithi069 ceph-mon[29472]: pgmap v30: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-14T13:56:19.513 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:56:19 smithi069 ceph-mon[24818]: pgmap v30: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-14T13:56:19.538 INFO:tasks.cephadm.ceph_manager.ceph:need seq 55834574888 got 55834574891 for osd.1 2024-04-14T13:56:19.539 DEBUG:teuthology.parallel:result is None 2024-04-14T13:56:19.783 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 14 13:56:19 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-y[25054]: ::ffff:172.21.15.122 - - [14/Apr/2024:13:56:19] "GET /metrics HTTP/1.1" 200 191497 "" "Prometheus/2.33.4" 2024-04-14T13:56:19.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:56:19 smithi122 ceph-mon[25721]: pgmap v30: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-14T13:56:20.803 INFO:tasks.cephadm.ceph_manager.ceph:need seq 231928233996 got 231928233999 for osd.7 2024-04-14T13:56:20.803 DEBUG:teuthology.parallel:result is None 2024-04-14T13:56:20.831 INFO:teuthology.orchestra.run.smithi069.stdout:77309411366 2024-04-14T13:56:21.437 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:56:21 smithi069 ceph-mon[24818]: pgmap v31: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-14T13:56:21.437 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:56:21 smithi069 ceph-mon[24818]: from='client.? 172.21.15.69:0/4076163878' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-04-14T13:56:21.437 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:56:21 smithi069 ceph-mon[29472]: pgmap v31: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-14T13:56:21.437 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:56:21 smithi069 ceph-mon[29472]: from='client.? 172.21.15.69:0/4076163878' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-04-14T13:56:21.595 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:56:21 smithi122 ceph-mon[25721]: pgmap v31: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-14T13:56:21.595 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:56:21 smithi122 ceph-mon[25721]: from='client.? 172.21.15.69:0/4076163878' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-04-14T13:56:22.227 INFO:teuthology.orchestra.run.smithi069.stdout:197568495635 2024-04-14T13:56:22.559 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:56:22 smithi069 ceph-mon[24818]: from='client.? 172.21.15.69:0/1175476369' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-04-14T13:56:22.559 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:56:22 smithi069 ceph-mon[29472]: from='client.? 172.21.15.69:0/1175476369' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-04-14T13:56:22.595 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:56:22 smithi122 ceph-mon[25721]: from='client.? 172.21.15.69:0/1175476369' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-04-14T13:56:22.838 INFO:teuthology.orchestra.run.smithi069.stdout:34359738416 2024-04-14T13:56:23.277 INFO:tasks.cephadm.ceph_manager.ceph:need seq 77309411362 got 77309411366 for osd.2 2024-04-14T13:56:23.277 DEBUG:teuthology.parallel:result is None 2024-04-14T13:56:23.484 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:56:23 smithi069 ceph-mon[24818]: pgmap v32: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-14T13:56:23.484 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:56:23 smithi069 ceph-mon[24818]: from='client.? 172.21.15.69:0/2928724448' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-04-14T13:56:23.484 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:56:23 smithi069 ceph-mon[29472]: pgmap v32: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-14T13:56:23.484 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:56:23 smithi069 ceph-mon[29472]: from='client.? 172.21.15.69:0/2928724448' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-04-14T13:56:23.595 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:56:23 smithi122 ceph-mon[25721]: pgmap v32: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-14T13:56:23.595 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:56:23 smithi122 ceph-mon[25721]: from='client.? 172.21.15.69:0/2928724448' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-04-14T13:56:23.980 INFO:tasks.cephadm.ceph_manager.ceph:need seq 34359738413 got 34359738416 for osd.0 2024-04-14T13:56:23.980 DEBUG:teuthology.parallel:result is None 2024-04-14T13:56:24.324 INFO:tasks.cephadm.ceph_manager.ceph:need seq 197568495632 got 197568495635 for osd.6 2024-04-14T13:56:24.324 DEBUG:teuthology.parallel:result is None 2024-04-14T13:56:24.324 INFO:tasks.cephadm.ceph_manager.ceph:waiting for clean 2024-04-14T13:56:24.325 DEBUG:teuthology.orchestra.run.smithi069:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid dc326fc2-fa65-11ee-bc8e-c7b262605968 -- ceph pg dump --format=json 2024-04-14T13:56:24.567 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:56:24 smithi069 ceph-mon[24818]: from='mgr.14448 172.21.15.69:0/453473650' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-04-14T13:56:24.567 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:56:24 smithi069 ceph-mon[24818]: from='mgr.14448 172.21.15.69:0/453473650' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-04-14T13:56:24.567 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:56:24 smithi069 ceph-mon[29472]: from='mgr.14448 172.21.15.69:0/453473650' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-04-14T13:56:24.568 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:56:24 smithi069 ceph-mon[29472]: from='mgr.14448 172.21.15.69:0/453473650' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-04-14T13:56:24.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:56:24 smithi122 ceph-mon[25721]: from='mgr.14448 172.21.15.69:0/453473650' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-04-14T13:56:24.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:56:24 smithi122 ceph-mon[25721]: from='mgr.14448 172.21.15.69:0/453473650' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-04-14T13:56:25.110 INFO:teuthology.orchestra.run.smithi069.stderr:Inferring config /var/lib/ceph/dc326fc2-fa65-11ee-bc8e-c7b262605968/mon.a/config 2024-04-14T13:56:25.769 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:56:25 smithi069 ceph-mon[24818]: pgmap v33: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-14T13:56:25.769 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:56:25 smithi069 ceph-mon[24818]: from='mgr.14448 172.21.15.69:0/453473650' entity='mgr.y' 2024-04-14T13:56:25.769 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:56:25 smithi069 ceph-mon[24818]: from='mgr.14448 172.21.15.69:0/453473650' entity='mgr.y' 2024-04-14T13:56:25.769 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:56:25 smithi069 ceph-mon[29472]: pgmap v33: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-14T13:56:25.769 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:56:25 smithi069 ceph-mon[29472]: from='mgr.14448 172.21.15.69:0/453473650' entity='mgr.y' 2024-04-14T13:56:25.769 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:56:25 smithi069 ceph-mon[29472]: from='mgr.14448 172.21.15.69:0/453473650' entity='mgr.y' 2024-04-14T13:56:25.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:56:25 smithi122 ceph-mon[25721]: pgmap v33: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-14T13:56:25.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:56:25 smithi122 ceph-mon[25721]: from='mgr.14448 172.21.15.69:0/453473650' entity='mgr.y' 2024-04-14T13:56:25.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:56:25 smithi122 ceph-mon[25721]: from='mgr.14448 172.21.15.69:0/453473650' entity='mgr.y' 2024-04-14T13:56:26.817 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:56:26 smithi069 ceph-mon[24818]: Deploying daemon alertmanager.a on smithi069 2024-04-14T13:56:26.817 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:56:26 smithi069 ceph-mon[29472]: Deploying daemon alertmanager.a on smithi069 2024-04-14T13:56:26.824 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-14T13:56:26.826 INFO:teuthology.orchestra.run.smithi069.stderr:dumped all 2024-04-14T13:56:26.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:56:26 smithi122 ceph-mon[25721]: Deploying daemon alertmanager.a on smithi069 2024-04-14T13:56:27.595 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:56:27 smithi122 ceph-mon[25721]: pgmap v34: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-14T13:56:27.595 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Apr 14 13:56:27 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-x[26913]: ::ffff:172.21.15.122 - - [14/Apr/2024:13:56:27] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-04-14T13:56:27.631 INFO:teuthology.orchestra.run.smithi069.stdout:{"pg_ready":true,"pg_map":{"version":34,"stamp":"2024-04-14T13:56:26.329281+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1586784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":128,"ondisk_log_size":128,"up":3,"acting":3,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":3,"num_osds":8,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":749961216,"kb_used":48912,"kb_used_data":6096,"kb_used_omap":0,"kb_used_meta":42752,"kb_avail":749912304,"statfs":{"total":767960285184,"available":767910199296,"internally_reserved":0,"allocated":6242304,"data_stored":3488916,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":43778048},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002063"},"pg_stats":[{"pgid":"1.0","version":"59'128","reported_seq":266,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-14T13:55:24.391850+0000","last_change":"2024-04-14T13:55:13.183515+0000","last_active":"2024-04-14T13:55:24.391850+0000","last_peered":"2024-04-14T13:55:24.391850+0000","last_clean":"2024-04-14T13:55:24.391850+0000","last_became_active":"2024-04-14T13:55:13.183047+0000","last_became_peered":"2024-04-14T13:55:13.183047+0000","last_unstale":"2024-04-14T13:55:24.391850+0000","last_undegraded":"2024-04-14T13:55:24.391850+0000","last_fullsized":"2024-04-14T13:55:24.391850+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T13:53:16.148701+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T13:53:16.148701+0000","last_clean_scrub_stamp":"2024-04-14T13:53:16.148701+0000","objects_scrubbed":0,"log_size":128,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T00:37:23.035958+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1586784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1586784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1597440,"data_stored":1586784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":128,"ondisk_log_size":128,"up":3,"acting":3,"num_store_stats":3}],"osd_stats":[{"osd":7,"up_from":54,"seq":231928234001,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6088,"kb_used_data":1088,"kb_used_omap":0,"kb_used_meta":4992,"kb_avail":93739064,"statfs":{"total":95995035648,"available":95988801536,"internally_reserved":0,"allocated":1114112,"data_stored":766769,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5111808},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Apr 14 13:56:10 2024","interfaces":[{"interface":"back","average":{"1min":0.499,"5min":0.499,"15min":0.499},"min":{"1min":0.38200000000000001,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":0.62,"5min":0.62,"15min":0.62},"last":0.47099999999999997},{"interface":"front","average":{"1min":0.51100000000000001,"5min":0.51100000000000001,"15min":0.51100000000000001},"min":{"1min":0.33700000000000002,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.73999999999999999,"5min":0.73999999999999999,"15min":0.73999999999999999},"last":0.622}]},{"osd":1,"last update":"Sun Apr 14 13:56:10 2024","interfaces":[{"interface":"back","average":{"1min":0.497,"5min":0.497,"15min":0.497},"min":{"1min":0.36099999999999999,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":0.64200000000000002,"5min":0.64200000000000002,"15min":0.64200000000000002},"last":0.56200000000000006},{"interface":"front","average":{"1min":0.504,"5min":0.504,"15min":0.504},"min":{"1min":0.32700000000000001,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":0.76300000000000001,"5min":0.76300000000000001,"15min":0.76300000000000001},"last":0.54300000000000004}]},{"osd":2,"last update":"Sun Apr 14 13:56:10 2024","interfaces":[{"interface":"back","average":{"1min":0.51600000000000001,"5min":0.51600000000000001,"15min":0.51600000000000001},"min":{"1min":0.34300000000000003,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":0.73099999999999998,"5min":0.73099999999999998,"15min":0.73099999999999998},"last":0.58199999999999996},{"interface":"front","average":{"1min":0.55600000000000005,"5min":0.55600000000000005,"15min":0.55600000000000005},"min":{"1min":0.41099999999999998,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":0.63500000000000001,"5min":0.63500000000000001,"15min":0.63500000000000001},"last":0.51700000000000002}]},{"osd":3,"last update":"Sun Apr 14 13:56:10 2024","interfaces":[{"interface":"back","average":{"1min":0.51400000000000001,"5min":0.51400000000000001,"15min":0.51400000000000001},"min":{"1min":0.377,"5min":0.377,"15min":0.377},"max":{"1min":0.68100000000000005,"5min":0.68100000000000005,"15min":0.68100000000000005},"last":0.52800000000000002},{"interface":"front","average":{"1min":0.53800000000000003,"5min":0.53800000000000003,"15min":0.53800000000000003},"min":{"1min":0.41799999999999998,"5min":0.41799999999999998,"15min":0.41799999999999998},"max":{"1min":0.69899999999999995,"5min":0.69899999999999995,"15min":0.69899999999999995},"last":0.60699999999999998}]},{"osd":4,"last update":"Sun Apr 14 13:56:10 2024","interfaces":[{"interface":"back","average":{"1min":0.47499999999999998,"5min":0.47499999999999998,"15min":0.47499999999999998},"min":{"1min":0.29799999999999999,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":0.72399999999999998,"5min":0.72399999999999998,"15min":0.72399999999999998},"last":0.59299999999999997},{"interface":"front","average":{"1min":0.46899999999999997,"5min":0.46899999999999997,"15min":0.46899999999999997},"min":{"1min":0.32200000000000001,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":0.79400000000000004,"5min":0.79400000000000004,"15min":0.79400000000000004},"last":0.65300000000000002}]},{"osd":5,"last update":"Sun Apr 14 13:56:10 2024","interfaces":[{"interface":"back","average":{"1min":0.49299999999999999,"5min":0.49299999999999999,"15min":0.49299999999999999},"min":{"1min":0.35699999999999998,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":0.67200000000000004,"5min":0.67200000000000004,"15min":0.67200000000000004},"last":0.66700000000000004},{"interface":"front","average":{"1min":0.498,"5min":0.498,"15min":0.498},"min":{"1min":0.36099999999999999,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":0.80000000000000004,"5min":0.80000000000000004,"15min":0.80000000000000004},"last":0.70099999999999996}]},{"osd":6,"last update":"Sun Apr 14 13:56:10 2024","interfaces":[{"interface":"back","average":{"1min":0.51700000000000002,"5min":0.51700000000000002,"15min":0.51700000000000002},"min":{"1min":0.33400000000000002,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":0.76100000000000001,"5min":0.76100000000000001,"15min":0.76100000000000001},"last":0.67800000000000005},{"interface":"front","average":{"1min":0.50900000000000001,"5min":0.50900000000000001,"15min":0.50900000000000001},"min":{"1min":0.34899999999999998,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":0.76700000000000002,"5min":0.76700000000000002,"15min":0.76700000000000002},"last":0.68799999999999994}]}]},{"osd":6,"up_from":46,"seq":197568495637,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6148,"kb_used_data":1084,"kb_used_omap":0,"kb_used_meta":5056,"kb_avail":93739004,"statfs":{"total":95995035648,"available":95988740096,"internally_reserved":0,"allocated":1110016,"data_stored":766496,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5177344},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Apr 14 13:55:47 2024","interfaces":[{"interface":"back","average":{"1min":0.53300000000000003,"5min":0.53300000000000003,"15min":0.53300000000000003},"min":{"1min":0.312,"5min":0.312,"15min":0.312},"max":{"1min":0.74099999999999999,"5min":0.74099999999999999,"15min":0.74099999999999999},"last":0.41999999999999998},{"interface":"front","average":{"1min":0.54200000000000004,"5min":0.54200000000000004,"15min":0.54200000000000004},"min":{"1min":0.38400000000000001,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":0.76500000000000001,"5min":0.76500000000000001,"15min":0.76500000000000001},"last":0.372}]},{"osd":1,"last update":"Sun Apr 14 13:55:47 2024","interfaces":[{"interface":"back","average":{"1min":0.52600000000000002,"5min":0.52600000000000002,"15min":0.52600000000000002},"min":{"1min":0.34599999999999997,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":0.82499999999999996,"5min":0.82499999999999996,"15min":0.82499999999999996},"last":0.44500000000000001},{"interface":"front","average":{"1min":0.54500000000000004,"5min":0.54500000000000004,"15min":0.54500000000000004},"min":{"1min":0.375,"5min":0.375,"15min":0.375},"max":{"1min":0.77100000000000002,"5min":0.77100000000000002,"15min":0.77100000000000002},"last":0.47399999999999998}]},{"osd":2,"last update":"Sun Apr 14 13:55:47 2024","interfaces":[{"interface":"back","average":{"1min":0.52000000000000002,"5min":0.52000000000000002,"15min":0.52000000000000002},"min":{"1min":0.35699999999999998,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":0.77200000000000002,"5min":0.77200000000000002,"15min":0.77200000000000002},"last":0.50800000000000001},{"interface":"front","average":{"1min":0.54400000000000004,"5min":0.54400000000000004,"15min":0.54400000000000004},"min":{"1min":0.36599999999999999,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":0.76900000000000002,"5min":0.76900000000000002,"15min":0.76900000000000002},"last":0.45700000000000002}]},{"osd":3,"last update":"Sun Apr 14 13:55:47 2024","interfaces":[{"interface":"back","average":{"1min":0.55500000000000005,"5min":0.55500000000000005,"15min":0.55500000000000005},"min":{"1min":0.40300000000000002,"5min":0.40300000000000002,"15min":0.40300000000000002},"max":{"1min":0.84099999999999997,"5min":0.84099999999999997,"15min":0.84099999999999997},"last":0.36099999999999999},{"interface":"front","average":{"1min":0.54400000000000004,"5min":0.54400000000000004,"15min":0.54400000000000004},"min":{"1min":0.37,"5min":0.37,"15min":0.37},"max":{"1min":0.76100000000000001,"5min":0.76100000000000001,"15min":0.76100000000000001},"last":0.53400000000000003}]},{"osd":4,"last update":"Sun Apr 14 13:55:47 2024","interfaces":[{"interface":"back","average":{"1min":0.42199999999999999,"5min":0.42199999999999999,"15min":0.42199999999999999},"min":{"1min":0.26300000000000001,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":0.58699999999999997,"5min":0.58699999999999997,"15min":0.58699999999999997},"last":0.439},{"interface":"front","average":{"1min":0.42999999999999999,"5min":0.42999999999999999,"15min":0.42999999999999999},"min":{"1min":0.26700000000000002,"5min":0.26700000000000002,"15min":0.26700000000000002},"max":{"1min":0.74299999999999999,"5min":0.74299999999999999,"15min":0.74299999999999999},"last":0.46800000000000003}]},{"osd":5,"last update":"Sun Apr 14 13:55:47 2024","interfaces":[{"interface":"back","average":{"1min":0.42799999999999999,"5min":0.42799999999999999,"15min":0.42799999999999999},"min":{"1min":0.30499999999999999,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":0.63,"5min":0.63,"15min":0.63},"last":0.40600000000000003},{"interface":"front","average":{"1min":0.45900000000000002,"5min":0.45900000000000002,"15min":0.45900000000000002},"min":{"1min":0.29899999999999999,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":0.81200000000000006,"5min":0.81200000000000006,"15min":0.81200000000000006},"last":0.39800000000000002}]},{"osd":7,"last update":"Sun Apr 14 13:56:09 2024","interfaces":[{"interface":"back","average":{"1min":0.5,"5min":0.5,"15min":0.5},"min":{"1min":0.28999999999999998,"5min":0.28999999999999998,"15min":0.28999999999999998},"max":{"1min":0.76000000000000001,"5min":0.76000000000000001,"15min":0.76000000000000001},"last":0.48799999999999999},{"interface":"front","average":{"1min":0.51800000000000002,"5min":0.51800000000000002,"15min":0.51800000000000002},"min":{"1min":0.34999999999999998,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":0.747,"5min":0.747,"15min":0.747},"last":0.496}]}]},{"osd":1,"up_from":13,"seq":55834574894,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":6336,"kb_used_data":568,"kb_used_omap":0,"kb_used_meta":5760,"kb_avail":93738816,"statfs":{"total":95995035648,"available":95988547584,"internally_reserved":0,"allocated":581632,"data_stored":237841,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5898240},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Apr 14 13:55:56 2024","interfaces":[{"interface":"back","average":{"1min":0.38,"5min":0.36099999999999999,"15min":0.35899999999999999},"min":{"1min":0.26400000000000001,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":0.55700000000000005,"5min":0.55700000000000005,"15min":0.55700000000000005},"last":0.39000000000000001},{"interface":"front","average":{"1min":0.40200000000000002,"5min":0.38,"15min":0.36799999999999999},"min":{"1min":0.26700000000000002,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":0.63800000000000001,"5min":0.63800000000000001,"15min":0.63800000000000001},"last":0.33700000000000002}]},{"osd":2,"last update":"Sun Apr 14 13:56:21 2024","interfaces":[{"interface":"back","average":{"1min":0.377,"5min":0.38800000000000001,"15min":0.39000000000000001},"min":{"1min":0.25,"5min":0.25,"15min":0.25},"max":{"1min":0.61799999999999999,"5min":0.61799999999999999,"15min":0.61799999999999999},"last":0.28799999999999998},{"interface":"front","average":{"1min":0.35599999999999998,"5min":0.39400000000000002,"15min":0.40300000000000002},"min":{"1min":0.24399999999999999,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":0.52300000000000002,"5min":0.54900000000000004,"15min":0.54900000000000004},"last":0.37}]},{"osd":3,"last update":"Sun Apr 14 13:55:40 2024","interfaces":[{"interface":"back","average":{"1min":0.41499999999999998,"5min":0.40799999999999997,"15min":0.40699999999999997},"min":{"1min":0.26000000000000001,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":0.57299999999999995,"5min":0.57299999999999995,"15min":0.57299999999999995},"last":0.34499999999999997},{"interface":"front","average":{"1min":0.434,"5min":0.439,"15min":0.44},"min":{"1min":0.28899999999999998,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":0.70199999999999996,"5min":0.70199999999999996,"15min":0.70199999999999996},"last":0.35599999999999998}]},{"osd":4,"last update":"Sun Apr 14 13:56:02 2024","interfaces":[{"interface":"back","average":{"1min":0.52500000000000002,"5min":0.51700000000000002,"15min":0.51600000000000001},"min":{"1min":0.38600000000000001,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":0.67000000000000004,"5min":0.67800000000000005,"15min":0.67800000000000005},"last":0.57399999999999995},{"interface":"front","average":{"1min":0.53900000000000003,"5min":0.52200000000000002,"15min":0.51900000000000002},"min":{"1min":0.39800000000000002,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":0.77700000000000002,"5min":0.77700000000000002,"15min":0.77700000000000002},"last":0.67700000000000005}]},{"osd":5,"last update":"Sun Apr 14 13:55:23 2024","interfaces":[{"interface":"back","average":{"1min":0.53100000000000003,"5min":0.53100000000000003,"15min":0.53100000000000003},"min":{"1min":0.373,"5min":0.373,"15min":0.373},"max":{"1min":0.749,"5min":0.749,"15min":0.749},"last":0.55500000000000005},{"interface":"front","average":{"1min":0.54300000000000004,"5min":0.54300000000000004,"15min":0.54300000000000004},"min":{"1min":0.374,"5min":0.374,"15min":0.374},"max":{"1min":0.72299999999999998,"5min":0.72299999999999998,"15min":0.72299999999999998},"last":0.61599999999999999}]},{"osd":6,"last update":"Sun Apr 14 13:55:46 2024","interfaces":[{"interface":"back","average":{"1min":0.59099999999999997,"5min":0.59099999999999997,"15min":0.59099999999999997},"min":{"1min":0.442,"5min":0.442,"15min":0.442},"max":{"1min":0.86299999999999999,"5min":0.86299999999999999,"15min":0.86299999999999999},"last":0.71299999999999997},{"interface":"front","average":{"1min":0.59199999999999997,"5min":0.59199999999999997,"15min":0.59199999999999997},"min":{"1min":0.39400000000000002,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":0.81399999999999995,"5min":0.81399999999999995,"15min":0.81399999999999995},"last":0.59899999999999998}]},{"osd":7,"last update":"Sun Apr 14 13:56:09 2024","interfaces":[{"interface":"back","average":{"1min":0.60399999999999998,"5min":0.60399999999999998,"15min":0.60399999999999998},"min":{"1min":0.42499999999999999,"5min":0.42499999999999999,"15min":0.42499999999999999},"max":{"1min":0.82499999999999996,"5min":0.82499999999999996,"15min":0.82499999999999996},"last":0.67200000000000004},{"interface":"front","average":{"1min":0.60599999999999998,"5min":0.60599999999999998,"15min":0.60599999999999998},"min":{"1min":0.40100000000000002,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":0.88500000000000001,"5min":0.88500000000000001,"15min":0.88500000000000001},"last":0.70299999999999996}]}]},{"osd":0,"up_from":8,"seq":34359738418,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6920,"kb_used_data":1088,"kb_used_omap":0,"kb_used_meta":5824,"kb_avail":93738232,"statfs":{"total":95995035648,"available":95987949568,"internally_reserved":0,"allocated":1114112,"data_stored":766769,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5963776},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sun Apr 14 13:55:54 2024","interfaces":[{"interface":"back","average":{"1min":0.35599999999999998,"5min":0.34399999999999997,"15min":0.33000000000000002},"min":{"1min":0.23300000000000001,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":0.50600000000000001,"5min":0.58099999999999996,"15min":0.58099999999999996},"last":0.40500000000000003},{"interface":"front","average":{"1min":0.36399999999999999,"5min":0.34899999999999998,"15min":0.33500000000000002},"min":{"1min":0.23799999999999999,"5min":0.223,"15min":0.223},"max":{"1min":0.58299999999999996,"5min":0.61099999999999999,"15min":0.61099999999999999},"last":0.39500000000000002}]},{"osd":2,"last update":"Sun Apr 14 13:55:20 2024","interfaces":[{"interface":"back","average":{"1min":0.40899999999999997,"5min":0.433,"15min":0.437},"min":{"1min":0.28199999999999997,"5min":0.28199999999999997,"15min":0.28199999999999997},"max":{"1min":0.52600000000000002,"5min":0.64600000000000002,"15min":0.64600000000000002},"last":0.377},{"interface":"front","average":{"1min":0.40899999999999997,"5min":0.41699999999999998,"15min":0.41799999999999998},"min":{"1min":0.26900000000000002,"5min":0.23000000000000001,"15min":0.23000000000000001},"max":{"1min":0.55300000000000005,"5min":0.55300000000000005,"15min":0.55300000000000005},"last":0.41799999999999998}]},{"osd":3,"last update":"Sun Apr 14 13:55:38 2024","interfaces":[{"interface":"back","average":{"1min":0.42899999999999999,"5min":0.46200000000000002,"15min":0.46800000000000003},"min":{"1min":0.30099999999999999,"5min":0.30099999999999999,"15min":0.30099999999999999},"max":{"1min":0.56599999999999995,"5min":0.78400000000000003,"15min":0.78400000000000003},"last":0.44700000000000001},{"interface":"front","average":{"1min":0.46600000000000003,"5min":0.49399999999999999,"15min":0.499},"min":{"1min":0.32500000000000001,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":0.63600000000000001,"5min":0.68899999999999995,"15min":0.68899999999999995},"last":0.442}]},{"osd":4,"last update":"Sun Apr 14 13:56:04 2024","interfaces":[{"interface":"back","average":{"1min":0.53200000000000003,"5min":0.59799999999999998,"15min":0.60899999999999999},"min":{"1min":0.371,"5min":0.371,"15min":0.371},"max":{"1min":0.80200000000000005,"5min":0.84499999999999997,"15min":0.84499999999999997},"last":0.60099999999999998},{"interface":"front","average":{"1min":0.505,"5min":0.60399999999999998,"15min":0.62},"min":{"1min":0.372,"5min":0.372,"15min":0.372},"max":{"1min":0.82999999999999996,"5min":0.995,"15min":0.995},"last":0.61399999999999999}]},{"osd":5,"last update":"Sun Apr 14 13:55:23 2024","interfaces":[{"interface":"back","average":{"1min":0.58399999999999996,"5min":0.58399999999999996,"15min":0.58399999999999996},"min":{"1min":0.38400000000000001,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":0.90600000000000003,"5min":0.90600000000000003,"15min":0.90600000000000003},"last":0.58299999999999996},{"interface":"front","average":{"1min":0.59399999999999997,"5min":0.59399999999999997,"15min":0.59399999999999997},"min":{"1min":0.376,"5min":0.376,"15min":0.376},"max":{"1min":0.875,"5min":0.875,"15min":0.875},"last":0.63200000000000001}]},{"osd":6,"last update":"Sun Apr 14 13:55:50 2024","interfaces":[{"interface":"back","average":{"1min":0.59299999999999997,"5min":0.59299999999999997,"15min":0.59299999999999997},"min":{"1min":0.40400000000000003,"5min":0.40400000000000003,"15min":0.40400000000000003},"max":{"1min":0.93500000000000005,"5min":0.93500000000000005,"15min":0.93500000000000005},"last":0.70399999999999996},{"interface":"front","average":{"1min":0.60499999999999998,"5min":0.60499999999999998,"15min":0.60499999999999998},"min":{"1min":0.39700000000000002,"5min":0.39700000000000002,"15min":0.39700000000000002},"max":{"1min":0.95099999999999996,"5min":0.95099999999999996,"15min":0.95099999999999996},"last":0.70999999999999996}]},{"osd":7,"last update":"Sun Apr 14 13:56:11 2024","interfaces":[{"interface":"back","average":{"1min":0.56299999999999994,"5min":0.56299999999999994,"15min":0.56299999999999994},"min":{"1min":0.38200000000000001,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":0.82699999999999996,"5min":0.82699999999999996,"15min":0.82699999999999996},"last":0.64200000000000002},{"interface":"front","average":{"1min":0.56499999999999995,"5min":0.56499999999999995,"15min":0.56499999999999995},"min":{"1min":0.46800000000000003,"5min":0.46800000000000003,"15min":0.46800000000000003},"max":{"1min":0.78000000000000003,"5min":0.78000000000000003,"15min":0.78000000000000003},"last":0.64800000000000002}]}]},{"osd":2,"up_from":18,"seq":77309411368,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":6272,"kb_used_data":568,"kb_used_omap":0,"kb_used_meta":5696,"kb_avail":93738880,"statfs":{"total":95995035648,"available":95988613120,"internally_reserved":0,"allocated":581632,"data_stored":237841,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5832704},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Apr 14 13:56:21 2024","interfaces":[{"interface":"back","average":{"1min":0.36899999999999999,"5min":0.39100000000000001,"15min":0.39300000000000002},"min":{"1min":0.25800000000000001,"5min":0.245,"15min":0.245},"max":{"1min":0.59299999999999997,"5min":0.59999999999999998,"15min":0.59999999999999998},"last":0.29199999999999998},{"interface":"front","average":{"1min":0.377,"5min":0.38300000000000001,"15min":0.38700000000000001},"min":{"1min":0.28499999999999998,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":0.56999999999999995,"5min":0.56999999999999995,"15min":0.56999999999999995},"last":0.34699999999999998}]},{"osd":1,"last update":"Sun Apr 14 13:56:21 2024","interfaces":[{"interface":"back","average":{"1min":0.376,"5min":0.39600000000000002,"15min":0.40000000000000002},"min":{"1min":0.255,"5min":0.255,"15min":0.255},"max":{"1min":0.53900000000000003,"5min":0.59599999999999997,"15min":0.59599999999999997},"last":0.36399999999999999},{"interface":"front","average":{"1min":0.40500000000000003,"5min":0.42299999999999999,"15min":0.42699999999999999},"min":{"1min":0.29699999999999999,"5min":0.27600000000000002,"15min":0.27600000000000002},"max":{"1min":0.64900000000000002,"5min":0.64900000000000002,"15min":0.64900000000000002},"last":0.33900000000000002}]},{"osd":3,"last update":"Sun Apr 14 13:55:43 2024","interfaces":[{"interface":"back","average":{"1min":0.433,"5min":0.437,"15min":0.437},"min":{"1min":0.28899999999999998,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":0.61799999999999999,"5min":0.61799999999999999,"15min":0.61799999999999999},"last":0.40500000000000003},{"interface":"front","average":{"1min":0.44900000000000001,"5min":0.46700000000000003,"15min":0.46999999999999997},"min":{"1min":0.29799999999999999,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":0.73399999999999999,"5min":0.749,"15min":0.749},"last":0.38300000000000001}]},{"osd":4,"last update":"Sun Apr 14 13:56:00 2024","interfaces":[{"interface":"back","average":{"1min":1.361,"5min":0.72299999999999998,"15min":0.61699999999999999},"min":{"1min":0.41899999999999998,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":16.198,"5min":16.198,"15min":16.198},"last":0.629},{"interface":"front","average":{"1min":0.53200000000000003,"5min":1.2130000000000001,"15min":1.327},"min":{"1min":0.39500000000000002,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":0.82399999999999995,"5min":16.658999999999999,"15min":16.658999999999999},"last":0.44600000000000001}]},{"osd":5,"last update":"Sun Apr 14 13:55:24 2024","interfaces":[{"interface":"back","average":{"1min":0.89400000000000002,"5min":0.89400000000000002,"15min":0.89400000000000002},"min":{"1min":0.38500000000000001,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":7.4480000000000004,"5min":7.4480000000000004,"15min":7.4480000000000004},"last":0.55900000000000005},{"interface":"front","average":{"1min":0.61099999999999999,"5min":0.61099999999999999,"15min":0.61099999999999999},"min":{"1min":0.434,"5min":0.434,"15min":0.434},"max":{"1min":0.89400000000000002,"5min":0.89400000000000002,"15min":0.89400000000000002},"last":0.53200000000000003}]},{"osd":6,"last update":"Sun Apr 14 13:55:46 2024","interfaces":[{"interface":"back","average":{"1min":0.56699999999999995,"5min":0.56699999999999995,"15min":0.56699999999999995},"min":{"1min":0.39200000000000002,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":0.81799999999999995,"5min":0.81799999999999995,"15min":0.81799999999999995},"last":0.51700000000000002},{"interface":"front","average":{"1min":0.57399999999999995,"5min":0.57399999999999995,"15min":0.57399999999999995},"min":{"1min":0.40899999999999997,"5min":0.40899999999999997,"15min":0.40899999999999997},"max":{"1min":0.76400000000000001,"5min":0.76400000000000001,"15min":0.76400000000000001},"last":0.50800000000000001}]},{"osd":7,"last update":"Sun Apr 14 13:56:09 2024","interfaces":[{"interface":"back","average":{"1min":0.61399999999999999,"5min":0.61399999999999999,"15min":0.61399999999999999},"min":{"1min":0.38100000000000001,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":1.0049999999999999,"5min":1.0049999999999999,"15min":1.0049999999999999},"last":0.59799999999999998},{"interface":"front","average":{"1min":0.63300000000000001,"5min":0.63300000000000001,"15min":0.63300000000000001},"min":{"1min":0.39400000000000002,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":1.1499999999999999,"5min":1.1499999999999999,"15min":1.1499999999999999},"last":0.53700000000000003}]}]},{"osd":3,"up_from":25,"seq":107374182435,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":5696,"kb_used_data":568,"kb_used_omap":0,"kb_used_meta":5120,"kb_avail":93739456,"statfs":{"total":95995035648,"available":95989202944,"internally_reserved":0,"allocated":581632,"data_stored":237841,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5242880},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Apr 14 13:55:43 2024","interfaces":[{"interface":"back","average":{"1min":0.38700000000000001,"5min":0.39600000000000002,"15min":0.39800000000000002},"min":{"1min":0.27200000000000002,"5min":0.25700000000000001,"15min":0.25700000000000001},"max":{"1min":0.67200000000000004,"5min":0.67200000000000004,"15min":0.67200000000000004},"last":0.30399999999999999},{"interface":"front","average":{"1min":0.41699999999999998,"5min":0.42299999999999999,"15min":0.42399999999999999},"min":{"1min":0.249,"5min":0.249,"15min":0.249},"max":{"1min":0.82299999999999995,"5min":0.82299999999999995,"15min":0.82299999999999995},"last":0.372}]},{"osd":1,"last update":"Sun Apr 14 13:55:43 2024","interfaces":[{"interface":"back","average":{"1min":0.437,"5min":0.441,"15min":0.442},"min":{"1min":0.28799999999999998,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":0.62,"5min":0.62,"15min":0.62},"last":0.34100000000000003},{"interface":"front","average":{"1min":0.42799999999999999,"5min":0.433,"15min":0.434},"min":{"1min":0.23699999999999999,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":0.74199999999999999,"5min":0.74199999999999999,"15min":0.74199999999999999},"last":0.35899999999999999}]},{"osd":2,"last update":"Sun Apr 14 13:55:43 2024","interfaces":[{"interface":"back","average":{"1min":0.48499999999999999,"5min":0.48199999999999998,"15min":0.48199999999999998},"min":{"1min":0.34599999999999997,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":0.71399999999999997,"5min":0.71399999999999997,"15min":0.71399999999999997},"last":0.38600000000000001},{"interface":"front","average":{"1min":0.45200000000000001,"5min":0.47799999999999998,"15min":0.48199999999999998},"min":{"1min":0.29199999999999998,"5min":0.28999999999999998,"15min":0.28999999999999998},"max":{"1min":0.63800000000000001,"5min":0.63800000000000001,"15min":0.63800000000000001},"last":0.73999999999999999}]},{"osd":4,"last update":"Sun Apr 14 13:56:00 2024","interfaces":[{"interface":"back","average":{"1min":0.59799999999999998,"5min":0.58599999999999997,"15min":0.58399999999999996},"min":{"1min":0.41499999999999998,"5min":0.41499999999999998,"15min":0.41499999999999998},"max":{"1min":0.86099999999999999,"5min":0.86099999999999999,"15min":0.86099999999999999},"last":0.52400000000000002},{"interface":"front","average":{"1min":0.58899999999999997,"5min":0.56299999999999994,"15min":0.55900000000000005},"min":{"1min":0.38500000000000001,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":0.85199999999999998,"5min":0.85199999999999998,"15min":0.85199999999999998},"last":0.81699999999999995}]},{"osd":5,"last update":"Sun Apr 14 13:55:24 2024","interfaces":[{"interface":"back","average":{"1min":0.57499999999999996,"5min":0.57499999999999996,"15min":0.57499999999999996},"min":{"1min":0.40699999999999997,"5min":0.40699999999999997,"15min":0.40699999999999997},"max":{"1min":0.89000000000000001,"5min":0.89000000000000001,"15min":0.89000000000000001},"last":0.502},{"interface":"front","average":{"1min":0.57499999999999996,"5min":0.57499999999999996,"15min":0.57499999999999996},"min":{"1min":0.44800000000000001,"5min":0.44800000000000001,"15min":0.44800000000000001},"max":{"1min":0.78000000000000003,"5min":0.78000000000000003,"15min":0.78000000000000003},"last":0.80000000000000004}]},{"osd":6,"last update":"Sun Apr 14 13:55:48 2024","interfaces":[{"interface":"back","average":{"1min":0.623,"5min":0.623,"15min":0.623},"min":{"1min":0.41499999999999998,"5min":0.41499999999999998,"15min":0.41499999999999998},"max":{"1min":0.94399999999999995,"5min":0.94399999999999995,"15min":0.94399999999999995},"last":0.57199999999999995},{"interface":"front","average":{"1min":0.60599999999999998,"5min":0.60599999999999998,"15min":0.60599999999999998},"min":{"1min":0.34300000000000003,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":0.92300000000000004,"5min":0.92300000000000004,"15min":0.92300000000000004},"last":0.83099999999999996}]},{"osd":7,"last update":"Sun Apr 14 13:56:09 2024","interfaces":[{"interface":"back","average":{"1min":0.59099999999999997,"5min":0.59099999999999997,"15min":0.59099999999999997},"min":{"1min":0.40200000000000002,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":0.872,"5min":0.872,"15min":0.872},"last":0.77100000000000002},{"interface":"front","average":{"1min":0.63400000000000001,"5min":0.63400000000000001,"15min":0.63400000000000001},"min":{"1min":0.42199999999999999,"5min":0.42199999999999999,"15min":0.42199999999999999},"max":{"1min":0.92100000000000004,"5min":0.92100000000000004,"15min":0.92100000000000004},"last":0.63700000000000001}]}]},{"osd":4,"up_from":30,"seq":128849018912,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":5760,"kb_used_data":568,"kb_used_omap":0,"kb_used_meta":5184,"kb_avail":93739392,"statfs":{"total":95995035648,"available":95989137408,"internally_reserved":0,"allocated":581632,"data_stored":237841,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5308416},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Apr 14 13:56:01 2024","interfaces":[{"interface":"back","average":{"1min":0.495,"5min":0.50900000000000001,"15min":0.51100000000000001},"min":{"1min":0.22800000000000001,"5min":0.22800000000000001,"15min":0.22800000000000001},"max":{"1min":0.80300000000000005,"5min":0.80300000000000005,"15min":0.80300000000000005},"last":0.58199999999999996},{"interface":"front","average":{"1min":0.51700000000000002,"5min":0.54100000000000004,"15min":0.54500000000000004},"min":{"1min":0.29999999999999999,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":0.86699999999999999,"5min":0.86699999999999999,"15min":0.86699999999999999},"last":0.59799999999999998}]},{"osd":1,"last update":"Sun Apr 14 13:56:01 2024","interfaces":[{"interface":"back","average":{"1min":0.51500000000000001,"5min":0.52100000000000002,"15min":0.52200000000000002},"min":{"1min":0.34200000000000003,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.89400000000000002,"5min":0.89400000000000002,"15min":0.89400000000000002},"last":0.53000000000000003},{"interface":"front","average":{"1min":0.51300000000000001,"5min":0.52100000000000002,"15min":0.52200000000000002},"min":{"1min":0.35399999999999998,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":0.81100000000000005,"5min":0.81100000000000005,"15min":0.81100000000000005},"last":0.55100000000000005}]},{"osd":2,"last update":"Sun Apr 14 13:56:01 2024","interfaces":[{"interface":"back","average":{"1min":0.51800000000000002,"5min":0.54200000000000004,"15min":0.54600000000000004},"min":{"1min":0.34699999999999998,"5min":0.34699999999999998,"15min":0.34699999999999998},"max":{"1min":0.83099999999999996,"5min":0.83099999999999996,"15min":0.83099999999999996},"last":0.51200000000000001},{"interface":"front","average":{"1min":0.52900000000000003,"5min":0.54100000000000004,"15min":0.54300000000000004},"min":{"1min":0.32300000000000001,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":0.85299999999999998,"5min":0.85299999999999998,"15min":0.85299999999999998},"last":0.56799999999999995}]},{"osd":3,"last update":"Sun Apr 14 13:56:01 2024","interfaces":[{"interface":"back","average":{"1min":0.55300000000000005,"5min":0.56899999999999995,"15min":0.57199999999999995},"min":{"1min":0.373,"5min":0.373,"15min":0.373},"max":{"1min":0.90400000000000003,"5min":0.90400000000000003,"15min":0.90400000000000003},"last":0.62},{"interface":"front","average":{"1min":0.55700000000000005,"5min":0.56100000000000005,"15min":0.56200000000000006},"min":{"1min":0.38500000000000001,"5min":0.37,"15min":0.37},"max":{"1min":0.876,"5min":0.876,"15min":0.876},"last":0.437}]},{"osd":5,"last update":"Sun Apr 14 13:55:25 2024","interfaces":[{"interface":"back","average":{"1min":0.38800000000000001,"5min":0.38800000000000001,"15min":0.38800000000000001},"min":{"1min":0.255,"5min":0.255,"15min":0.255},"max":{"1min":0.52900000000000003,"5min":0.52900000000000003,"15min":0.52900000000000003},"last":0.65500000000000003},{"interface":"front","average":{"1min":0.40100000000000002,"5min":0.40100000000000002,"15min":0.40100000000000002},"min":{"1min":0.28000000000000003,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":0.59099999999999997,"5min":0.59099999999999997,"15min":0.59099999999999997},"last":0.47699999999999998}]},{"osd":6,"last update":"Sun Apr 14 13:55:47 2024","interfaces":[{"interface":"back","average":{"1min":0.47999999999999998,"5min":0.47999999999999998,"15min":0.47999999999999998},"min":{"1min":0.29699999999999999,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":0.93600000000000005,"5min":0.93600000000000005,"15min":0.93600000000000005},"last":0.66800000000000004},{"interface":"front","average":{"1min":0.49199999999999999,"5min":0.49199999999999999,"15min":0.49199999999999999},"min":{"1min":0.315,"5min":0.315,"15min":0.315},"max":{"1min":0.98799999999999999,"5min":0.98799999999999999,"15min":0.98799999999999999},"last":0.67800000000000005}]},{"osd":7,"last update":"Sun Apr 14 13:56:09 2024","interfaces":[{"interface":"back","average":{"1min":0.502,"5min":0.502,"15min":0.502},"min":{"1min":0.311,"5min":0.311,"15min":0.311},"max":{"1min":0.84899999999999998,"5min":0.84899999999999998,"15min":0.84899999999999998},"last":0.64400000000000002},{"interface":"front","average":{"1min":0.52500000000000002,"5min":0.52500000000000002,"15min":0.52500000000000002},"min":{"1min":0.36599999999999999,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":0.91700000000000004,"5min":0.91700000000000004,"15min":0.91700000000000004},"last":0.63}]}]},{"osd":5,"up_from":38,"seq":163208757276,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":5692,"kb_used_data":564,"kb_used_omap":0,"kb_used_meta":5120,"kb_avail":93739460,"statfs":{"total":95995035648,"available":95989207040,"internally_reserved":0,"allocated":577536,"data_stored":237518,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5242880},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Apr 14 13:55:25 2024","interfaces":[{"interface":"back","average":{"1min":0.54800000000000004,"5min":0.54800000000000004,"15min":0.54800000000000004},"min":{"1min":0.30499999999999999,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":0.81999999999999995,"5min":0.81999999999999995,"15min":0.81999999999999995},"last":0.61799999999999999},{"interface":"front","average":{"1min":0.55500000000000005,"5min":0.55500000000000005,"15min":0.55500000000000005},"min":{"1min":0.34899999999999998,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":0.86099999999999999,"5min":0.86099999999999999,"15min":0.86099999999999999},"last":0.54200000000000004}]},{"osd":1,"last update":"Sun Apr 14 13:55:25 2024","interfaces":[{"interface":"back","average":{"1min":0.58699999999999997,"5min":0.58699999999999997,"15min":0.58699999999999997},"min":{"1min":0.33900000000000002,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":0.98999999999999999,"5min":0.98999999999999999,"15min":0.98999999999999999},"last":0.65700000000000003},{"interface":"front","average":{"1min":0.56200000000000006,"5min":0.56200000000000006,"15min":0.56200000000000006},"min":{"1min":0.35999999999999999,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":0.89000000000000001,"5min":0.89000000000000001,"15min":0.89000000000000001},"last":0.497}]},{"osd":2,"last update":"Sun Apr 14 13:55:25 2024","interfaces":[{"interface":"back","average":{"1min":0.56599999999999995,"5min":0.56599999999999995,"15min":0.56599999999999995},"min":{"1min":0.29399999999999998,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":0.93600000000000005,"5min":0.93600000000000005,"15min":0.93600000000000005},"last":0.57699999999999996},{"interface":"front","average":{"1min":0.54900000000000004,"5min":0.54900000000000004,"15min":0.54900000000000004},"min":{"1min":0.36199999999999999,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":0.89700000000000002,"5min":0.89700000000000002,"15min":0.89700000000000002},"last":0.63800000000000001}]},{"osd":3,"last update":"Sun Apr 14 13:55:25 2024","interfaces":[{"interface":"back","average":{"1min":0.56000000000000005,"5min":0.56000000000000005,"15min":0.56000000000000005},"min":{"1min":0.33600000000000002,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":0.95999999999999996,"5min":0.95999999999999996,"15min":0.95999999999999996},"last":0.53300000000000003},{"interface":"front","average":{"1min":0.55500000000000005,"5min":0.55500000000000005,"15min":0.55500000000000005},"min":{"1min":0.32500000000000001,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":0.92100000000000004,"5min":0.92100000000000004,"15min":0.92100000000000004},"last":0.67400000000000004}]},{"osd":4,"last update":"Sun Apr 14 13:55:25 2024","interfaces":[{"interface":"back","average":{"1min":0.42899999999999999,"5min":0.42899999999999999,"15min":0.42899999999999999},"min":{"1min":0.28299999999999997,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":0.80900000000000005,"5min":0.80900000000000005,"15min":0.80900000000000005},"last":0.33700000000000002},{"interface":"front","average":{"1min":0.41999999999999998,"5min":0.41999999999999998,"15min":0.41999999999999998},"min":{"1min":0.27900000000000003,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":0.90700000000000003,"5min":0.90700000000000003,"15min":0.90700000000000003},"last":0.38700000000000001}]},{"osd":6,"last update":"Sun Apr 14 13:55:45 2024","interfaces":[{"interface":"back","average":{"1min":0.48599999999999999,"5min":0.48599999999999999,"15min":0.48599999999999999},"min":{"1min":0.32000000000000001,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.877,"5min":0.877,"15min":0.877},"last":0.46500000000000002},{"interface":"front","average":{"1min":0.51400000000000001,"5min":0.51400000000000001,"15min":0.51400000000000001},"min":{"1min":0.30199999999999999,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":0.97199999999999998,"5min":0.97199999999999998,"15min":0.97199999999999998},"last":0.50900000000000001}]},{"osd":7,"last update":"Sun Apr 14 13:56:13 2024","interfaces":[{"interface":"back","average":{"1min":0.56299999999999994,"5min":0.56299999999999994,"15min":0.56299999999999994},"min":{"1min":0.35199999999999998,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.94699999999999995,"5min":0.94699999999999995,"15min":0.94699999999999995},"last":0.58699999999999997},{"interface":"front","average":{"1min":0.54800000000000004,"5min":0.54800000000000004,"15min":0.54800000000000004},"min":{"1min":0.34200000000000003,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.86299999999999999,"5min":0.86299999999999999,"15min":0.86299999999999999},"last":0.442}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-14T13:56:27.632 DEBUG:teuthology.orchestra.run.smithi069:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid dc326fc2-fa65-11ee-bc8e-c7b262605968 -- ceph pg dump --format=json 2024-04-14T13:56:27.817 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:56:27 smithi069 ceph-mon[24818]: pgmap v34: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-14T13:56:27.817 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:56:27 smithi069 ceph-mon[29472]: pgmap v34: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-14T13:56:27.859 INFO:teuthology.orchestra.run.smithi069.stderr:Inferring config /var/lib/ceph/dc326fc2-fa65-11ee-bc8e-c7b262605968/mon.a/config 2024-04-14T13:56:28.663 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:56:28 smithi069 ceph-mon[24818]: from='client.24502 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-14T13:56:28.663 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:56:28 smithi069 ceph-mon[29472]: from='client.24502 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-14T13:56:28.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:56:28 smithi122 ceph-mon[25721]: from='client.24502 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-14T13:56:29.817 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:56:29 smithi069 ceph-mon[24818]: pgmap v35: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-14T13:56:29.817 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 14 13:56:29 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-y[25054]: ::ffff:172.21.15.122 - - [14/Apr/2024:13:56:29] "GET /metrics HTTP/1.1" 200 191492 "" "Prometheus/2.33.4" 2024-04-14T13:56:29.817 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:56:29 smithi069 ceph-mon[29472]: pgmap v35: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-14T13:56:29.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:56:29 smithi122 ceph-mon[25721]: pgmap v35: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-14T13:56:30.518 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-14T13:56:30.519 INFO:teuthology.orchestra.run.smithi069.stderr:dumped all 2024-04-14T13:56:31.567 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:56:31 smithi069 ceph-mon[24818]: pgmap v36: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-14T13:56:31.568 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:56:31 smithi069 ceph-mon[29472]: pgmap v36: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-14T13:56:31.595 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:56:31 smithi122 ceph-mon[25721]: pgmap v36: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-14T13:56:31.997 INFO:teuthology.orchestra.run.smithi069.stdout:{"pg_ready":true,"pg_map":{"version":36,"stamp":"2024-04-14T13:56:30.329900+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1586784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":128,"ondisk_log_size":128,"up":3,"acting":3,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":3,"num_osds":8,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":749961216,"kb_used":48912,"kb_used_data":6096,"kb_used_omap":0,"kb_used_meta":42752,"kb_avail":749912304,"statfs":{"total":767960285184,"available":767910199296,"internally_reserved":0,"allocated":6242304,"data_stored":3488916,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":43778048},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002023"},"pg_stats":[{"pgid":"1.0","version":"59'128","reported_seq":266,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-14T13:55:24.391850+0000","last_change":"2024-04-14T13:55:13.183515+0000","last_active":"2024-04-14T13:55:24.391850+0000","last_peered":"2024-04-14T13:55:24.391850+0000","last_clean":"2024-04-14T13:55:24.391850+0000","last_became_active":"2024-04-14T13:55:13.183047+0000","last_became_peered":"2024-04-14T13:55:13.183047+0000","last_unstale":"2024-04-14T13:55:24.391850+0000","last_undegraded":"2024-04-14T13:55:24.391850+0000","last_fullsized":"2024-04-14T13:55:24.391850+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-14T13:53:16.148701+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-14T13:53:16.148701+0000","last_clean_scrub_stamp":"2024-04-14T13:53:16.148701+0000","objects_scrubbed":0,"log_size":128,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T00:37:23.035958+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1586784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1586784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1597440,"data_stored":1586784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":128,"ondisk_log_size":128,"up":3,"acting":3,"num_store_stats":3}],"osd_stats":[{"osd":7,"up_from":54,"seq":231928234001,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6088,"kb_used_data":1088,"kb_used_omap":0,"kb_used_meta":4992,"kb_avail":93739064,"statfs":{"total":95995035648,"available":95988801536,"internally_reserved":0,"allocated":1114112,"data_stored":766769,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5111808},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Apr 14 13:56:10 2024","interfaces":[{"interface":"back","average":{"1min":0.499,"5min":0.499,"15min":0.499},"min":{"1min":0.38200000000000001,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":0.62,"5min":0.62,"15min":0.62},"last":0.47099999999999997},{"interface":"front","average":{"1min":0.51100000000000001,"5min":0.51100000000000001,"15min":0.51100000000000001},"min":{"1min":0.33700000000000002,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.73999999999999999,"5min":0.73999999999999999,"15min":0.73999999999999999},"last":0.622}]},{"osd":1,"last update":"Sun Apr 14 13:56:10 2024","interfaces":[{"interface":"back","average":{"1min":0.497,"5min":0.497,"15min":0.497},"min":{"1min":0.36099999999999999,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":0.64200000000000002,"5min":0.64200000000000002,"15min":0.64200000000000002},"last":0.56200000000000006},{"interface":"front","average":{"1min":0.504,"5min":0.504,"15min":0.504},"min":{"1min":0.32700000000000001,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":0.76300000000000001,"5min":0.76300000000000001,"15min":0.76300000000000001},"last":0.54300000000000004}]},{"osd":2,"last update":"Sun Apr 14 13:56:10 2024","interfaces":[{"interface":"back","average":{"1min":0.51600000000000001,"5min":0.51600000000000001,"15min":0.51600000000000001},"min":{"1min":0.34300000000000003,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":0.73099999999999998,"5min":0.73099999999999998,"15min":0.73099999999999998},"last":0.58199999999999996},{"interface":"front","average":{"1min":0.55600000000000005,"5min":0.55600000000000005,"15min":0.55600000000000005},"min":{"1min":0.41099999999999998,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":0.63500000000000001,"5min":0.63500000000000001,"15min":0.63500000000000001},"last":0.51700000000000002}]},{"osd":3,"last update":"Sun Apr 14 13:56:10 2024","interfaces":[{"interface":"back","average":{"1min":0.51400000000000001,"5min":0.51400000000000001,"15min":0.51400000000000001},"min":{"1min":0.377,"5min":0.377,"15min":0.377},"max":{"1min":0.68100000000000005,"5min":0.68100000000000005,"15min":0.68100000000000005},"last":0.52800000000000002},{"interface":"front","average":{"1min":0.53800000000000003,"5min":0.53800000000000003,"15min":0.53800000000000003},"min":{"1min":0.41799999999999998,"5min":0.41799999999999998,"15min":0.41799999999999998},"max":{"1min":0.69899999999999995,"5min":0.69899999999999995,"15min":0.69899999999999995},"last":0.60699999999999998}]},{"osd":4,"last update":"Sun Apr 14 13:56:10 2024","interfaces":[{"interface":"back","average":{"1min":0.47499999999999998,"5min":0.47499999999999998,"15min":0.47499999999999998},"min":{"1min":0.29799999999999999,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":0.72399999999999998,"5min":0.72399999999999998,"15min":0.72399999999999998},"last":0.59299999999999997},{"interface":"front","average":{"1min":0.46899999999999997,"5min":0.46899999999999997,"15min":0.46899999999999997},"min":{"1min":0.32200000000000001,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":0.79400000000000004,"5min":0.79400000000000004,"15min":0.79400000000000004},"last":0.65300000000000002}]},{"osd":5,"last update":"Sun Apr 14 13:56:10 2024","interfaces":[{"interface":"back","average":{"1min":0.49299999999999999,"5min":0.49299999999999999,"15min":0.49299999999999999},"min":{"1min":0.35699999999999998,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":0.67200000000000004,"5min":0.67200000000000004,"15min":0.67200000000000004},"last":0.66700000000000004},{"interface":"front","average":{"1min":0.498,"5min":0.498,"15min":0.498},"min":{"1min":0.36099999999999999,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":0.80000000000000004,"5min":0.80000000000000004,"15min":0.80000000000000004},"last":0.70099999999999996}]},{"osd":6,"last update":"Sun Apr 14 13:56:10 2024","interfaces":[{"interface":"back","average":{"1min":0.51700000000000002,"5min":0.51700000000000002,"15min":0.51700000000000002},"min":{"1min":0.33400000000000002,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":0.76100000000000001,"5min":0.76100000000000001,"15min":0.76100000000000001},"last":0.67800000000000005},{"interface":"front","average":{"1min":0.50900000000000001,"5min":0.50900000000000001,"15min":0.50900000000000001},"min":{"1min":0.34899999999999998,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":0.76700000000000002,"5min":0.76700000000000002,"15min":0.76700000000000002},"last":0.68799999999999994}]}]},{"osd":6,"up_from":46,"seq":197568495637,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6148,"kb_used_data":1084,"kb_used_omap":0,"kb_used_meta":5056,"kb_avail":93739004,"statfs":{"total":95995035648,"available":95988740096,"internally_reserved":0,"allocated":1110016,"data_stored":766496,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5177344},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Apr 14 13:55:47 2024","interfaces":[{"interface":"back","average":{"1min":0.53300000000000003,"5min":0.53300000000000003,"15min":0.53300000000000003},"min":{"1min":0.312,"5min":0.312,"15min":0.312},"max":{"1min":0.74099999999999999,"5min":0.74099999999999999,"15min":0.74099999999999999},"last":0.41999999999999998},{"interface":"front","average":{"1min":0.54200000000000004,"5min":0.54200000000000004,"15min":0.54200000000000004},"min":{"1min":0.38400000000000001,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":0.76500000000000001,"5min":0.76500000000000001,"15min":0.76500000000000001},"last":0.372}]},{"osd":1,"last update":"Sun Apr 14 13:55:47 2024","interfaces":[{"interface":"back","average":{"1min":0.52600000000000002,"5min":0.52600000000000002,"15min":0.52600000000000002},"min":{"1min":0.34599999999999997,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":0.82499999999999996,"5min":0.82499999999999996,"15min":0.82499999999999996},"last":0.44500000000000001},{"interface":"front","average":{"1min":0.54500000000000004,"5min":0.54500000000000004,"15min":0.54500000000000004},"min":{"1min":0.375,"5min":0.375,"15min":0.375},"max":{"1min":0.77100000000000002,"5min":0.77100000000000002,"15min":0.77100000000000002},"last":0.47399999999999998}]},{"osd":2,"last update":"Sun Apr 14 13:55:47 2024","interfaces":[{"interface":"back","average":{"1min":0.52000000000000002,"5min":0.52000000000000002,"15min":0.52000000000000002},"min":{"1min":0.35699999999999998,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":0.77200000000000002,"5min":0.77200000000000002,"15min":0.77200000000000002},"last":0.50800000000000001},{"interface":"front","average":{"1min":0.54400000000000004,"5min":0.54400000000000004,"15min":0.54400000000000004},"min":{"1min":0.36599999999999999,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":0.76900000000000002,"5min":0.76900000000000002,"15min":0.76900000000000002},"last":0.45700000000000002}]},{"osd":3,"last update":"Sun Apr 14 13:55:47 2024","interfaces":[{"interface":"back","average":{"1min":0.55500000000000005,"5min":0.55500000000000005,"15min":0.55500000000000005},"min":{"1min":0.40300000000000002,"5min":0.40300000000000002,"15min":0.40300000000000002},"max":{"1min":0.84099999999999997,"5min":0.84099999999999997,"15min":0.84099999999999997},"last":0.36099999999999999},{"interface":"front","average":{"1min":0.54400000000000004,"5min":0.54400000000000004,"15min":0.54400000000000004},"min":{"1min":0.37,"5min":0.37,"15min":0.37},"max":{"1min":0.76100000000000001,"5min":0.76100000000000001,"15min":0.76100000000000001},"last":0.53400000000000003}]},{"osd":4,"last update":"Sun Apr 14 13:55:47 2024","interfaces":[{"interface":"back","average":{"1min":0.42199999999999999,"5min":0.42199999999999999,"15min":0.42199999999999999},"min":{"1min":0.26300000000000001,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":0.58699999999999997,"5min":0.58699999999999997,"15min":0.58699999999999997},"last":0.439},{"interface":"front","average":{"1min":0.42999999999999999,"5min":0.42999999999999999,"15min":0.42999999999999999},"min":{"1min":0.26700000000000002,"5min":0.26700000000000002,"15min":0.26700000000000002},"max":{"1min":0.74299999999999999,"5min":0.74299999999999999,"15min":0.74299999999999999},"last":0.46800000000000003}]},{"osd":5,"last update":"Sun Apr 14 13:55:47 2024","interfaces":[{"interface":"back","average":{"1min":0.42799999999999999,"5min":0.42799999999999999,"15min":0.42799999999999999},"min":{"1min":0.30499999999999999,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":0.63,"5min":0.63,"15min":0.63},"last":0.40600000000000003},{"interface":"front","average":{"1min":0.45900000000000002,"5min":0.45900000000000002,"15min":0.45900000000000002},"min":{"1min":0.29899999999999999,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":0.81200000000000006,"5min":0.81200000000000006,"15min":0.81200000000000006},"last":0.39800000000000002}]},{"osd":7,"last update":"Sun Apr 14 13:56:09 2024","interfaces":[{"interface":"back","average":{"1min":0.5,"5min":0.5,"15min":0.5},"min":{"1min":0.28999999999999998,"5min":0.28999999999999998,"15min":0.28999999999999998},"max":{"1min":0.76000000000000001,"5min":0.76000000000000001,"15min":0.76000000000000001},"last":0.48799999999999999},{"interface":"front","average":{"1min":0.51800000000000002,"5min":0.51800000000000002,"15min":0.51800000000000002},"min":{"1min":0.34999999999999998,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":0.747,"5min":0.747,"15min":0.747},"last":0.496}]}]},{"osd":1,"up_from":13,"seq":55834574894,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":6336,"kb_used_data":568,"kb_used_omap":0,"kb_used_meta":5760,"kb_avail":93738816,"statfs":{"total":95995035648,"available":95988547584,"internally_reserved":0,"allocated":581632,"data_stored":237841,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5898240},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Apr 14 13:55:56 2024","interfaces":[{"interface":"back","average":{"1min":0.38,"5min":0.36099999999999999,"15min":0.35899999999999999},"min":{"1min":0.26400000000000001,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":0.55700000000000005,"5min":0.55700000000000005,"15min":0.55700000000000005},"last":0.39000000000000001},{"interface":"front","average":{"1min":0.40200000000000002,"5min":0.38,"15min":0.36799999999999999},"min":{"1min":0.26700000000000002,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":0.63800000000000001,"5min":0.63800000000000001,"15min":0.63800000000000001},"last":0.33700000000000002}]},{"osd":2,"last update":"Sun Apr 14 13:56:21 2024","interfaces":[{"interface":"back","average":{"1min":0.377,"5min":0.38800000000000001,"15min":0.39000000000000001},"min":{"1min":0.25,"5min":0.25,"15min":0.25},"max":{"1min":0.61799999999999999,"5min":0.61799999999999999,"15min":0.61799999999999999},"last":0.28799999999999998},{"interface":"front","average":{"1min":0.35599999999999998,"5min":0.39400000000000002,"15min":0.40300000000000002},"min":{"1min":0.24399999999999999,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":0.52300000000000002,"5min":0.54900000000000004,"15min":0.54900000000000004},"last":0.37}]},{"osd":3,"last update":"Sun Apr 14 13:55:40 2024","interfaces":[{"interface":"back","average":{"1min":0.41499999999999998,"5min":0.40799999999999997,"15min":0.40699999999999997},"min":{"1min":0.26000000000000001,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":0.57299999999999995,"5min":0.57299999999999995,"15min":0.57299999999999995},"last":0.34499999999999997},{"interface":"front","average":{"1min":0.434,"5min":0.439,"15min":0.44},"min":{"1min":0.28899999999999998,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":0.70199999999999996,"5min":0.70199999999999996,"15min":0.70199999999999996},"last":0.35599999999999998}]},{"osd":4,"last update":"Sun Apr 14 13:56:02 2024","interfaces":[{"interface":"back","average":{"1min":0.52500000000000002,"5min":0.51700000000000002,"15min":0.51600000000000001},"min":{"1min":0.38600000000000001,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":0.67000000000000004,"5min":0.67800000000000005,"15min":0.67800000000000005},"last":0.57399999999999995},{"interface":"front","average":{"1min":0.53900000000000003,"5min":0.52200000000000002,"15min":0.51900000000000002},"min":{"1min":0.39800000000000002,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":0.77700000000000002,"5min":0.77700000000000002,"15min":0.77700000000000002},"last":0.67700000000000005}]},{"osd":5,"last update":"Sun Apr 14 13:55:23 2024","interfaces":[{"interface":"back","average":{"1min":0.53100000000000003,"5min":0.53100000000000003,"15min":0.53100000000000003},"min":{"1min":0.373,"5min":0.373,"15min":0.373},"max":{"1min":0.749,"5min":0.749,"15min":0.749},"last":0.55500000000000005},{"interface":"front","average":{"1min":0.54300000000000004,"5min":0.54300000000000004,"15min":0.54300000000000004},"min":{"1min":0.374,"5min":0.374,"15min":0.374},"max":{"1min":0.72299999999999998,"5min":0.72299999999999998,"15min":0.72299999999999998},"last":0.61599999999999999}]},{"osd":6,"last update":"Sun Apr 14 13:55:46 2024","interfaces":[{"interface":"back","average":{"1min":0.59099999999999997,"5min":0.59099999999999997,"15min":0.59099999999999997},"min":{"1min":0.442,"5min":0.442,"15min":0.442},"max":{"1min":0.86299999999999999,"5min":0.86299999999999999,"15min":0.86299999999999999},"last":0.71299999999999997},{"interface":"front","average":{"1min":0.59199999999999997,"5min":0.59199999999999997,"15min":0.59199999999999997},"min":{"1min":0.39400000000000002,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":0.81399999999999995,"5min":0.81399999999999995,"15min":0.81399999999999995},"last":0.59899999999999998}]},{"osd":7,"last update":"Sun Apr 14 13:56:09 2024","interfaces":[{"interface":"back","average":{"1min":0.60399999999999998,"5min":0.60399999999999998,"15min":0.60399999999999998},"min":{"1min":0.42499999999999999,"5min":0.42499999999999999,"15min":0.42499999999999999},"max":{"1min":0.82499999999999996,"5min":0.82499999999999996,"15min":0.82499999999999996},"last":0.67200000000000004},{"interface":"front","average":{"1min":0.60599999999999998,"5min":0.60599999999999998,"15min":0.60599999999999998},"min":{"1min":0.40100000000000002,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":0.88500000000000001,"5min":0.88500000000000001,"15min":0.88500000000000001},"last":0.70299999999999996}]}]},{"osd":0,"up_from":8,"seq":34359738418,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6920,"kb_used_data":1088,"kb_used_omap":0,"kb_used_meta":5824,"kb_avail":93738232,"statfs":{"total":95995035648,"available":95987949568,"internally_reserved":0,"allocated":1114112,"data_stored":766769,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5963776},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sun Apr 14 13:55:54 2024","interfaces":[{"interface":"back","average":{"1min":0.35599999999999998,"5min":0.34399999999999997,"15min":0.33000000000000002},"min":{"1min":0.23300000000000001,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":0.50600000000000001,"5min":0.58099999999999996,"15min":0.58099999999999996},"last":0.40500000000000003},{"interface":"front","average":{"1min":0.36399999999999999,"5min":0.34899999999999998,"15min":0.33500000000000002},"min":{"1min":0.23799999999999999,"5min":0.223,"15min":0.223},"max":{"1min":0.58299999999999996,"5min":0.61099999999999999,"15min":0.61099999999999999},"last":0.39500000000000002}]},{"osd":2,"last update":"Sun Apr 14 13:55:20 2024","interfaces":[{"interface":"back","average":{"1min":0.40899999999999997,"5min":0.433,"15min":0.437},"min":{"1min":0.28199999999999997,"5min":0.28199999999999997,"15min":0.28199999999999997},"max":{"1min":0.52600000000000002,"5min":0.64600000000000002,"15min":0.64600000000000002},"last":0.377},{"interface":"front","average":{"1min":0.40899999999999997,"5min":0.41699999999999998,"15min":0.41799999999999998},"min":{"1min":0.26900000000000002,"5min":0.23000000000000001,"15min":0.23000000000000001},"max":{"1min":0.55300000000000005,"5min":0.55300000000000005,"15min":0.55300000000000005},"last":0.41799999999999998}]},{"osd":3,"last update":"Sun Apr 14 13:55:38 2024","interfaces":[{"interface":"back","average":{"1min":0.42899999999999999,"5min":0.46200000000000002,"15min":0.46800000000000003},"min":{"1min":0.30099999999999999,"5min":0.30099999999999999,"15min":0.30099999999999999},"max":{"1min":0.56599999999999995,"5min":0.78400000000000003,"15min":0.78400000000000003},"last":0.44700000000000001},{"interface":"front","average":{"1min":0.46600000000000003,"5min":0.49399999999999999,"15min":0.499},"min":{"1min":0.32500000000000001,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":0.63600000000000001,"5min":0.68899999999999995,"15min":0.68899999999999995},"last":0.442}]},{"osd":4,"last update":"Sun Apr 14 13:56:04 2024","interfaces":[{"interface":"back","average":{"1min":0.53200000000000003,"5min":0.59799999999999998,"15min":0.60899999999999999},"min":{"1min":0.371,"5min":0.371,"15min":0.371},"max":{"1min":0.80200000000000005,"5min":0.84499999999999997,"15min":0.84499999999999997},"last":0.60099999999999998},{"interface":"front","average":{"1min":0.505,"5min":0.60399999999999998,"15min":0.62},"min":{"1min":0.372,"5min":0.372,"15min":0.372},"max":{"1min":0.82999999999999996,"5min":0.995,"15min":0.995},"last":0.61399999999999999}]},{"osd":5,"last update":"Sun Apr 14 13:55:23 2024","interfaces":[{"interface":"back","average":{"1min":0.58399999999999996,"5min":0.58399999999999996,"15min":0.58399999999999996},"min":{"1min":0.38400000000000001,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":0.90600000000000003,"5min":0.90600000000000003,"15min":0.90600000000000003},"last":0.58299999999999996},{"interface":"front","average":{"1min":0.59399999999999997,"5min":0.59399999999999997,"15min":0.59399999999999997},"min":{"1min":0.376,"5min":0.376,"15min":0.376},"max":{"1min":0.875,"5min":0.875,"15min":0.875},"last":0.63200000000000001}]},{"osd":6,"last update":"Sun Apr 14 13:55:50 2024","interfaces":[{"interface":"back","average":{"1min":0.59299999999999997,"5min":0.59299999999999997,"15min":0.59299999999999997},"min":{"1min":0.40400000000000003,"5min":0.40400000000000003,"15min":0.40400000000000003},"max":{"1min":0.93500000000000005,"5min":0.93500000000000005,"15min":0.93500000000000005},"last":0.70399999999999996},{"interface":"front","average":{"1min":0.60499999999999998,"5min":0.60499999999999998,"15min":0.60499999999999998},"min":{"1min":0.39700000000000002,"5min":0.39700000000000002,"15min":0.39700000000000002},"max":{"1min":0.95099999999999996,"5min":0.95099999999999996,"15min":0.95099999999999996},"last":0.70999999999999996}]},{"osd":7,"last update":"Sun Apr 14 13:56:11 2024","interfaces":[{"interface":"back","average":{"1min":0.56299999999999994,"5min":0.56299999999999994,"15min":0.56299999999999994},"min":{"1min":0.38200000000000001,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":0.82699999999999996,"5min":0.82699999999999996,"15min":0.82699999999999996},"last":0.64200000000000002},{"interface":"front","average":{"1min":0.56499999999999995,"5min":0.56499999999999995,"15min":0.56499999999999995},"min":{"1min":0.46800000000000003,"5min":0.46800000000000003,"15min":0.46800000000000003},"max":{"1min":0.78000000000000003,"5min":0.78000000000000003,"15min":0.78000000000000003},"last":0.64800000000000002}]}]},{"osd":2,"up_from":18,"seq":77309411368,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":6272,"kb_used_data":568,"kb_used_omap":0,"kb_used_meta":5696,"kb_avail":93738880,"statfs":{"total":95995035648,"available":95988613120,"internally_reserved":0,"allocated":581632,"data_stored":237841,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5832704},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Apr 14 13:56:21 2024","interfaces":[{"interface":"back","average":{"1min":0.36899999999999999,"5min":0.39100000000000001,"15min":0.39300000000000002},"min":{"1min":0.25800000000000001,"5min":0.245,"15min":0.245},"max":{"1min":0.59299999999999997,"5min":0.59999999999999998,"15min":0.59999999999999998},"last":0.29199999999999998},{"interface":"front","average":{"1min":0.377,"5min":0.38300000000000001,"15min":0.38700000000000001},"min":{"1min":0.28499999999999998,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":0.56999999999999995,"5min":0.56999999999999995,"15min":0.56999999999999995},"last":0.34699999999999998}]},{"osd":1,"last update":"Sun Apr 14 13:56:21 2024","interfaces":[{"interface":"back","average":{"1min":0.376,"5min":0.39600000000000002,"15min":0.40000000000000002},"min":{"1min":0.255,"5min":0.255,"15min":0.255},"max":{"1min":0.53900000000000003,"5min":0.59599999999999997,"15min":0.59599999999999997},"last":0.36399999999999999},{"interface":"front","average":{"1min":0.40500000000000003,"5min":0.42299999999999999,"15min":0.42699999999999999},"min":{"1min":0.29699999999999999,"5min":0.27600000000000002,"15min":0.27600000000000002},"max":{"1min":0.64900000000000002,"5min":0.64900000000000002,"15min":0.64900000000000002},"last":0.33900000000000002}]},{"osd":3,"last update":"Sun Apr 14 13:55:43 2024","interfaces":[{"interface":"back","average":{"1min":0.433,"5min":0.437,"15min":0.437},"min":{"1min":0.28899999999999998,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":0.61799999999999999,"5min":0.61799999999999999,"15min":0.61799999999999999},"last":0.40500000000000003},{"interface":"front","average":{"1min":0.44900000000000001,"5min":0.46700000000000003,"15min":0.46999999999999997},"min":{"1min":0.29799999999999999,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":0.73399999999999999,"5min":0.749,"15min":0.749},"last":0.38300000000000001}]},{"osd":4,"last update":"Sun Apr 14 13:56:00 2024","interfaces":[{"interface":"back","average":{"1min":1.361,"5min":0.72299999999999998,"15min":0.61699999999999999},"min":{"1min":0.41899999999999998,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":16.198,"5min":16.198,"15min":16.198},"last":0.629},{"interface":"front","average":{"1min":0.53200000000000003,"5min":1.2130000000000001,"15min":1.327},"min":{"1min":0.39500000000000002,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":0.82399999999999995,"5min":16.658999999999999,"15min":16.658999999999999},"last":0.44600000000000001}]},{"osd":5,"last update":"Sun Apr 14 13:55:24 2024","interfaces":[{"interface":"back","average":{"1min":0.89400000000000002,"5min":0.89400000000000002,"15min":0.89400000000000002},"min":{"1min":0.38500000000000001,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":7.4480000000000004,"5min":7.4480000000000004,"15min":7.4480000000000004},"last":0.55900000000000005},{"interface":"front","average":{"1min":0.61099999999999999,"5min":0.61099999999999999,"15min":0.61099999999999999},"min":{"1min":0.434,"5min":0.434,"15min":0.434},"max":{"1min":0.89400000000000002,"5min":0.89400000000000002,"15min":0.89400000000000002},"last":0.53200000000000003}]},{"osd":6,"last update":"Sun Apr 14 13:55:46 2024","interfaces":[{"interface":"back","average":{"1min":0.56699999999999995,"5min":0.56699999999999995,"15min":0.56699999999999995},"min":{"1min":0.39200000000000002,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":0.81799999999999995,"5min":0.81799999999999995,"15min":0.81799999999999995},"last":0.51700000000000002},{"interface":"front","average":{"1min":0.57399999999999995,"5min":0.57399999999999995,"15min":0.57399999999999995},"min":{"1min":0.40899999999999997,"5min":0.40899999999999997,"15min":0.40899999999999997},"max":{"1min":0.76400000000000001,"5min":0.76400000000000001,"15min":0.76400000000000001},"last":0.50800000000000001}]},{"osd":7,"last update":"Sun Apr 14 13:56:09 2024","interfaces":[{"interface":"back","average":{"1min":0.61399999999999999,"5min":0.61399999999999999,"15min":0.61399999999999999},"min":{"1min":0.38100000000000001,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":1.0049999999999999,"5min":1.0049999999999999,"15min":1.0049999999999999},"last":0.59799999999999998},{"interface":"front","average":{"1min":0.63300000000000001,"5min":0.63300000000000001,"15min":0.63300000000000001},"min":{"1min":0.39400000000000002,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":1.1499999999999999,"5min":1.1499999999999999,"15min":1.1499999999999999},"last":0.53700000000000003}]}]},{"osd":3,"up_from":25,"seq":107374182435,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":5696,"kb_used_data":568,"kb_used_omap":0,"kb_used_meta":5120,"kb_avail":93739456,"statfs":{"total":95995035648,"available":95989202944,"internally_reserved":0,"allocated":581632,"data_stored":237841,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5242880},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Apr 14 13:55:43 2024","interfaces":[{"interface":"back","average":{"1min":0.38700000000000001,"5min":0.39600000000000002,"15min":0.39800000000000002},"min":{"1min":0.27200000000000002,"5min":0.25700000000000001,"15min":0.25700000000000001},"max":{"1min":0.67200000000000004,"5min":0.67200000000000004,"15min":0.67200000000000004},"last":0.30399999999999999},{"interface":"front","average":{"1min":0.41699999999999998,"5min":0.42299999999999999,"15min":0.42399999999999999},"min":{"1min":0.249,"5min":0.249,"15min":0.249},"max":{"1min":0.82299999999999995,"5min":0.82299999999999995,"15min":0.82299999999999995},"last":0.372}]},{"osd":1,"last update":"Sun Apr 14 13:55:43 2024","interfaces":[{"interface":"back","average":{"1min":0.437,"5min":0.441,"15min":0.442},"min":{"1min":0.28799999999999998,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":0.62,"5min":0.62,"15min":0.62},"last":0.34100000000000003},{"interface":"front","average":{"1min":0.42799999999999999,"5min":0.433,"15min":0.434},"min":{"1min":0.23699999999999999,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":0.74199999999999999,"5min":0.74199999999999999,"15min":0.74199999999999999},"last":0.35899999999999999}]},{"osd":2,"last update":"Sun Apr 14 13:55:43 2024","interfaces":[{"interface":"back","average":{"1min":0.48499999999999999,"5min":0.48199999999999998,"15min":0.48199999999999998},"min":{"1min":0.34599999999999997,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":0.71399999999999997,"5min":0.71399999999999997,"15min":0.71399999999999997},"last":0.38600000000000001},{"interface":"front","average":{"1min":0.45200000000000001,"5min":0.47799999999999998,"15min":0.48199999999999998},"min":{"1min":0.29199999999999998,"5min":0.28999999999999998,"15min":0.28999999999999998},"max":{"1min":0.63800000000000001,"5min":0.63800000000000001,"15min":0.63800000000000001},"last":0.73999999999999999}]},{"osd":4,"last update":"Sun Apr 14 13:56:00 2024","interfaces":[{"interface":"back","average":{"1min":0.59799999999999998,"5min":0.58599999999999997,"15min":0.58399999999999996},"min":{"1min":0.41499999999999998,"5min":0.41499999999999998,"15min":0.41499999999999998},"max":{"1min":0.86099999999999999,"5min":0.86099999999999999,"15min":0.86099999999999999},"last":0.52400000000000002},{"interface":"front","average":{"1min":0.58899999999999997,"5min":0.56299999999999994,"15min":0.55900000000000005},"min":{"1min":0.38500000000000001,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":0.85199999999999998,"5min":0.85199999999999998,"15min":0.85199999999999998},"last":0.81699999999999995}]},{"osd":5,"last update":"Sun Apr 14 13:55:24 2024","interfaces":[{"interface":"back","average":{"1min":0.57499999999999996,"5min":0.57499999999999996,"15min":0.57499999999999996},"min":{"1min":0.40699999999999997,"5min":0.40699999999999997,"15min":0.40699999999999997},"max":{"1min":0.89000000000000001,"5min":0.89000000000000001,"15min":0.89000000000000001},"last":0.502},{"interface":"front","average":{"1min":0.57499999999999996,"5min":0.57499999999999996,"15min":0.57499999999999996},"min":{"1min":0.44800000000000001,"5min":0.44800000000000001,"15min":0.44800000000000001},"max":{"1min":0.78000000000000003,"5min":0.78000000000000003,"15min":0.78000000000000003},"last":0.80000000000000004}]},{"osd":6,"last update":"Sun Apr 14 13:55:48 2024","interfaces":[{"interface":"back","average":{"1min":0.623,"5min":0.623,"15min":0.623},"min":{"1min":0.41499999999999998,"5min":0.41499999999999998,"15min":0.41499999999999998},"max":{"1min":0.94399999999999995,"5min":0.94399999999999995,"15min":0.94399999999999995},"last":0.57199999999999995},{"interface":"front","average":{"1min":0.60599999999999998,"5min":0.60599999999999998,"15min":0.60599999999999998},"min":{"1min":0.34300000000000003,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":0.92300000000000004,"5min":0.92300000000000004,"15min":0.92300000000000004},"last":0.83099999999999996}]},{"osd":7,"last update":"Sun Apr 14 13:56:09 2024","interfaces":[{"interface":"back","average":{"1min":0.59099999999999997,"5min":0.59099999999999997,"15min":0.59099999999999997},"min":{"1min":0.40200000000000002,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":0.872,"5min":0.872,"15min":0.872},"last":0.77100000000000002},{"interface":"front","average":{"1min":0.63400000000000001,"5min":0.63400000000000001,"15min":0.63400000000000001},"min":{"1min":0.42199999999999999,"5min":0.42199999999999999,"15min":0.42199999999999999},"max":{"1min":0.92100000000000004,"5min":0.92100000000000004,"15min":0.92100000000000004},"last":0.63700000000000001}]}]},{"osd":4,"up_from":30,"seq":128849018912,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":5760,"kb_used_data":568,"kb_used_omap":0,"kb_used_meta":5184,"kb_avail":93739392,"statfs":{"total":95995035648,"available":95989137408,"internally_reserved":0,"allocated":581632,"data_stored":237841,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5308416},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Apr 14 13:56:01 2024","interfaces":[{"interface":"back","average":{"1min":0.495,"5min":0.50900000000000001,"15min":0.51100000000000001},"min":{"1min":0.22800000000000001,"5min":0.22800000000000001,"15min":0.22800000000000001},"max":{"1min":0.80300000000000005,"5min":0.80300000000000005,"15min":0.80300000000000005},"last":0.58199999999999996},{"interface":"front","average":{"1min":0.51700000000000002,"5min":0.54100000000000004,"15min":0.54500000000000004},"min":{"1min":0.29999999999999999,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":0.86699999999999999,"5min":0.86699999999999999,"15min":0.86699999999999999},"last":0.59799999999999998}]},{"osd":1,"last update":"Sun Apr 14 13:56:01 2024","interfaces":[{"interface":"back","average":{"1min":0.51500000000000001,"5min":0.52100000000000002,"15min":0.52200000000000002},"min":{"1min":0.34200000000000003,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.89400000000000002,"5min":0.89400000000000002,"15min":0.89400000000000002},"last":0.53000000000000003},{"interface":"front","average":{"1min":0.51300000000000001,"5min":0.52100000000000002,"15min":0.52200000000000002},"min":{"1min":0.35399999999999998,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":0.81100000000000005,"5min":0.81100000000000005,"15min":0.81100000000000005},"last":0.55100000000000005}]},{"osd":2,"last update":"Sun Apr 14 13:56:01 2024","interfaces":[{"interface":"back","average":{"1min":0.51800000000000002,"5min":0.54200000000000004,"15min":0.54600000000000004},"min":{"1min":0.34699999999999998,"5min":0.34699999999999998,"15min":0.34699999999999998},"max":{"1min":0.83099999999999996,"5min":0.83099999999999996,"15min":0.83099999999999996},"last":0.51200000000000001},{"interface":"front","average":{"1min":0.52900000000000003,"5min":0.54100000000000004,"15min":0.54300000000000004},"min":{"1min":0.32300000000000001,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":0.85299999999999998,"5min":0.85299999999999998,"15min":0.85299999999999998},"last":0.56799999999999995}]},{"osd":3,"last update":"Sun Apr 14 13:56:01 2024","interfaces":[{"interface":"back","average":{"1min":0.55300000000000005,"5min":0.56899999999999995,"15min":0.57199999999999995},"min":{"1min":0.373,"5min":0.373,"15min":0.373},"max":{"1min":0.90400000000000003,"5min":0.90400000000000003,"15min":0.90400000000000003},"last":0.62},{"interface":"front","average":{"1min":0.55700000000000005,"5min":0.56100000000000005,"15min":0.56200000000000006},"min":{"1min":0.38500000000000001,"5min":0.37,"15min":0.37},"max":{"1min":0.876,"5min":0.876,"15min":0.876},"last":0.437}]},{"osd":5,"last update":"Sun Apr 14 13:55:25 2024","interfaces":[{"interface":"back","average":{"1min":0.38800000000000001,"5min":0.38800000000000001,"15min":0.38800000000000001},"min":{"1min":0.255,"5min":0.255,"15min":0.255},"max":{"1min":0.52900000000000003,"5min":0.52900000000000003,"15min":0.52900000000000003},"last":0.65500000000000003},{"interface":"front","average":{"1min":0.40100000000000002,"5min":0.40100000000000002,"15min":0.40100000000000002},"min":{"1min":0.28000000000000003,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":0.59099999999999997,"5min":0.59099999999999997,"15min":0.59099999999999997},"last":0.47699999999999998}]},{"osd":6,"last update":"Sun Apr 14 13:55:47 2024","interfaces":[{"interface":"back","average":{"1min":0.47999999999999998,"5min":0.47999999999999998,"15min":0.47999999999999998},"min":{"1min":0.29699999999999999,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":0.93600000000000005,"5min":0.93600000000000005,"15min":0.93600000000000005},"last":0.66800000000000004},{"interface":"front","average":{"1min":0.49199999999999999,"5min":0.49199999999999999,"15min":0.49199999999999999},"min":{"1min":0.315,"5min":0.315,"15min":0.315},"max":{"1min":0.98799999999999999,"5min":0.98799999999999999,"15min":0.98799999999999999},"last":0.67800000000000005}]},{"osd":7,"last update":"Sun Apr 14 13:56:09 2024","interfaces":[{"interface":"back","average":{"1min":0.502,"5min":0.502,"15min":0.502},"min":{"1min":0.311,"5min":0.311,"15min":0.311},"max":{"1min":0.84899999999999998,"5min":0.84899999999999998,"15min":0.84899999999999998},"last":0.64400000000000002},{"interface":"front","average":{"1min":0.52500000000000002,"5min":0.52500000000000002,"15min":0.52500000000000002},"min":{"1min":0.36599999999999999,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":0.91700000000000004,"5min":0.91700000000000004,"15min":0.91700000000000004},"last":0.63}]}]},{"osd":5,"up_from":38,"seq":163208757276,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":5692,"kb_used_data":564,"kb_used_omap":0,"kb_used_meta":5120,"kb_avail":93739460,"statfs":{"total":95995035648,"available":95989207040,"internally_reserved":0,"allocated":577536,"data_stored":237518,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5242880},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Apr 14 13:55:25 2024","interfaces":[{"interface":"back","average":{"1min":0.54800000000000004,"5min":0.54800000000000004,"15min":0.54800000000000004},"min":{"1min":0.30499999999999999,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":0.81999999999999995,"5min":0.81999999999999995,"15min":0.81999999999999995},"last":0.61799999999999999},{"interface":"front","average":{"1min":0.55500000000000005,"5min":0.55500000000000005,"15min":0.55500000000000005},"min":{"1min":0.34899999999999998,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":0.86099999999999999,"5min":0.86099999999999999,"15min":0.86099999999999999},"last":0.54200000000000004}]},{"osd":1,"last update":"Sun Apr 14 13:55:25 2024","interfaces":[{"interface":"back","average":{"1min":0.58699999999999997,"5min":0.58699999999999997,"15min":0.58699999999999997},"min":{"1min":0.33900000000000002,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":0.98999999999999999,"5min":0.98999999999999999,"15min":0.98999999999999999},"last":0.65700000000000003},{"interface":"front","average":{"1min":0.56200000000000006,"5min":0.56200000000000006,"15min":0.56200000000000006},"min":{"1min":0.35999999999999999,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":0.89000000000000001,"5min":0.89000000000000001,"15min":0.89000000000000001},"last":0.497}]},{"osd":2,"last update":"Sun Apr 14 13:55:25 2024","interfaces":[{"interface":"back","average":{"1min":0.56599999999999995,"5min":0.56599999999999995,"15min":0.56599999999999995},"min":{"1min":0.29399999999999998,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":0.93600000000000005,"5min":0.93600000000000005,"15min":0.93600000000000005},"last":0.57699999999999996},{"interface":"front","average":{"1min":0.54900000000000004,"5min":0.54900000000000004,"15min":0.54900000000000004},"min":{"1min":0.36199999999999999,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":0.89700000000000002,"5min":0.89700000000000002,"15min":0.89700000000000002},"last":0.63800000000000001}]},{"osd":3,"last update":"Sun Apr 14 13:55:25 2024","interfaces":[{"interface":"back","average":{"1min":0.56000000000000005,"5min":0.56000000000000005,"15min":0.56000000000000005},"min":{"1min":0.33600000000000002,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":0.95999999999999996,"5min":0.95999999999999996,"15min":0.95999999999999996},"last":0.53300000000000003},{"interface":"front","average":{"1min":0.55500000000000005,"5min":0.55500000000000005,"15min":0.55500000000000005},"min":{"1min":0.32500000000000001,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":0.92100000000000004,"5min":0.92100000000000004,"15min":0.92100000000000004},"last":0.67400000000000004}]},{"osd":4,"last update":"Sun Apr 14 13:55:25 2024","interfaces":[{"interface":"back","average":{"1min":0.42899999999999999,"5min":0.42899999999999999,"15min":0.42899999999999999},"min":{"1min":0.28299999999999997,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":0.80900000000000005,"5min":0.80900000000000005,"15min":0.80900000000000005},"last":0.33700000000000002},{"interface":"front","average":{"1min":0.41999999999999998,"5min":0.41999999999999998,"15min":0.41999999999999998},"min":{"1min":0.27900000000000003,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":0.90700000000000003,"5min":0.90700000000000003,"15min":0.90700000000000003},"last":0.38700000000000001}]},{"osd":6,"last update":"Sun Apr 14 13:55:45 2024","interfaces":[{"interface":"back","average":{"1min":0.48599999999999999,"5min":0.48599999999999999,"15min":0.48599999999999999},"min":{"1min":0.32000000000000001,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.877,"5min":0.877,"15min":0.877},"last":0.46500000000000002},{"interface":"front","average":{"1min":0.51400000000000001,"5min":0.51400000000000001,"15min":0.51400000000000001},"min":{"1min":0.30199999999999999,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":0.97199999999999998,"5min":0.97199999999999998,"15min":0.97199999999999998},"last":0.50900000000000001}]},{"osd":7,"last update":"Sun Apr 14 13:56:13 2024","interfaces":[{"interface":"back","average":{"1min":0.56299999999999994,"5min":0.56299999999999994,"15min":0.56299999999999994},"min":{"1min":0.35199999999999998,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.94699999999999995,"5min":0.94699999999999995,"15min":0.94699999999999995},"last":0.58699999999999997},{"interface":"front","average":{"1min":0.54800000000000004,"5min":0.54800000000000004,"15min":0.54800000000000004},"min":{"1min":0.34200000000000003,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.86299999999999999,"5min":0.86299999999999999,"15min":0.86299999999999999},"last":0.442}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-14T13:56:31.999 INFO:tasks.cephadm.ceph_manager.ceph:clean! 2024-04-14T13:56:31.999 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2024-04-14T13:56:31.999 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy 2024-04-14T13:56:31.999 DEBUG:teuthology.orchestra.run.smithi069:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid dc326fc2-fa65-11ee-bc8e-c7b262605968 -- ceph health --format=json 2024-04-14T13:56:32.567 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:56:32 smithi069 ceph-mon[24818]: from='client.14619 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-14T13:56:32.567 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:56:32 smithi069 ceph-mon[29472]: from='client.14619 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-14T13:56:32.595 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:56:32 smithi122 ceph-mon[25721]: from='client.14619 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-14T13:56:33.530 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:56:33 smithi069 ceph-mon[29472]: pgmap v37: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-14T13:56:33.530 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:56:33 smithi069 ceph-mon[24818]: pgmap v37: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-14T13:56:33.595 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:56:33 smithi122 ceph-mon[25721]: pgmap v37: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-14T13:56:34.087 INFO:teuthology.orchestra.run.smithi069.stderr:Inferring config /var/lib/ceph/dc326fc2-fa65-11ee-bc8e-c7b262605968/mon.a/config 2024-04-14T13:56:35.624 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:56:35 smithi069 ceph-mon[24818]: pgmap v38: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-14T13:56:35.625 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:56:35 smithi069 ceph-mon[29472]: pgmap v38: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-14T13:56:35.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:56:35 smithi122 ceph-mon[25721]: pgmap v38: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-14T13:56:36.078 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-14T13:56:36.078 INFO:teuthology.orchestra.run.smithi069.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-04-14T13:56:36.646 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:56:36 smithi069 ceph-mon[24818]: from='client.? 172.21.15.69:0/1694854703' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-04-14T13:56:36.646 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:56:36 smithi069 ceph-mon[29472]: from='client.? 172.21.15.69:0/1694854703' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-04-14T13:56:36.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:56:36 smithi122 ceph-mon[25721]: from='client.? 172.21.15.69:0/1694854703' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-04-14T13:56:36.967 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy done 2024-04-14T13:56:36.967 INFO:tasks.cephadm:Setup complete, yielding 2024-04-14T13:56:36.967 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-04-14T13:56:36.977 INFO:tasks.cephadm:Running commands on role mon.a host ubuntu@smithi069.front.sepia.ceph.com 2024-04-14T13:56:36.977 DEBUG:teuthology.orchestra.run.smithi069:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid dc326fc2-fa65-11ee-bc8e-c7b262605968 -- bash -c 'ceph config set mgr mgr/cephadm/use_repo_digest false --force' 2024-04-14T13:56:37.518 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:56:37 smithi122 ceph-mon[25721]: pgmap v39: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-14T13:56:37.518 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Apr 14 13:56:37 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-x[26913]: ::ffff:172.21.15.122 - - [14/Apr/2024:13:56:37] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-04-14T13:56:37.567 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:56:37 smithi069 ceph-mon[24818]: pgmap v39: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-14T13:56:37.567 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:56:37 smithi069 ceph-mon[29472]: pgmap v39: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-14T13:56:38.703 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:56:38 smithi069 ceph-mon[24818]: from='mgr.14448 172.21.15.69:0/453473650' entity='mgr.y' 2024-04-14T13:56:38.703 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:56:38 smithi069 ceph-mon[24818]: from='mgr.14448 172.21.15.69:0/453473650' entity='mgr.y' 2024-04-14T13:56:38.704 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:56:38 smithi069 ceph-mon[24818]: from='mgr.14448 172.21.15.69:0/453473650' entity='mgr.y' 2024-04-14T13:56:38.704 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:56:38 smithi069 ceph-mon[24818]: from='mgr.14448 172.21.15.69:0/453473650' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-04-14T13:56:38.704 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:56:38 smithi069 ceph-mon[24818]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-04-14T13:56:38.704 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:56:38 smithi069 ceph-mon[24818]: from='mgr.14448 172.21.15.69:0/453473650' entity='mgr.y' 2024-04-14T13:56:38.704 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:56:38 smithi069 ceph-mon[24818]: Deploying daemon grafana.a on smithi122 2024-04-14T13:56:38.704 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:56:38 smithi069 ceph-mon[29472]: from='mgr.14448 172.21.15.69:0/453473650' entity='mgr.y' 2024-04-14T13:56:38.704 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:56:38 smithi069 ceph-mon[29472]: from='mgr.14448 172.21.15.69:0/453473650' entity='mgr.y' 2024-04-14T13:56:38.704 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:56:38 smithi069 ceph-mon[29472]: from='mgr.14448 172.21.15.69:0/453473650' entity='mgr.y' 2024-04-14T13:56:38.704 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:56:38 smithi069 ceph-mon[29472]: from='mgr.14448 172.21.15.69:0/453473650' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-04-14T13:56:38.704 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:56:38 smithi069 ceph-mon[29472]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-04-14T13:56:38.705 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:56:38 smithi069 ceph-mon[29472]: from='mgr.14448 172.21.15.69:0/453473650' entity='mgr.y' 2024-04-14T13:56:38.705 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:56:38 smithi069 ceph-mon[29472]: Deploying daemon grafana.a on smithi122 2024-04-14T13:56:38.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:56:38 smithi122 ceph-mon[25721]: from='mgr.14448 172.21.15.69:0/453473650' entity='mgr.y' 2024-04-14T13:56:38.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:56:38 smithi122 ceph-mon[25721]: from='mgr.14448 172.21.15.69:0/453473650' entity='mgr.y' 2024-04-14T13:56:38.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:56:38 smithi122 ceph-mon[25721]: from='mgr.14448 172.21.15.69:0/453473650' entity='mgr.y' 2024-04-14T13:56:38.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:56:38 smithi122 ceph-mon[25721]: from='mgr.14448 172.21.15.69:0/453473650' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-04-14T13:56:38.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:56:38 smithi122 ceph-mon[25721]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-04-14T13:56:38.846 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:56:38 smithi122 ceph-mon[25721]: from='mgr.14448 172.21.15.69:0/453473650' entity='mgr.y' 2024-04-14T13:56:38.846 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:56:38 smithi122 ceph-mon[25721]: Deploying daemon grafana.a on smithi122 2024-04-14T13:56:39.565 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:56:39 smithi069 ceph-mon[24818]: pgmap v40: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-14T13:56:39.566 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:56:39 smithi069 ceph-mon[24818]: from='client.? 172.21.15.69:0/2818146678' entity='client.admin' 2024-04-14T13:56:39.566 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:56:39 smithi069 ceph-mon[24818]: from='mgr.14448 172.21.15.69:0/453473650' entity='mgr.y' 2024-04-14T13:56:39.566 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:56:39 smithi069 ceph-mon[29472]: pgmap v40: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-14T13:56:39.566 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:56:39 smithi069 ceph-mon[29472]: from='client.? 172.21.15.69:0/2818146678' entity='client.admin' 2024-04-14T13:56:39.566 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:56:39 smithi069 ceph-mon[29472]: from='mgr.14448 172.21.15.69:0/453473650' entity='mgr.y' 2024-04-14T13:56:39.817 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 14 13:56:39 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-y[25054]: ::ffff:172.21.15.122 - - [14/Apr/2024:13:56:39] "GET /metrics HTTP/1.1" 200 191499 "" "Prometheus/2.33.4" 2024-04-14T13:56:39.841 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-04-14T13:56:39.856 INFO:tasks.cephadm:Running commands on role mon.a host ubuntu@smithi069.front.sepia.ceph.com 2024-04-14T13:56:39.857 DEBUG:teuthology.orchestra.run.smithi069:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid dc326fc2-fa65-11ee-bc8e-c7b262605968 -e sha1=a971fd5f5719038c70b817bafacd6d5400b1d888 -- bash -c 'radosgw-admin realm create --rgw-realm=r --default' 2024-04-14T13:56:39.858 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:56:39 smithi122 ceph-mon[25721]: pgmap v40: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-14T13:56:39.858 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:56:39 smithi122 ceph-mon[25721]: from='client.? 172.21.15.69:0/2818146678' entity='client.admin' 2024-04-14T13:56:39.858 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:56:39 smithi122 ceph-mon[25721]: from='mgr.14448 172.21.15.69:0/453473650' entity='mgr.y' 2024-04-14T13:56:41.396 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:56:41 smithi069 ceph-mon[24818]: pgmap v41: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-14T13:56:41.396 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:56:41 smithi069 ceph-mon[29472]: pgmap v41: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-14T13:56:41.595 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:56:41 smithi122 ceph-mon[25721]: pgmap v41: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-14T13:56:43.242 INFO:teuthology.orchestra.run.smithi069.stdout:{ 2024-04-14T13:56:43.242 INFO:teuthology.orchestra.run.smithi069.stdout: "id": "b0e084dc-c5d6-4c72-bfd7-73d3bab36e46", 2024-04-14T13:56:43.243 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "r", 2024-04-14T13:56:43.243 INFO:teuthology.orchestra.run.smithi069.stdout: "current_period": "0eee9837-2986-4a88-8277-5da8368ceedf", 2024-04-14T13:56:43.243 INFO:teuthology.orchestra.run.smithi069.stdout: "epoch": 1 2024-04-14T13:56:43.243 INFO:teuthology.orchestra.run.smithi069.stdout:} 2024-04-14T13:56:43.534 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:56:43 smithi069 ceph-mon[24818]: osdmap e60: 8 total, 8 up, 8 in 2024-04-14T13:56:43.534 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:56:43 smithi069 ceph-mon[24818]: from='client.? 172.21.15.69:0/2967078535' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]: dispatch 2024-04-14T13:56:43.534 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:56:43 smithi069 ceph-mon[24818]: pgmap v43: 33 pgs: 32 unknown, 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-14T13:56:43.535 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:56:43 smithi069 ceph-mon[29472]: osdmap e60: 8 total, 8 up, 8 in 2024-04-14T13:56:43.535 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:56:43 smithi069 ceph-mon[29472]: from='client.? 172.21.15.69:0/2967078535' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]: dispatch 2024-04-14T13:56:43.535 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:56:43 smithi069 ceph-mon[29472]: pgmap v43: 33 pgs: 32 unknown, 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-14T13:56:43.595 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:56:43 smithi122 ceph-mon[25721]: osdmap e60: 8 total, 8 up, 8 in 2024-04-14T13:56:43.595 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:56:43 smithi122 ceph-mon[25721]: from='client.? 172.21.15.69:0/2967078535' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]: dispatch 2024-04-14T13:56:43.595 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:56:43 smithi122 ceph-mon[25721]: pgmap v43: 33 pgs: 32 unknown, 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-14T13:56:43.864 DEBUG:teuthology.orchestra.run.smithi069:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid dc326fc2-fa65-11ee-bc8e-c7b262605968 -e sha1=a971fd5f5719038c70b817bafacd6d5400b1d888 -- bash -c 'radosgw-admin zonegroup create --rgw-zonegroup=default --master --default' 2024-04-14T13:56:44.425 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:56:44 smithi069 ceph-mon[24818]: from='client.? 172.21.15.69:0/2967078535' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]': finished 2024-04-14T13:56:44.426 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:56:44 smithi069 ceph-mon[24818]: osdmap e61: 8 total, 8 up, 8 in 2024-04-14T13:56:44.426 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:56:44 smithi069 ceph-mon[29472]: from='client.? 172.21.15.69:0/2967078535' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]': finished 2024-04-14T13:56:44.426 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:56:44 smithi069 ceph-mon[29472]: osdmap e61: 8 total, 8 up, 8 in 2024-04-14T13:56:44.595 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:56:44 smithi122 ceph-mon[25721]: from='client.? 172.21.15.69:0/2967078535' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]': finished 2024-04-14T13:56:44.595 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:56:44 smithi122 ceph-mon[25721]: osdmap e61: 8 total, 8 up, 8 in 2024-04-14T13:56:45.422 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:56:45 smithi069 ceph-mon[24818]: osdmap e62: 8 total, 8 up, 8 in 2024-04-14T13:56:45.422 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:56:45 smithi069 ceph-mon[24818]: pgmap v46: 33 pgs: 32 unknown, 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-14T13:56:45.423 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:56:45 smithi069 ceph-mon[29472]: osdmap e62: 8 total, 8 up, 8 in 2024-04-14T13:56:45.423 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:56:45 smithi069 ceph-mon[29472]: pgmap v46: 33 pgs: 32 unknown, 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-14T13:56:45.452 INFO:teuthology.orchestra.run.smithi069.stdout:{ 2024-04-14T13:56:45.452 INFO:teuthology.orchestra.run.smithi069.stdout: "id": "ed3c12f2-b3fa-4f9e-ba8c-e639cb70a4bf", 2024-04-14T13:56:45.452 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "default", 2024-04-14T13:56:45.452 INFO:teuthology.orchestra.run.smithi069.stdout: "api_name": "default", 2024-04-14T13:56:45.452 INFO:teuthology.orchestra.run.smithi069.stdout: "is_master": "true", 2024-04-14T13:56:45.452 INFO:teuthology.orchestra.run.smithi069.stdout: "endpoints": [], 2024-04-14T13:56:45.452 INFO:teuthology.orchestra.run.smithi069.stdout: "hostnames": [], 2024-04-14T13:56:45.452 INFO:teuthology.orchestra.run.smithi069.stdout: "hostnames_s3website": [], 2024-04-14T13:56:45.453 INFO:teuthology.orchestra.run.smithi069.stdout: "master_zone": "", 2024-04-14T13:56:45.453 INFO:teuthology.orchestra.run.smithi069.stdout: "zones": [], 2024-04-14T13:56:45.453 INFO:teuthology.orchestra.run.smithi069.stdout: "placement_targets": [], 2024-04-14T13:56:45.453 INFO:teuthology.orchestra.run.smithi069.stdout: "default_placement": "", 2024-04-14T13:56:45.453 INFO:teuthology.orchestra.run.smithi069.stdout: "realm_id": "b0e084dc-c5d6-4c72-bfd7-73d3bab36e46", 2024-04-14T13:56:45.453 INFO:teuthology.orchestra.run.smithi069.stdout: "sync_policy": { 2024-04-14T13:56:45.453 INFO:teuthology.orchestra.run.smithi069.stdout: "groups": [] 2024-04-14T13:56:45.453 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-14T13:56:45.453 INFO:teuthology.orchestra.run.smithi069.stdout:} 2024-04-14T13:56:45.595 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:56:45 smithi122 ceph-mon[25721]: osdmap e62: 8 total, 8 up, 8 in 2024-04-14T13:56:45.595 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:56:45 smithi122 ceph-mon[25721]: pgmap v46: 33 pgs: 32 unknown, 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-14T13:56:46.055 DEBUG:teuthology.orchestra.run.smithi069:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid dc326fc2-fa65-11ee-bc8e-c7b262605968 -e sha1=a971fd5f5719038c70b817bafacd6d5400b1d888 -- bash -c 'radosgw-admin zone create --rgw-zonegroup=default --rgw-zone=z --master --default' 2024-04-14T13:56:47.356 INFO:journalctl@ceph.alertmanager.a.smithi069.stdout:Apr 14 13:56:47 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-alertmanager-a[47876]: level=info ts=2024-04-14T13:56:47.265Z caller=cluster.go:688 component=cluster msg="gossip settled; proceeding" elapsed=10.003486015s 2024-04-14T13:56:47.506 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:56:47 smithi122 ceph-mon[25721]: pgmap v47: 33 pgs: 33 active+clean; 577 KiB data, 49 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s wr, 1 op/s 2024-04-14T13:56:47.506 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Apr 14 13:56:47 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-x[26913]: ::ffff:172.21.15.122 - - [14/Apr/2024:13:56:47] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-04-14T13:56:47.667 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:56:47 smithi069 ceph-mon[29472]: pgmap v47: 33 pgs: 33 active+clean; 577 KiB data, 49 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s wr, 1 op/s 2024-04-14T13:56:47.667 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:56:47 smithi069 ceph-mon[24818]: pgmap v47: 33 pgs: 33 active+clean; 577 KiB data, 49 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s wr, 1 op/s 2024-04-14T13:56:47.767 INFO:teuthology.orchestra.run.smithi069.stdout:{ 2024-04-14T13:56:47.767 INFO:teuthology.orchestra.run.smithi069.stdout: "id": "a4450052-7331-4c8a-956b-296946e49785", 2024-04-14T13:56:47.767 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "z", 2024-04-14T13:56:47.767 INFO:teuthology.orchestra.run.smithi069.stdout: "domain_root": "z.rgw.meta:root", 2024-04-14T13:56:47.767 INFO:teuthology.orchestra.run.smithi069.stdout: "control_pool": "z.rgw.control", 2024-04-14T13:56:47.767 INFO:teuthology.orchestra.run.smithi069.stdout: "gc_pool": "z.rgw.log:gc", 2024-04-14T13:56:47.767 INFO:teuthology.orchestra.run.smithi069.stdout: "lc_pool": "z.rgw.log:lc", 2024-04-14T13:56:47.767 INFO:teuthology.orchestra.run.smithi069.stdout: "log_pool": "z.rgw.log", 2024-04-14T13:56:47.767 INFO:teuthology.orchestra.run.smithi069.stdout: "intent_log_pool": "z.rgw.log:intent", 2024-04-14T13:56:47.767 INFO:teuthology.orchestra.run.smithi069.stdout: "usage_log_pool": "z.rgw.log:usage", 2024-04-14T13:56:47.768 INFO:teuthology.orchestra.run.smithi069.stdout: "roles_pool": "z.rgw.meta:roles", 2024-04-14T13:56:47.768 INFO:teuthology.orchestra.run.smithi069.stdout: "reshard_pool": "z.rgw.log:reshard", 2024-04-14T13:56:47.768 INFO:teuthology.orchestra.run.smithi069.stdout: "user_keys_pool": "z.rgw.meta:users.keys", 2024-04-14T13:56:47.768 INFO:teuthology.orchestra.run.smithi069.stdout: "user_email_pool": "z.rgw.meta:users.email", 2024-04-14T13:56:47.768 INFO:teuthology.orchestra.run.smithi069.stdout: "user_swift_pool": "z.rgw.meta:users.swift", 2024-04-14T13:56:47.768 INFO:teuthology.orchestra.run.smithi069.stdout: "user_uid_pool": "z.rgw.meta:users.uid", 2024-04-14T13:56:47.768 INFO:teuthology.orchestra.run.smithi069.stdout: "otp_pool": "z.rgw.otp", 2024-04-14T13:56:47.768 INFO:teuthology.orchestra.run.smithi069.stdout: "system_key": { 2024-04-14T13:56:47.768 INFO:teuthology.orchestra.run.smithi069.stdout: "access_key": "", 2024-04-14T13:56:47.768 INFO:teuthology.orchestra.run.smithi069.stdout: "secret_key": "" 2024-04-14T13:56:47.768 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-14T13:56:47.768 INFO:teuthology.orchestra.run.smithi069.stdout: "placement_pools": [ 2024-04-14T13:56:47.768 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-14T13:56:47.768 INFO:teuthology.orchestra.run.smithi069.stdout: "key": "default-placement", 2024-04-14T13:56:47.769 INFO:teuthology.orchestra.run.smithi069.stdout: "val": { 2024-04-14T13:56:47.769 INFO:teuthology.orchestra.run.smithi069.stdout: "index_pool": "z.rgw.buckets.index", 2024-04-14T13:56:47.769 INFO:teuthology.orchestra.run.smithi069.stdout: "storage_classes": { 2024-04-14T13:56:47.769 INFO:teuthology.orchestra.run.smithi069.stdout: "STANDARD": { 2024-04-14T13:56:47.769 INFO:teuthology.orchestra.run.smithi069.stdout: "data_pool": "z.rgw.buckets.data" 2024-04-14T13:56:47.769 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-14T13:56:47.769 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-14T13:56:47.769 INFO:teuthology.orchestra.run.smithi069.stdout: "data_extra_pool": "z.rgw.buckets.non-ec", 2024-04-14T13:56:47.769 INFO:teuthology.orchestra.run.smithi069.stdout: "index_type": 0 2024-04-14T13:56:47.769 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-14T13:56:47.769 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-14T13:56:47.769 INFO:teuthology.orchestra.run.smithi069.stdout: ], 2024-04-14T13:56:47.769 INFO:teuthology.orchestra.run.smithi069.stdout: "realm_id": "b0e084dc-c5d6-4c72-bfd7-73d3bab36e46", 2024-04-14T13:56:47.769 INFO:teuthology.orchestra.run.smithi069.stdout: "notif_pool": "z.rgw.log:notif" 2024-04-14T13:56:47.769 INFO:teuthology.orchestra.run.smithi069.stdout:} 2024-04-14T13:56:48.319 DEBUG:teuthology.orchestra.run.smithi069:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid dc326fc2-fa65-11ee-bc8e-c7b262605968 -e sha1=a971fd5f5719038c70b817bafacd6d5400b1d888 -- bash -c 'radosgw-admin period update --rgw-realm=r --commit' 2024-04-14T13:56:49.595 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:56:49 smithi122 ceph-mon[25721]: pgmap v48: 33 pgs: 33 active+clean; 577 KiB data, 49 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s wr, 1 op/s 2024-04-14T13:56:49.651 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:56:49 smithi069 ceph-mon[24818]: pgmap v48: 33 pgs: 33 active+clean; 577 KiB data, 49 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s wr, 1 op/s 2024-04-14T13:56:49.652 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 14 13:56:49 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-y[25054]: ::ffff:172.21.15.122 - - [14/Apr/2024:13:56:49] "GET /metrics HTTP/1.1" 200 191499 "" "Prometheus/2.33.4" 2024-04-14T13:56:49.652 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:56:49 smithi069 ceph-mon[29472]: pgmap v48: 33 pgs: 33 active+clean; 577 KiB data, 49 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s wr, 1 op/s 2024-04-14T13:56:50.345 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Apr 14 13:56:50 smithi122 podman[41842]: 2024-04-14 13:56:50.277504777 +0000 UTC m=+0.147289789 container create 6c0d806d20ea2df3985053e7ce0ab16ace983a6480fb12b94b7b65c9fa414f72 (image=quay.io/ceph/ceph-grafana:8.3.5, name=ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-grafana-a, version=8.5, vendor=Red Hat, Inc., io.buildah.version=1.24.2, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., io.k8s.display-name=Red Hat Universal Base Image 8, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, name=ubi8, io.openshift.expose-services=, vcs-type=git, description=Ceph Grafana Container, architecture=x86_64, summary=Grafana Container configured for Ceph mgr/dashboard integration, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, io.openshift.tags=base rhel8, build-date=2022-03-28T10:36:18.413762, release=236.1648460182, com.redhat.component=ubi8-container, distribution-scope=public, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-236.1648460182, maintainer=Paul Cuzner ) 2024-04-14T13:56:50.658 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Apr 14 13:56:50 smithi122 podman[41842]: 2024-04-14 13:56:50.563541001 +0000 UTC m=+0.433326013 container init 6c0d806d20ea2df3985053e7ce0ab16ace983a6480fb12b94b7b65c9fa414f72 (image=quay.io/ceph/ceph-grafana:8.3.5, name=ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-grafana-a, release=236.1648460182, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, io.k8s.display-name=Red Hat Universal Base Image 8, distribution-scope=public, io.openshift.expose-services=, vendor=Red Hat, Inc., io.openshift.tags=base rhel8, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, build-date=2022-03-28T10:36:18.413762, vcs-type=git, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, io.buildah.version=1.24.2, architecture=x86_64, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-236.1648460182, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., summary=Grafana Container configured for Ceph mgr/dashboard integration, com.redhat.component=ubi8-container, version=8.5, description=Ceph Grafana Container, maintainer=Paul Cuzner , name=ubi8) 2024-04-14T13:56:50.658 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Apr 14 13:56:50 smithi122 podman[41842]: 2024-04-14 13:56:50.566167529 +0000 UTC m=+0.435952542 container start 6c0d806d20ea2df3985053e7ce0ab16ace983a6480fb12b94b7b65c9fa414f72 (image=quay.io/ceph/ceph-grafana:8.3.5, name=ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-grafana-a, build-date=2022-03-28T10:36:18.413762, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., architecture=x86_64, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, maintainer=Paul Cuzner , io.openshift.tags=base rhel8, vendor=Red Hat, Inc., distribution-scope=public, summary=Grafana Container configured for Ceph mgr/dashboard integration, name=ubi8, io.buildah.version=1.24.2, com.redhat.component=ubi8-container, version=8.5, description=Ceph Grafana Container, io.k8s.display-name=Red Hat Universal Base Image 8, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, io.openshift.expose-services=, release=236.1648460182, vcs-type=git, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-236.1648460182) 2024-04-14T13:56:50.658 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Apr 14 13:56:50 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-grafana-a[41857]: t=2024-04-14T13:56:50+0000 lvl=info msg="The state of unified alerting is still not defined. The decision will be made during as we run the database migrations" logger=settings 2024-04-14T13:56:50.658 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Apr 14 13:56:50 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-grafana-a[41857]: t=2024-04-14T13:56:50+0000 lvl=warn msg="falling back to legacy setting of 'min_interval_seconds'; please use the configuration option in the `unified_alerting` section if Grafana 8 alerts are enabled." logger=settings 2024-04-14T13:56:50.659 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Apr 14 13:56:50 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-grafana-a[41857]: t=2024-04-14T13:56:50+0000 lvl=info msg="Config loaded from" logger=settings file=/usr/share/grafana/conf/defaults.ini 2024-04-14T13:56:50.659 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Apr 14 13:56:50 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-grafana-a[41857]: t=2024-04-14T13:56:50+0000 lvl=info msg="Config loaded from" logger=settings file=/etc/grafana/grafana.ini 2024-04-14T13:56:50.659 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Apr 14 13:56:50 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-grafana-a[41857]: t=2024-04-14T13:56:50+0000 lvl=info msg="Config overridden from Environment variable" logger=settings var="GF_PATHS_DATA=/var/lib/grafana" 2024-04-14T13:56:50.659 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Apr 14 13:56:50 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-grafana-a[41857]: t=2024-04-14T13:56:50+0000 lvl=info msg="Config overridden from Environment variable" logger=settings var="GF_PATHS_LOGS=/var/log/grafana" 2024-04-14T13:56:50.659 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Apr 14 13:56:50 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-grafana-a[41857]: t=2024-04-14T13:56:50+0000 lvl=info msg="Config overridden from Environment variable" logger=settings var="GF_PATHS_PLUGINS=/var/lib/grafana/plugins" 2024-04-14T13:56:50.660 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Apr 14 13:56:50 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-grafana-a[41857]: t=2024-04-14T13:56:50+0000 lvl=info msg="Config overridden from Environment variable" logger=settings var="GF_PATHS_PROVISIONING=/etc/grafana/provisioning" 2024-04-14T13:56:50.660 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Apr 14 13:56:50 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-grafana-a[41857]: t=2024-04-14T13:56:50+0000 lvl=info msg="Path Home" logger=settings path=/usr/share/grafana 2024-04-14T13:56:50.660 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Apr 14 13:56:50 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-grafana-a[41857]: t=2024-04-14T13:56:50+0000 lvl=info msg="Path Data" logger=settings path=/var/lib/grafana 2024-04-14T13:56:50.660 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Apr 14 13:56:50 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-grafana-a[41857]: t=2024-04-14T13:56:50+0000 lvl=info msg="Path Logs" logger=settings path=/var/log/grafana 2024-04-14T13:56:50.660 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Apr 14 13:56:50 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-grafana-a[41857]: t=2024-04-14T13:56:50+0000 lvl=info msg="Path Plugins" logger=settings path=/var/lib/grafana/plugins 2024-04-14T13:56:50.660 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Apr 14 13:56:50 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-grafana-a[41857]: t=2024-04-14T13:56:50+0000 lvl=info msg="Path Provisioning" logger=settings path=/etc/grafana/provisioning 2024-04-14T13:56:50.660 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Apr 14 13:56:50 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-grafana-a[41857]: t=2024-04-14T13:56:50+0000 lvl=info msg="App mode production" logger=settings 2024-04-14T13:56:50.660 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Apr 14 13:56:50 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-grafana-a[41857]: t=2024-04-14T13:56:50+0000 lvl=info msg="Connecting to DB" logger=sqlstore dbtype=sqlite3 2024-04-14T13:56:50.660 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Apr 14 13:56:50 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-grafana-a[41857]: t=2024-04-14T13:56:50+0000 lvl=warn msg="SQLite database file has broader permissions than it should" logger=sqlstore path=/var/lib/grafana/grafana.db mode=-rw-r--r-- expected=-rw-r----- 2024-04-14T13:56:50.660 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Apr 14 13:56:50 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-grafana-a[41857]: t=2024-04-14T13:56:50+0000 lvl=info msg="Starting DB migrations" logger=migrator 2024-04-14T13:56:50.660 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Apr 14 13:56:50 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-grafana-a[41857]: t=2024-04-14T13:56:50+0000 lvl=info msg="Executing migration" logger=migrator id="create migration_log table" 2024-04-14T13:56:50.660 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Apr 14 13:56:50 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-grafana-a[41857]: t=2024-04-14T13:56:50+0000 lvl=info msg="Executing migration" logger=migrator id="create user table" 2024-04-14T13:56:50.660 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Apr 14 13:56:50 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-grafana-a[41857]: t=2024-04-14T13:56:50+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index user.login" 2024-04-14T13:56:50.660 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Apr 14 13:56:50 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-grafana-a[41857]: t=2024-04-14T13:56:50+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index user.email" 2024-04-14T13:56:50.661 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Apr 14 13:56:50 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-grafana-a[41857]: t=2024-04-14T13:56:50+0000 lvl=info msg="Executing migration" logger=migrator id="drop index UQE_user_login - v1" 2024-04-14T13:56:50.661 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Apr 14 13:56:50 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-grafana-a[41857]: t=2024-04-14T13:56:50+0000 lvl=info msg="Executing migration" logger=migrator id="drop index UQE_user_email - v1" 2024-04-14T13:56:50.661 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Apr 14 13:56:50 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-grafana-a[41857]: t=2024-04-14T13:56:50+0000 lvl=info msg="Executing migration" logger=migrator id="Rename table user to user_v1 - v1" 2024-04-14T13:56:50.661 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Apr 14 13:56:50 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-grafana-a[41857]: t=2024-04-14T13:56:50+0000 lvl=info msg="Executing migration" logger=migrator id="create user table v2" 2024-04-14T13:56:50.661 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Apr 14 13:56:50 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-grafana-a[41857]: t=2024-04-14T13:56:50+0000 lvl=info msg="Executing migration" logger=migrator id="create index UQE_user_login - v2" 2024-04-14T13:56:50.661 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Apr 14 13:56:50 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-grafana-a[41857]: t=2024-04-14T13:56:50+0000 lvl=info msg="Executing migration" logger=migrator id="create index UQE_user_email - v2" 2024-04-14T13:56:50.661 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Apr 14 13:56:50 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-grafana-a[41857]: t=2024-04-14T13:56:50+0000 lvl=info msg="Executing migration" logger=migrator id="copy data_source v1 to v2" 2024-04-14T13:56:50.661 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Apr 14 13:56:50 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-grafana-a[41857]: t=2024-04-14T13:56:50+0000 lvl=info msg="Executing migration" logger=migrator id="Drop old table user_v1" 2024-04-14T13:56:50.661 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Apr 14 13:56:50 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-grafana-a[41857]: t=2024-04-14T13:56:50+0000 lvl=info msg="Executing migration" logger=migrator id="Add column help_flags1 to user table" 2024-04-14T13:56:50.661 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Apr 14 13:56:50 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-grafana-a[41857]: t=2024-04-14T13:56:50+0000 lvl=info msg="Executing migration" logger=migrator id="Update user table charset" 2024-04-14T13:56:50.661 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Apr 14 13:56:50 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-grafana-a[41857]: t=2024-04-14T13:56:50+0000 lvl=info msg="Executing migration" logger=migrator id="Add last_seen_at column to user" 2024-04-14T13:56:50.661 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Apr 14 13:56:50 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-grafana-a[41857]: t=2024-04-14T13:56:50+0000 lvl=info msg="Executing migration" logger=migrator id="Add missing user data" 2024-04-14T13:56:50.661 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Apr 14 13:56:50 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-grafana-a[41857]: t=2024-04-14T13:56:50+0000 lvl=info msg="Executing migration" logger=migrator id="Add is_disabled column to user" 2024-04-14T13:56:50.661 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Apr 14 13:56:50 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-grafana-a[41857]: t=2024-04-14T13:56:50+0000 lvl=info msg="Executing migration" logger=migrator id="Add index user.login/user.email" 2024-04-14T13:56:50.662 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Apr 14 13:56:50 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-grafana-a[41857]: t=2024-04-14T13:56:50+0000 lvl=info msg="Executing migration" logger=migrator id="Add is_service_account column to user" 2024-04-14T13:56:50.662 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Apr 14 13:56:50 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-grafana-a[41857]: t=2024-04-14T13:56:50+0000 lvl=info msg="Executing migration" logger=migrator id="create temp user table v1-7" 2024-04-14T13:56:50.662 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Apr 14 13:56:50 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-grafana-a[41857]: t=2024-04-14T13:56:50+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_temp_user_email - v1-7" 2024-04-14T13:56:50.662 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Apr 14 13:56:50 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-grafana-a[41857]: t=2024-04-14T13:56:50+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_temp_user_org_id - v1-7" 2024-04-14T13:56:50.662 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Apr 14 13:56:50 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-grafana-a[41857]: t=2024-04-14T13:56:50+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_temp_user_code - v1-7" 2024-04-14T13:56:50.662 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Apr 14 13:56:50 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-grafana-a[41857]: t=2024-04-14T13:56:50+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_temp_user_status - v1-7" 2024-04-14T13:56:50.662 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Apr 14 13:56:50 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-grafana-a[41857]: t=2024-04-14T13:56:50+0000 lvl=info msg="Executing migration" logger=migrator id="Update temp_user table charset" 2024-04-14T13:56:50.662 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Apr 14 13:56:50 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-grafana-a[41857]: t=2024-04-14T13:56:50+0000 lvl=info msg="Executing migration" logger=migrator id="drop index IDX_temp_user_email - v1" 2024-04-14T13:56:50.662 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Apr 14 13:56:50 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-grafana-a[41857]: t=2024-04-14T13:56:50+0000 lvl=info msg="Executing migration" logger=migrator id="drop index IDX_temp_user_org_id - v1" 2024-04-14T13:56:50.662 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Apr 14 13:56:50 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-grafana-a[41857]: t=2024-04-14T13:56:50+0000 lvl=info msg="Executing migration" logger=migrator id="drop index IDX_temp_user_code - v1" 2024-04-14T13:56:50.663 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Apr 14 13:56:50 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-grafana-a[41857]: t=2024-04-14T13:56:50+0000 lvl=info msg="Executing migration" logger=migrator id="drop index IDX_temp_user_status - v1" 2024-04-14T13:56:50.663 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Apr 14 13:56:50 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-grafana-a[41857]: t=2024-04-14T13:56:50+0000 lvl=info msg="Executing migration" logger=migrator id="Rename table temp_user to temp_user_tmp_qwerty - v1" 2024-04-14T13:56:50.663 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Apr 14 13:56:50 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-grafana-a[41857]: t=2024-04-14T13:56:50+0000 lvl=info msg="Executing migration" logger=migrator id="create temp_user v2" 2024-04-14T13:56:50.663 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Apr 14 13:56:50 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-grafana-a[41857]: t=2024-04-14T13:56:50+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_temp_user_email - v2" 2024-04-14T13:56:50.663 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Apr 14 13:56:50 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-grafana-a[41857]: t=2024-04-14T13:56:50+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_temp_user_org_id - v2" 2024-04-14T13:56:50.663 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Apr 14 13:56:50 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-grafana-a[41857]: t=2024-04-14T13:56:50+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_temp_user_code - v2" 2024-04-14T13:56:50.663 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Apr 14 13:56:50 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-grafana-a[41857]: t=2024-04-14T13:56:50+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_temp_user_status - v2" 2024-04-14T13:56:50.663 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Apr 14 13:56:50 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-grafana-a[41857]: t=2024-04-14T13:56:50+0000 lvl=info msg="Executing migration" logger=migrator id="copy temp_user v1 to v2" 2024-04-14T13:56:50.664 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Apr 14 13:56:50 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-grafana-a[41857]: t=2024-04-14T13:56:50+0000 lvl=info msg="Executing migration" logger=migrator id="drop temp_user_tmp_qwerty" 2024-04-14T13:56:50.664 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Apr 14 13:56:50 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-grafana-a[41857]: t=2024-04-14T13:56:50+0000 lvl=info msg="Executing migration" logger=migrator id="Set created for temp users that will otherwise prematurely expire" 2024-04-14T13:56:50.664 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Apr 14 13:56:50 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-grafana-a[41857]: t=2024-04-14T13:56:50+0000 lvl=info msg="Executing migration" logger=migrator id="create star table" 2024-04-14T13:56:50.664 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Apr 14 13:56:50 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-grafana-a[41857]: t=2024-04-14T13:56:50+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index star.user_id_dashboard_id" 2024-04-14T13:56:50.664 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Apr 14 13:56:50 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-grafana-a[41857]: t=2024-04-14T13:56:50+0000 lvl=info msg="Executing migration" logger=migrator id="create org table v1" 2024-04-14T13:56:50.664 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Apr 14 13:56:50 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-grafana-a[41857]: t=2024-04-14T13:56:50+0000 lvl=info msg="Executing migration" logger=migrator id="create index UQE_org_name - v1" 2024-04-14T13:56:50.664 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Apr 14 13:56:50 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-grafana-a[41857]: t=2024-04-14T13:56:50+0000 lvl=info msg="Executing migration" logger=migrator id="create org_user table v1" 2024-04-14T13:56:50.664 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Apr 14 13:56:50 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-grafana-a[41857]: t=2024-04-14T13:56:50+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_org_user_org_id - v1" 2024-04-14T13:56:50.664 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Apr 14 13:56:50 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-grafana-a[41857]: t=2024-04-14T13:56:50+0000 lvl=info msg="Executing migration" logger=migrator id="create index UQE_org_user_org_id_user_id - v1" 2024-04-14T13:56:50.664 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Apr 14 13:56:50 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-grafana-a[41857]: t=2024-04-14T13:56:50+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_org_user_user_id - v1" 2024-04-14T13:56:50.664 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Apr 14 13:56:50 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-grafana-a[41857]: t=2024-04-14T13:56:50+0000 lvl=info msg="Executing migration" logger=migrator id="Update org table charset" 2024-04-14T13:56:50.664 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Apr 14 13:56:50 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-grafana-a[41857]: t=2024-04-14T13:56:50+0000 lvl=info msg="Executing migration" logger=migrator id="Update org_user table charset" 2024-04-14T13:56:50.664 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Apr 14 13:56:50 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-grafana-a[41857]: t=2024-04-14T13:56:50+0000 lvl=info msg="Executing migration" logger=migrator id="Migrate all Read Only Viewers to Viewers" 2024-04-14T13:56:50.665 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Apr 14 13:56:50 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-grafana-a[41857]: t=2024-04-14T13:56:50+0000 lvl=info msg="Executing migration" logger=migrator id="create dashboard table" 2024-04-14T13:56:50.665 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Apr 14 13:56:50 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-grafana-a[41857]: t=2024-04-14T13:56:50+0000 lvl=info msg="Executing migration" logger=migrator id="add index dashboard.account_id" 2024-04-14T13:56:50.665 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Apr 14 13:56:50 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-grafana-a[41857]: t=2024-04-14T13:56:50+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index dashboard_account_id_slug" 2024-04-14T13:56:50.665 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Apr 14 13:56:50 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-grafana-a[41857]: t=2024-04-14T13:56:50+0000 lvl=info msg="Executing migration" logger=migrator id="create dashboard_tag table" 2024-04-14T13:56:50.665 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Apr 14 13:56:50 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-grafana-a[41857]: t=2024-04-14T13:56:50+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index dashboard_tag.dasboard_id_term" 2024-04-14T13:56:50.665 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Apr 14 13:56:50 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-grafana-a[41857]: t=2024-04-14T13:56:50+0000 lvl=info msg="Executing migration" logger=migrator id="drop index UQE_dashboard_tag_dashboard_id_term - v1" 2024-04-14T13:56:50.666 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Apr 14 13:56:50 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-grafana-a[41857]: t=2024-04-14T13:56:50+0000 lvl=info msg="Executing migration" logger=migrator id="Rename table dashboard to dashboard_v1 - v1" 2024-04-14T13:56:50.666 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Apr 14 13:56:50 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-grafana-a[41857]: t=2024-04-14T13:56:50+0000 lvl=info msg="Executing migration" logger=migrator id="create dashboard v2" 2024-04-14T13:56:50.666 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Apr 14 13:56:50 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-grafana-a[41857]: t=2024-04-14T13:56:50+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_dashboard_org_id - v2" 2024-04-14T13:56:50.666 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Apr 14 13:56:50 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-grafana-a[41857]: t=2024-04-14T13:56:50+0000 lvl=info msg="Executing migration" logger=migrator id="create index UQE_dashboard_org_id_slug - v2" 2024-04-14T13:56:50.668 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Apr 14 13:56:50 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-grafana-a[41857]: t=2024-04-14T13:56:50+0000 lvl=info msg="Executing migration" logger=migrator id="copy dashboard v1 to v2" 2024-04-14T13:56:50.668 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Apr 14 13:56:50 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-grafana-a[41857]: t=2024-04-14T13:56:50+0000 lvl=info msg="Executing migration" logger=migrator id="drop table dashboard_v1" 2024-04-14T13:56:50.668 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Apr 14 13:56:50 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-grafana-a[41857]: t=2024-04-14T13:56:50+0000 lvl=info msg="Executing migration" logger=migrator id="alter dashboard.data to mediumtext v1" 2024-04-14T13:56:50.668 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Apr 14 13:56:50 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-grafana-a[41857]: t=2024-04-14T13:56:50+0000 lvl=info msg="Executing migration" logger=migrator id="Add column updated_by in dashboard - v2" 2024-04-14T13:56:50.668 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Apr 14 13:56:50 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-grafana-a[41857]: t=2024-04-14T13:56:50+0000 lvl=info msg="Executing migration" logger=migrator id="Add column created_by in dashboard - v2" 2024-04-14T13:56:50.668 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Apr 14 13:56:50 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-grafana-a[41857]: t=2024-04-14T13:56:50+0000 lvl=info msg="Executing migration" logger=migrator id="Add column gnetId in dashboard" 2024-04-14T13:56:50.668 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Apr 14 13:56:50 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-grafana-a[41857]: t=2024-04-14T13:56:50+0000 lvl=info msg="Executing migration" logger=migrator id="Add index for gnetId in dashboard" 2024-04-14T13:56:50.668 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Apr 14 13:56:50 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-grafana-a[41857]: t=2024-04-14T13:56:50+0000 lvl=info msg="Executing migration" logger=migrator id="Add column plugin_id in dashboard" 2024-04-14T13:56:50.668 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Apr 14 13:56:50 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-grafana-a[41857]: t=2024-04-14T13:56:50+0000 lvl=info msg="Executing migration" logger=migrator id="Add index for plugin_id in dashboard" 2024-04-14T13:56:50.669 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Apr 14 13:56:50 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-grafana-a[41857]: t=2024-04-14T13:56:50+0000 lvl=info msg="Executing migration" logger=migrator id="Add index for dashboard_id in dashboard_tag" 2024-04-14T13:56:50.669 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Apr 14 13:56:50 smithi122 bash[41842]: 6c0d806d20ea2df3985053e7ce0ab16ace983a6480fb12b94b7b65c9fa414f72 2024-04-14T13:56:50.669 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Apr 14 13:56:50 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-grafana-a[41857]: t=2024-04-14T13:56:50+0000 lvl=info msg="Executing migration" logger=migrator id="Update dashboard table charset" 2024-04-14T13:56:50.669 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Apr 14 13:56:50 smithi122 systemd[1]: Started Ceph grafana.a for dc326fc2-fa65-11ee-bc8e-c7b262605968. 2024-04-14T13:56:50.669 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Apr 14 13:56:50 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-grafana-a[41857]: t=2024-04-14T13:56:50+0000 lvl=info msg="Executing migration" logger=migrator id="Update dashboard_tag table charset" 2024-04-14T13:56:50.670 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Apr 14 13:56:50 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-grafana-a[41857]: t=2024-04-14T13:56:50+0000 lvl=info msg="Executing migration" logger=migrator id="Add column folder_id in dashboard" 2024-04-14T13:56:50.670 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Apr 14 13:56:50 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-grafana-a[41857]: t=2024-04-14T13:56:50+0000 lvl=info msg="Executing migration" logger=migrator id="Add column isFolder in dashboard" 2024-04-14T13:56:50.670 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Apr 14 13:56:50 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-grafana-a[41857]: t=2024-04-14T13:56:50+0000 lvl=info msg="Executing migration" logger=migrator id="Add column has_acl in dashboard" 2024-04-14T13:56:50.910 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Apr 14 13:56:50 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-grafana-a[41857]: t=2024-04-14T13:56:50+0000 lvl=info msg="Executing migration" logger=migrator id="Add column uid in dashboard" 2024-04-14T13:56:50.910 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Apr 14 13:56:50 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-grafana-a[41857]: t=2024-04-14T13:56:50+0000 lvl=info msg="Executing migration" logger=migrator id="Update uid column values in dashboard" 2024-04-14T13:56:50.911 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Apr 14 13:56:50 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-grafana-a[41857]: t=2024-04-14T13:56:50+0000 lvl=info msg="Executing migration" logger=migrator id="Add unique index dashboard_org_id_uid" 2024-04-14T13:56:50.911 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Apr 14 13:56:50 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-grafana-a[41857]: t=2024-04-14T13:56:50+0000 lvl=info msg="Executing migration" logger=migrator id="Remove unique index org_id_slug" 2024-04-14T13:56:50.911 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Apr 14 13:56:50 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-grafana-a[41857]: t=2024-04-14T13:56:50+0000 lvl=info msg="Executing migration" logger=migrator id="Update dashboard title length" 2024-04-14T13:56:50.911 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Apr 14 13:56:50 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-grafana-a[41857]: t=2024-04-14T13:56:50+0000 lvl=info msg="Executing migration" logger=migrator id="Add unique index for dashboard_org_id_title_folder_id" 2024-04-14T13:56:50.911 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Apr 14 13:56:50 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-grafana-a[41857]: t=2024-04-14T13:56:50+0000 lvl=info msg="Executing migration" logger=migrator id="create dashboard_provisioning" 2024-04-14T13:56:50.911 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Apr 14 13:56:50 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-grafana-a[41857]: t=2024-04-14T13:56:50+0000 lvl=info msg="Executing migration" logger=migrator id="Rename table dashboard_provisioning to dashboard_provisioning_tmp_qwerty - v1" 2024-04-14T13:56:50.911 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Apr 14 13:56:50 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-grafana-a[41857]: t=2024-04-14T13:56:50+0000 lvl=info msg="Executing migration" logger=migrator id="create dashboard_provisioning v2" 2024-04-14T13:56:50.912 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Apr 14 13:56:50 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-grafana-a[41857]: t=2024-04-14T13:56:50+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_dashboard_provisioning_dashboard_id - v2" 2024-04-14T13:56:50.912 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Apr 14 13:56:50 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-grafana-a[41857]: t=2024-04-14T13:56:50+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_dashboard_provisioning_dashboard_id_name - v2" 2024-04-14T13:56:50.912 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Apr 14 13:56:50 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-grafana-a[41857]: t=2024-04-14T13:56:50+0000 lvl=info msg="Executing migration" logger=migrator id="copy dashboard_provisioning v1 to v2" 2024-04-14T13:56:50.912 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Apr 14 13:56:50 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-grafana-a[41857]: t=2024-04-14T13:56:50+0000 lvl=info msg="Executing migration" logger=migrator id="drop dashboard_provisioning_tmp_qwerty" 2024-04-14T13:56:50.912 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Apr 14 13:56:50 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-grafana-a[41857]: t=2024-04-14T13:56:50+0000 lvl=info msg="Executing migration" logger=migrator id="Add check_sum column" 2024-04-14T13:56:50.912 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Apr 14 13:56:50 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-grafana-a[41857]: t=2024-04-14T13:56:50+0000 lvl=info msg="Executing migration" logger=migrator id="Add index for dashboard_title" 2024-04-14T13:56:50.913 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Apr 14 13:56:50 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-grafana-a[41857]: t=2024-04-14T13:56:50+0000 lvl=info msg="Executing migration" logger=migrator id="delete tags for deleted dashboards" 2024-04-14T13:56:50.913 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Apr 14 13:56:50 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-grafana-a[41857]: t=2024-04-14T13:56:50+0000 lvl=info msg="Executing migration" logger=migrator id="delete stars for deleted dashboards" 2024-04-14T13:56:50.913 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Apr 14 13:56:50 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-grafana-a[41857]: t=2024-04-14T13:56:50+0000 lvl=info msg="Executing migration" logger=migrator id="Add index for dashboard_is_folder" 2024-04-14T13:56:50.913 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Apr 14 13:56:50 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-grafana-a[41857]: t=2024-04-14T13:56:50+0000 lvl=info msg="Executing migration" logger=migrator id="create data_source table" 2024-04-14T13:56:50.913 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Apr 14 13:56:50 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-grafana-a[41857]: t=2024-04-14T13:56:50+0000 lvl=info msg="Executing migration" logger=migrator id="add index data_source.account_id" 2024-04-14T13:56:50.913 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Apr 14 13:56:50 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-grafana-a[41857]: t=2024-04-14T13:56:50+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index data_source.account_id_name" 2024-04-14T13:56:50.913 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Apr 14 13:56:50 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-grafana-a[41857]: t=2024-04-14T13:56:50+0000 lvl=info msg="Executing migration" logger=migrator id="drop index IDX_data_source_account_id - v1" 2024-04-14T13:56:50.914 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Apr 14 13:56:50 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-grafana-a[41857]: t=2024-04-14T13:56:50+0000 lvl=info msg="Executing migration" logger=migrator id="drop index UQE_data_source_account_id_name - v1" 2024-04-14T13:56:50.914 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Apr 14 13:56:50 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-grafana-a[41857]: t=2024-04-14T13:56:50+0000 lvl=info msg="Executing migration" logger=migrator id="Rename table data_source to data_source_v1 - v1" 2024-04-14T13:56:50.914 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Apr 14 13:56:50 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-grafana-a[41857]: t=2024-04-14T13:56:50+0000 lvl=info msg="Executing migration" logger=migrator id="create data_source table v2" 2024-04-14T13:56:50.914 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Apr 14 13:56:50 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-grafana-a[41857]: t=2024-04-14T13:56:50+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_data_source_org_id - v2" 2024-04-14T13:56:50.914 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Apr 14 13:56:50 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-grafana-a[41857]: t=2024-04-14T13:56:50+0000 lvl=info msg="Executing migration" logger=migrator id="create index UQE_data_source_org_id_name - v2" 2024-04-14T13:56:50.914 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Apr 14 13:56:50 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-grafana-a[41857]: t=2024-04-14T13:56:50+0000 lvl=info msg="Executing migration" logger=migrator id="copy data_source v1 to v2" 2024-04-14T13:56:50.914 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Apr 14 13:56:50 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-grafana-a[41857]: t=2024-04-14T13:56:50+0000 lvl=info msg="Executing migration" logger=migrator id="Drop old table data_source_v1 #2" 2024-04-14T13:56:50.915 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Apr 14 13:56:50 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-grafana-a[41857]: t=2024-04-14T13:56:50+0000 lvl=info msg="Executing migration" logger=migrator id="Add column with_credentials" 2024-04-14T13:56:50.915 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Apr 14 13:56:50 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-grafana-a[41857]: t=2024-04-14T13:56:50+0000 lvl=info msg="Executing migration" logger=migrator id="Add secure json data column" 2024-04-14T13:56:50.915 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Apr 14 13:56:50 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-grafana-a[41857]: t=2024-04-14T13:56:50+0000 lvl=info msg="Executing migration" logger=migrator id="Update data_source table charset" 2024-04-14T13:56:50.915 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Apr 14 13:56:50 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-grafana-a[41857]: t=2024-04-14T13:56:50+0000 lvl=info msg="Executing migration" logger=migrator id="Update initial version to 1" 2024-04-14T13:56:50.915 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Apr 14 13:56:50 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-grafana-a[41857]: t=2024-04-14T13:56:50+0000 lvl=info msg="Executing migration" logger=migrator id="Add read_only data column" 2024-04-14T13:56:50.915 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Apr 14 13:56:50 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-grafana-a[41857]: t=2024-04-14T13:56:50+0000 lvl=info msg="Executing migration" logger=migrator id="Migrate logging ds to loki ds" 2024-04-14T13:56:50.916 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Apr 14 13:56:50 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-grafana-a[41857]: t=2024-04-14T13:56:50+0000 lvl=info msg="Executing migration" logger=migrator id="Update json_data with nulls" 2024-04-14T13:56:50.916 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Apr 14 13:56:50 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-grafana-a[41857]: t=2024-04-14T13:56:50+0000 lvl=info msg="Executing migration" logger=migrator id="Add uid column" 2024-04-14T13:56:50.916 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Apr 14 13:56:50 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-grafana-a[41857]: t=2024-04-14T13:56:50+0000 lvl=info msg="Executing migration" logger=migrator id="Update uid value" 2024-04-14T13:56:50.916 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Apr 14 13:56:50 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-grafana-a[41857]: t=2024-04-14T13:56:50+0000 lvl=info msg="Executing migration" logger=migrator id="Add unique index datasource_org_id_uid" 2024-04-14T13:56:50.918 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Apr 14 13:56:50 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-grafana-a[41857]: t=2024-04-14T13:56:50+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index datasource_org_id_is_default" 2024-04-14T13:56:50.918 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Apr 14 13:56:50 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-grafana-a[41857]: t=2024-04-14T13:56:50+0000 lvl=info msg="Executing migration" logger=migrator id="create api_key table" 2024-04-14T13:56:50.918 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Apr 14 13:56:50 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-grafana-a[41857]: t=2024-04-14T13:56:50+0000 lvl=info msg="Executing migration" logger=migrator id="add index api_key.account_id" 2024-04-14T13:56:50.918 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Apr 14 13:56:50 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-grafana-a[41857]: t=2024-04-14T13:56:50+0000 lvl=info msg="Executing migration" logger=migrator id="add index api_key.key" 2024-04-14T13:56:50.919 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Apr 14 13:56:50 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-grafana-a[41857]: t=2024-04-14T13:56:50+0000 lvl=info msg="Executing migration" logger=migrator id="add index api_key.account_id_name" 2024-04-14T13:56:50.919 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Apr 14 13:56:50 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-grafana-a[41857]: t=2024-04-14T13:56:50+0000 lvl=info msg="Executing migration" logger=migrator id="drop index IDX_api_key_account_id - v1" 2024-04-14T13:56:50.919 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Apr 14 13:56:50 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-grafana-a[41857]: t=2024-04-14T13:56:50+0000 lvl=info msg="Executing migration" logger=migrator id="drop index UQE_api_key_key - v1" 2024-04-14T13:56:50.919 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Apr 14 13:56:50 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-grafana-a[41857]: t=2024-04-14T13:56:50+0000 lvl=info msg="Executing migration" logger=migrator id="drop index UQE_api_key_account_id_name - v1" 2024-04-14T13:56:50.919 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Apr 14 13:56:50 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-grafana-a[41857]: t=2024-04-14T13:56:50+0000 lvl=info msg="Executing migration" logger=migrator id="Rename table api_key to api_key_v1 - v1" 2024-04-14T13:56:50.919 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Apr 14 13:56:50 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-grafana-a[41857]: t=2024-04-14T13:56:50+0000 lvl=info msg="Executing migration" logger=migrator id="create api_key table v2" 2024-04-14T13:56:50.919 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Apr 14 13:56:50 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-grafana-a[41857]: t=2024-04-14T13:56:50+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_api_key_org_id - v2" 2024-04-14T13:56:50.920 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Apr 14 13:56:50 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-grafana-a[41857]: t=2024-04-14T13:56:50+0000 lvl=info msg="Executing migration" logger=migrator id="create index UQE_api_key_key - v2" 2024-04-14T13:56:50.920 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Apr 14 13:56:50 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-grafana-a[41857]: t=2024-04-14T13:56:50+0000 lvl=info msg="Executing migration" logger=migrator id="create index UQE_api_key_org_id_name - v2" 2024-04-14T13:56:50.920 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Apr 14 13:56:50 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-grafana-a[41857]: t=2024-04-14T13:56:50+0000 lvl=info msg="Executing migration" logger=migrator id="copy api_key v1 to v2" 2024-04-14T13:56:50.920 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Apr 14 13:56:50 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-grafana-a[41857]: t=2024-04-14T13:56:50+0000 lvl=info msg="Executing migration" logger=migrator id="Drop old table api_key_v1" 2024-04-14T13:56:50.920 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Apr 14 13:56:50 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-grafana-a[41857]: t=2024-04-14T13:56:50+0000 lvl=info msg="Executing migration" logger=migrator id="Update api_key table charset" 2024-04-14T13:56:50.920 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Apr 14 13:56:50 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-grafana-a[41857]: t=2024-04-14T13:56:50+0000 lvl=info msg="Executing migration" logger=migrator id="Add expires to api_key table" 2024-04-14T13:56:50.920 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Apr 14 13:56:50 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-grafana-a[41857]: t=2024-04-14T13:56:50+0000 lvl=info msg="Executing migration" logger=migrator id="Add service account foreign key" 2024-04-14T13:56:50.920 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Apr 14 13:56:50 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-grafana-a[41857]: t=2024-04-14T13:56:50+0000 lvl=info msg="Executing migration" logger=migrator id="create dashboard_snapshot table v4" 2024-04-14T13:56:50.921 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Apr 14 13:56:50 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-grafana-a[41857]: t=2024-04-14T13:56:50+0000 lvl=info msg="Executing migration" logger=migrator id="drop table dashboard_snapshot_v4 #1" 2024-04-14T13:56:50.921 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Apr 14 13:56:50 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-grafana-a[41857]: t=2024-04-14T13:56:50+0000 lvl=info msg="Executing migration" logger=migrator id="create dashboard_snapshot table v5 #2" 2024-04-14T13:56:50.921 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Apr 14 13:56:50 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-grafana-a[41857]: t=2024-04-14T13:56:50+0000 lvl=info msg="Executing migration" logger=migrator id="create index UQE_dashboard_snapshot_key - v5" 2024-04-14T13:56:50.921 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Apr 14 13:56:50 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-grafana-a[41857]: t=2024-04-14T13:56:50+0000 lvl=info msg="Executing migration" logger=migrator id="create index UQE_dashboard_snapshot_delete_key - v5" 2024-04-14T13:56:50.921 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Apr 14 13:56:50 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-grafana-a[41857]: t=2024-04-14T13:56:50+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_dashboard_snapshot_user_id - v5" 2024-04-14T13:56:50.921 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Apr 14 13:56:50 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-grafana-a[41857]: t=2024-04-14T13:56:50+0000 lvl=info msg="Executing migration" logger=migrator id="alter dashboard_snapshot to mediumtext v2" 2024-04-14T13:56:50.921 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Apr 14 13:56:50 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-grafana-a[41857]: t=2024-04-14T13:56:50+0000 lvl=info msg="Executing migration" logger=migrator id="Update dashboard_snapshot table charset" 2024-04-14T13:56:50.922 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Apr 14 13:56:50 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-grafana-a[41857]: t=2024-04-14T13:56:50+0000 lvl=info msg="Executing migration" logger=migrator id="Add column external_delete_url to dashboard_snapshots table" 2024-04-14T13:56:50.922 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Apr 14 13:56:50 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-grafana-a[41857]: t=2024-04-14T13:56:50+0000 lvl=info msg="Executing migration" logger=migrator id="Add encrypted dashboard json column" 2024-04-14T13:56:50.922 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Apr 14 13:56:50 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-grafana-a[41857]: t=2024-04-14T13:56:50+0000 lvl=info msg="Executing migration" logger=migrator id="Change dashboard_encrypted column to MEDIUMBLOB" 2024-04-14T13:56:50.922 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Apr 14 13:56:50 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-grafana-a[41857]: t=2024-04-14T13:56:50+0000 lvl=info msg="Executing migration" logger=migrator id="create quota table v1" 2024-04-14T13:56:50.922 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Apr 14 13:56:50 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-grafana-a[41857]: t=2024-04-14T13:56:50+0000 lvl=info msg="Executing migration" logger=migrator id="create index UQE_quota_org_id_user_id_target - v1" 2024-04-14T13:56:50.922 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Apr 14 13:56:50 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-grafana-a[41857]: t=2024-04-14T13:56:50+0000 lvl=info msg="Executing migration" logger=migrator id="Update quota table charset" 2024-04-14T13:56:50.922 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Apr 14 13:56:50 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-grafana-a[41857]: t=2024-04-14T13:56:50+0000 lvl=info msg="Executing migration" logger=migrator id="create plugin_setting table" 2024-04-14T13:56:50.923 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Apr 14 13:56:50 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-grafana-a[41857]: t=2024-04-14T13:56:50+0000 lvl=info msg="Executing migration" logger=migrator id="create index UQE_plugin_setting_org_id_plugin_id - v1" 2024-04-14T13:56:50.923 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Apr 14 13:56:50 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-grafana-a[41857]: t=2024-04-14T13:56:50+0000 lvl=info msg="Executing migration" logger=migrator id="Add column plugin_version to plugin_settings" 2024-04-14T13:56:50.923 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Apr 14 13:56:50 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-grafana-a[41857]: t=2024-04-14T13:56:50+0000 lvl=info msg="Executing migration" logger=migrator id="Update plugin_setting table charset" 2024-04-14T13:56:50.923 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Apr 14 13:56:50 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-grafana-a[41857]: t=2024-04-14T13:56:50+0000 lvl=info msg="Executing migration" logger=migrator id="create session table" 2024-04-14T13:56:50.923 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Apr 14 13:56:50 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-grafana-a[41857]: t=2024-04-14T13:56:50+0000 lvl=info msg="Executing migration" logger=migrator id="Drop old table playlist table" 2024-04-14T13:56:50.923 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Apr 14 13:56:50 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-grafana-a[41857]: t=2024-04-14T13:56:50+0000 lvl=info msg="Executing migration" logger=migrator id="Drop old table playlist_item table" 2024-04-14T13:56:50.923 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Apr 14 13:56:50 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-grafana-a[41857]: t=2024-04-14T13:56:50+0000 lvl=info msg="Executing migration" logger=migrator id="create playlist table v2" 2024-04-14T13:56:50.924 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Apr 14 13:56:50 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-grafana-a[41857]: t=2024-04-14T13:56:50+0000 lvl=info msg="Executing migration" logger=migrator id="create playlist item table v2" 2024-04-14T13:56:50.924 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Apr 14 13:56:50 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-grafana-a[41857]: t=2024-04-14T13:56:50+0000 lvl=info msg="Executing migration" logger=migrator id="Update playlist table charset" 2024-04-14T13:56:50.924 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Apr 14 13:56:50 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-grafana-a[41857]: t=2024-04-14T13:56:50+0000 lvl=info msg="Executing migration" logger=migrator id="Update playlist_item table charset" 2024-04-14T13:56:50.924 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Apr 14 13:56:50 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-grafana-a[41857]: t=2024-04-14T13:56:50+0000 lvl=info msg="Executing migration" logger=migrator id="drop preferences table v2" 2024-04-14T13:56:50.924 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Apr 14 13:56:50 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-grafana-a[41857]: t=2024-04-14T13:56:50+0000 lvl=info msg="Executing migration" logger=migrator id="drop preferences table v3" 2024-04-14T13:56:50.924 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Apr 14 13:56:50 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-grafana-a[41857]: t=2024-04-14T13:56:50+0000 lvl=info msg="Executing migration" logger=migrator id="create preferences table v3" 2024-04-14T13:56:50.924 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Apr 14 13:56:50 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-grafana-a[41857]: t=2024-04-14T13:56:50+0000 lvl=info msg="Executing migration" logger=migrator id="Update preferences table charset" 2024-04-14T13:56:50.924 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Apr 14 13:56:50 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-grafana-a[41857]: t=2024-04-14T13:56:50+0000 lvl=info msg="Executing migration" logger=migrator id="Add column team_id in preferences" 2024-04-14T13:56:50.925 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Apr 14 13:56:50 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-grafana-a[41857]: t=2024-04-14T13:56:50+0000 lvl=info msg="Executing migration" logger=migrator id="Update team_id column values in preferences" 2024-04-14T13:56:50.925 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Apr 14 13:56:50 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-grafana-a[41857]: t=2024-04-14T13:56:50+0000 lvl=info msg="Executing migration" logger=migrator id="Add column week_start in preferences" 2024-04-14T13:56:50.925 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Apr 14 13:56:50 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-grafana-a[41857]: t=2024-04-14T13:56:50+0000 lvl=info msg="Executing migration" logger=migrator id="create alert table v1" 2024-04-14T13:56:50.925 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Apr 14 13:56:50 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-grafana-a[41857]: t=2024-04-14T13:56:50+0000 lvl=info msg="Executing migration" logger=migrator id="add index alert org_id & id " 2024-04-14T13:56:50.925 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Apr 14 13:56:50 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-grafana-a[41857]: t=2024-04-14T13:56:50+0000 lvl=info msg="Executing migration" logger=migrator id="add index alert state" 2024-04-14T13:56:50.925 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Apr 14 13:56:50 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-grafana-a[41857]: t=2024-04-14T13:56:50+0000 lvl=info msg="Executing migration" logger=migrator id="add index alert dashboard_id" 2024-04-14T13:56:50.925 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Apr 14 13:56:50 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-grafana-a[41857]: t=2024-04-14T13:56:50+0000 lvl=info msg="Executing migration" logger=migrator id="Create alert_rule_tag table v1" 2024-04-14T13:56:50.925 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Apr 14 13:56:50 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-grafana-a[41857]: t=2024-04-14T13:56:50+0000 lvl=info msg="Executing migration" logger=migrator id="Add unique index alert_rule_tag.alert_id_tag_id" 2024-04-14T13:56:50.926 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Apr 14 13:56:50 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-grafana-a[41857]: t=2024-04-14T13:56:50+0000 lvl=info msg="Executing migration" logger=migrator id="drop index UQE_alert_rule_tag_alert_id_tag_id - v1" 2024-04-14T13:56:50.926 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Apr 14 13:56:50 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-grafana-a[41857]: t=2024-04-14T13:56:50+0000 lvl=info msg="Executing migration" logger=migrator id="Rename table alert_rule_tag to alert_rule_tag_v1 - v1" 2024-04-14T13:56:50.926 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Apr 14 13:56:50 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-grafana-a[41857]: t=2024-04-14T13:56:50+0000 lvl=info msg="Executing migration" logger=migrator id="Create alert_rule_tag table v2" 2024-04-14T13:56:50.926 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Apr 14 13:56:50 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-grafana-a[41857]: t=2024-04-14T13:56:50+0000 lvl=info msg="Executing migration" logger=migrator id="create index UQE_alert_rule_tag_alert_id_tag_id - Add unique index alert_rule_tag.alert_id_tag_id V2" 2024-04-14T13:56:50.926 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Apr 14 13:56:50 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-grafana-a[41857]: t=2024-04-14T13:56:50+0000 lvl=info msg="Executing migration" logger=migrator id="copy alert_rule_tag v1 to v2" 2024-04-14T13:56:50.926 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Apr 14 13:56:50 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-grafana-a[41857]: t=2024-04-14T13:56:50+0000 lvl=info msg="Executing migration" logger=migrator id="drop table alert_rule_tag_v1" 2024-04-14T13:56:50.926 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Apr 14 13:56:50 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-grafana-a[41857]: t=2024-04-14T13:56:50+0000 lvl=info msg="Executing migration" logger=migrator id="create alert_notification table v1" 2024-04-14T13:56:50.927 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Apr 14 13:56:50 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-grafana-a[41857]: t=2024-04-14T13:56:50+0000 lvl=info msg="Executing migration" logger=migrator id="Add column is_default" 2024-04-14T13:56:50.927 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Apr 14 13:56:50 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-grafana-a[41857]: t=2024-04-14T13:56:50+0000 lvl=info msg="Executing migration" logger=migrator id="Add column frequency" 2024-04-14T13:56:50.927 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Apr 14 13:56:50 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-grafana-a[41857]: t=2024-04-14T13:56:50+0000 lvl=info msg="Executing migration" logger=migrator id="Add column send_reminder" 2024-04-14T13:56:50.927 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Apr 14 13:56:50 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-grafana-a[41857]: t=2024-04-14T13:56:50+0000 lvl=info msg="Executing migration" logger=migrator id="Add column disable_resolve_message" 2024-04-14T13:56:50.927 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Apr 14 13:56:50 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-grafana-a[41857]: t=2024-04-14T13:56:50+0000 lvl=info msg="Executing migration" logger=migrator id="add index alert_notification org_id & name" 2024-04-14T13:56:50.927 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Apr 14 13:56:50 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-grafana-a[41857]: t=2024-04-14T13:56:50+0000 lvl=info msg="Executing migration" logger=migrator id="Update alert table charset" 2024-04-14T13:56:50.927 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Apr 14 13:56:50 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-grafana-a[41857]: t=2024-04-14T13:56:50+0000 lvl=info msg="Executing migration" logger=migrator id="Update alert_notification table charset" 2024-04-14T13:56:50.928 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Apr 14 13:56:50 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-grafana-a[41857]: t=2024-04-14T13:56:50+0000 lvl=info msg="Executing migration" logger=migrator id="create notification_journal table v1" 2024-04-14T13:56:50.928 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Apr 14 13:56:50 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-grafana-a[41857]: t=2024-04-14T13:56:50+0000 lvl=info msg="Executing migration" logger=migrator id="add index notification_journal org_id & alert_id & notifier_id" 2024-04-14T13:56:50.928 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Apr 14 13:56:50 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-grafana-a[41857]: t=2024-04-14T13:56:50+0000 lvl=info msg="Executing migration" logger=migrator id="drop alert_notification_journal" 2024-04-14T13:56:50.928 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Apr 14 13:56:50 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-grafana-a[41857]: t=2024-04-14T13:56:50+0000 lvl=info msg="Executing migration" logger=migrator id="create alert_notification_state table v1" 2024-04-14T13:56:50.928 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Apr 14 13:56:50 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-grafana-a[41857]: t=2024-04-14T13:56:50+0000 lvl=info msg="Executing migration" logger=migrator id="add index alert_notification_state org_id & alert_id & notifier_id" 2024-04-14T13:56:50.928 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Apr 14 13:56:50 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-grafana-a[41857]: t=2024-04-14T13:56:50+0000 lvl=info msg="Executing migration" logger=migrator id="Add for to alert table" 2024-04-14T13:56:50.928 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Apr 14 13:56:50 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-grafana-a[41857]: t=2024-04-14T13:56:50+0000 lvl=info msg="Executing migration" logger=migrator id="Add column uid in alert_notification" 2024-04-14T13:56:50.929 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Apr 14 13:56:50 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-grafana-a[41857]: t=2024-04-14T13:56:50+0000 lvl=info msg="Executing migration" logger=migrator id="Update uid column values in alert_notification" 2024-04-14T13:56:50.929 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Apr 14 13:56:50 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-grafana-a[41857]: t=2024-04-14T13:56:50+0000 lvl=info msg="Executing migration" logger=migrator id="Add unique index alert_notification_org_id_uid" 2024-04-14T13:56:50.929 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Apr 14 13:56:50 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-grafana-a[41857]: t=2024-04-14T13:56:50+0000 lvl=info msg="Executing migration" logger=migrator id="Remove unique index org_id_name" 2024-04-14T13:56:50.929 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Apr 14 13:56:50 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-grafana-a[41857]: t=2024-04-14T13:56:50+0000 lvl=info msg="Executing migration" logger=migrator id="Add column secure_settings in alert_notification" 2024-04-14T13:56:50.929 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Apr 14 13:56:50 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-grafana-a[41857]: t=2024-04-14T13:56:50+0000 lvl=info msg="Executing migration" logger=migrator id="alter alert.settings to mediumtext" 2024-04-14T13:56:50.929 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Apr 14 13:56:50 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-grafana-a[41857]: t=2024-04-14T13:56:50+0000 lvl=info msg="Executing migration" logger=migrator id="Add non-unique index alert_notification_state_alert_id" 2024-04-14T13:56:50.929 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Apr 14 13:56:50 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-grafana-a[41857]: t=2024-04-14T13:56:50+0000 lvl=info msg="Executing migration" logger=migrator id="Add non-unique index alert_rule_tag_alert_id" 2024-04-14T13:56:50.930 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Apr 14 13:56:50 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-grafana-a[41857]: t=2024-04-14T13:56:50+0000 lvl=info msg="Executing migration" logger=migrator id="Drop old annotation table v4" 2024-04-14T13:56:50.930 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Apr 14 13:56:50 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-grafana-a[41857]: t=2024-04-14T13:56:50+0000 lvl=info msg="Executing migration" logger=migrator id="create annotation table v5" 2024-04-14T13:56:50.930 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Apr 14 13:56:50 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-grafana-a[41857]: t=2024-04-14T13:56:50+0000 lvl=info msg="Executing migration" logger=migrator id="add index annotation 0 v3" 2024-04-14T13:56:50.930 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Apr 14 13:56:50 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-grafana-a[41857]: t=2024-04-14T13:56:50+0000 lvl=info msg="Executing migration" logger=migrator id="add index annotation 1 v3" 2024-04-14T13:56:50.930 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Apr 14 13:56:50 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-grafana-a[41857]: t=2024-04-14T13:56:50+0000 lvl=info msg="Executing migration" logger=migrator id="add index annotation 2 v3" 2024-04-14T13:56:50.930 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Apr 14 13:56:50 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-grafana-a[41857]: t=2024-04-14T13:56:50+0000 lvl=info msg="Executing migration" logger=migrator id="add index annotation 3 v3" 2024-04-14T13:56:50.930 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Apr 14 13:56:50 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-grafana-a[41857]: t=2024-04-14T13:56:50+0000 lvl=info msg="Executing migration" logger=migrator id="add index annotation 4 v3" 2024-04-14T13:56:50.931 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Apr 14 13:56:50 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-grafana-a[41857]: t=2024-04-14T13:56:50+0000 lvl=info msg="Executing migration" logger=migrator id="Update annotation table charset" 2024-04-14T13:56:50.931 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Apr 14 13:56:50 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-grafana-a[41857]: t=2024-04-14T13:56:50+0000 lvl=info msg="Executing migration" logger=migrator id="Add column region_id to annotation table" 2024-04-14T13:56:50.931 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Apr 14 13:56:50 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-grafana-a[41857]: t=2024-04-14T13:56:50+0000 lvl=info msg="Executing migration" logger=migrator id="Drop category_id index" 2024-04-14T13:56:50.931 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Apr 14 13:56:50 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-grafana-a[41857]: t=2024-04-14T13:56:50+0000 lvl=info msg="Executing migration" logger=migrator id="Add column tags to annotation table" 2024-04-14T13:56:50.931 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Apr 14 13:56:50 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-grafana-a[41857]: t=2024-04-14T13:56:50+0000 lvl=info msg="Executing migration" logger=migrator id="Create annotation_tag table v2" 2024-04-14T13:56:50.931 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Apr 14 13:56:50 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-grafana-a[41857]: t=2024-04-14T13:56:50+0000 lvl=info msg="Executing migration" logger=migrator id="Add unique index annotation_tag.annotation_id_tag_id" 2024-04-14T13:56:50.932 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Apr 14 13:56:50 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-grafana-a[41857]: t=2024-04-14T13:56:50+0000 lvl=info msg="Executing migration" logger=migrator id="drop index UQE_annotation_tag_annotation_id_tag_id - v2" 2024-04-14T13:56:50.932 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Apr 14 13:56:50 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-grafana-a[41857]: t=2024-04-14T13:56:50+0000 lvl=info msg="Executing migration" logger=migrator id="Rename table annotation_tag to annotation_tag_v2 - v2" 2024-04-14T13:56:50.932 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Apr 14 13:56:50 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-grafana-a[41857]: t=2024-04-14T13:56:50+0000 lvl=info msg="Executing migration" logger=migrator id="Create annotation_tag table v3" 2024-04-14T13:56:50.932 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Apr 14 13:56:50 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-grafana-a[41857]: t=2024-04-14T13:56:50+0000 lvl=info msg="Executing migration" logger=migrator id="create index UQE_annotation_tag_annotation_id_tag_id - Add unique index annotation_tag.annotation_id_tag_id V3" 2024-04-14T13:56:50.932 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Apr 14 13:56:50 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-grafana-a[41857]: t=2024-04-14T13:56:50+0000 lvl=info msg="Executing migration" logger=migrator id="copy annotation_tag v2 to v3" 2024-04-14T13:56:50.932 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Apr 14 13:56:50 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-grafana-a[41857]: t=2024-04-14T13:56:50+0000 lvl=info msg="Executing migration" logger=migrator id="drop table annotation_tag_v2" 2024-04-14T13:56:50.932 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Apr 14 13:56:50 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-grafana-a[41857]: t=2024-04-14T13:56:50+0000 lvl=info msg="Executing migration" logger=migrator id="Update alert annotations and set TEXT to empty" 2024-04-14T13:56:50.933 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Apr 14 13:56:50 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-grafana-a[41857]: t=2024-04-14T13:56:50+0000 lvl=info msg="Executing migration" logger=migrator id="Add created time to annotation table" 2024-04-14T13:56:50.933 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Apr 14 13:56:50 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-grafana-a[41857]: t=2024-04-14T13:56:50+0000 lvl=info msg="Executing migration" logger=migrator id="Add updated time to annotation table" 2024-04-14T13:56:50.933 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Apr 14 13:56:50 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-grafana-a[41857]: t=2024-04-14T13:56:50+0000 lvl=info msg="Executing migration" logger=migrator id="Add index for created in annotation table" 2024-04-14T13:56:50.933 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Apr 14 13:56:50 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-grafana-a[41857]: t=2024-04-14T13:56:50+0000 lvl=info msg="Executing migration" logger=migrator id="Add index for updated in annotation table" 2024-04-14T13:56:50.933 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Apr 14 13:56:50 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-grafana-a[41857]: t=2024-04-14T13:56:50+0000 lvl=info msg="Executing migration" logger=migrator id="Convert existing annotations from seconds to milliseconds" 2024-04-14T13:56:50.933 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Apr 14 13:56:50 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-grafana-a[41857]: t=2024-04-14T13:56:50+0000 lvl=info msg="Executing migration" logger=migrator id="Add epoch_end column" 2024-04-14T13:56:50.933 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Apr 14 13:56:50 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-grafana-a[41857]: t=2024-04-14T13:56:50+0000 lvl=info msg="Executing migration" logger=migrator id="Add index for epoch_end" 2024-04-14T13:56:50.933 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Apr 14 13:56:50 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-grafana-a[41857]: t=2024-04-14T13:56:50+0000 lvl=info msg="Executing migration" logger=migrator id="Make epoch_end the same as epoch" 2024-04-14T13:56:50.934 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Apr 14 13:56:50 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-grafana-a[41857]: t=2024-04-14T13:56:50+0000 lvl=info msg="Executing migration" logger=migrator id="Move region to single row" 2024-04-14T13:56:50.934 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Apr 14 13:56:50 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-grafana-a[41857]: t=2024-04-14T13:56:50+0000 lvl=info msg="Executing migration" logger=migrator id="Remove index org_id_epoch from annotation table" 2024-04-14T13:56:50.934 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Apr 14 13:56:50 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-grafana-a[41857]: t=2024-04-14T13:56:50+0000 lvl=info msg="Executing migration" logger=migrator id="Remove index org_id_dashboard_id_panel_id_epoch from annotation table" 2024-04-14T13:56:50.934 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Apr 14 13:56:50 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-grafana-a[41857]: t=2024-04-14T13:56:50+0000 lvl=info msg="Executing migration" logger=migrator id="Add index for org_id_dashboard_id_epoch_end_epoch on annotation table" 2024-04-14T13:56:50.934 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Apr 14 13:56:50 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-grafana-a[41857]: t=2024-04-14T13:56:50+0000 lvl=info msg="Executing migration" logger=migrator id="Add index for org_id_epoch_end_epoch on annotation table" 2024-04-14T13:56:50.934 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Apr 14 13:56:50 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-grafana-a[41857]: t=2024-04-14T13:56:50+0000 lvl=info msg="Executing migration" logger=migrator id="Remove index org_id_epoch_epoch_end from annotation table" 2024-04-14T13:56:50.934 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Apr 14 13:56:50 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-grafana-a[41857]: t=2024-04-14T13:56:50+0000 lvl=info msg="Executing migration" logger=migrator id="Add index for alert_id on annotation table" 2024-04-14T13:56:50.935 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Apr 14 13:56:50 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-grafana-a[41857]: t=2024-04-14T13:56:50+0000 lvl=info msg="Executing migration" logger=migrator id="create test_data table" 2024-04-14T13:56:50.935 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Apr 14 13:56:50 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-grafana-a[41857]: t=2024-04-14T13:56:50+0000 lvl=info msg="Executing migration" logger=migrator id="create dashboard_version table v1" 2024-04-14T13:56:50.935 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Apr 14 13:56:50 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-grafana-a[41857]: t=2024-04-14T13:56:50+0000 lvl=info msg="Executing migration" logger=migrator id="add index dashboard_version.dashboard_id" 2024-04-14T13:56:50.935 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Apr 14 13:56:50 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-grafana-a[41857]: t=2024-04-14T13:56:50+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index dashboard_version.dashboard_id and dashboard_version.version" 2024-04-14T13:56:50.935 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Apr 14 13:56:50 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-grafana-a[41857]: t=2024-04-14T13:56:50+0000 lvl=info msg="Executing migration" logger=migrator id="Set dashboard version to 1 where 0" 2024-04-14T13:56:50.935 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Apr 14 13:56:50 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-grafana-a[41857]: t=2024-04-14T13:56:50+0000 lvl=info msg="Executing migration" logger=migrator id="save existing dashboard data in dashboard_version table v1" 2024-04-14T13:56:50.935 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Apr 14 13:56:50 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-grafana-a[41857]: t=2024-04-14T13:56:50+0000 lvl=info msg="Executing migration" logger=migrator id="alter dashboard_version.data to mediumtext v1" 2024-04-14T13:56:50.936 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Apr 14 13:56:50 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-grafana-a[41857]: t=2024-04-14T13:56:50+0000 lvl=info msg="Executing migration" logger=migrator id="create team table" 2024-04-14T13:56:50.936 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Apr 14 13:56:50 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-grafana-a[41857]: t=2024-04-14T13:56:50+0000 lvl=info msg="Executing migration" logger=migrator id="add index team.org_id" 2024-04-14T13:56:51.162 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Apr 14 13:56:50 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-grafana-a[41857]: t=2024-04-14T13:56:50+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index team_org_id_name" 2024-04-14T13:56:51.162 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Apr 14 13:56:50 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-grafana-a[41857]: t=2024-04-14T13:56:50+0000 lvl=info msg="Executing migration" logger=migrator id="create team member table" 2024-04-14T13:56:51.162 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Apr 14 13:56:50 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-grafana-a[41857]: t=2024-04-14T13:56:50+0000 lvl=info msg="Executing migration" logger=migrator id="add index team_member.org_id" 2024-04-14T13:56:51.162 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Apr 14 13:56:50 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-grafana-a[41857]: t=2024-04-14T13:56:50+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index team_member_org_id_team_id_user_id" 2024-04-14T13:56:51.162 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Apr 14 13:56:50 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-grafana-a[41857]: t=2024-04-14T13:56:50+0000 lvl=info msg="Executing migration" logger=migrator id="add index team_member.team_id" 2024-04-14T13:56:51.163 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Apr 14 13:56:50 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-grafana-a[41857]: t=2024-04-14T13:56:50+0000 lvl=info msg="Executing migration" logger=migrator id="Add column email to team table" 2024-04-14T13:56:51.163 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Apr 14 13:56:50 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-grafana-a[41857]: t=2024-04-14T13:56:50+0000 lvl=info msg="Executing migration" logger=migrator id="Add column external to team_member table" 2024-04-14T13:56:51.163 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Apr 14 13:56:50 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-grafana-a[41857]: t=2024-04-14T13:56:50+0000 lvl=info msg="Executing migration" logger=migrator id="Add column permission to team_member table" 2024-04-14T13:56:51.163 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Apr 14 13:56:50 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-grafana-a[41857]: t=2024-04-14T13:56:50+0000 lvl=info msg="Executing migration" logger=migrator id="create dashboard acl table" 2024-04-14T13:56:51.163 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Apr 14 13:56:50 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-grafana-a[41857]: t=2024-04-14T13:56:50+0000 lvl=info msg="Executing migration" logger=migrator id="add index dashboard_acl_dashboard_id" 2024-04-14T13:56:51.163 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Apr 14 13:56:50 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-grafana-a[41857]: t=2024-04-14T13:56:50+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index dashboard_acl_dashboard_id_user_id" 2024-04-14T13:56:51.163 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Apr 14 13:56:50 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-grafana-a[41857]: t=2024-04-14T13:56:50+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index dashboard_acl_dashboard_id_team_id" 2024-04-14T13:56:51.163 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Apr 14 13:56:50 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-grafana-a[41857]: t=2024-04-14T13:56:50+0000 lvl=info msg="Executing migration" logger=migrator id="add index dashboard_acl_user_id" 2024-04-14T13:56:51.163 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Apr 14 13:56:50 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-grafana-a[41857]: t=2024-04-14T13:56:50+0000 lvl=info msg="Executing migration" logger=migrator id="add index dashboard_acl_team_id" 2024-04-14T13:56:51.163 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Apr 14 13:56:50 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-grafana-a[41857]: t=2024-04-14T13:56:50+0000 lvl=info msg="Executing migration" logger=migrator id="add index dashboard_acl_org_id_role" 2024-04-14T13:56:51.163 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Apr 14 13:56:50 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-grafana-a[41857]: t=2024-04-14T13:56:50+0000 lvl=info msg="Executing migration" logger=migrator id="add index dashboard_permission" 2024-04-14T13:56:51.163 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Apr 14 13:56:50 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-grafana-a[41857]: t=2024-04-14T13:56:50+0000 lvl=info msg="Executing migration" logger=migrator id="save default acl rules in dashboard_acl table" 2024-04-14T13:56:51.163 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Apr 14 13:56:50 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-grafana-a[41857]: t=2024-04-14T13:56:50+0000 lvl=info msg="Executing migration" logger=migrator id="delete acl rules for deleted dashboards and folders" 2024-04-14T13:56:51.163 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Apr 14 13:56:50 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-grafana-a[41857]: t=2024-04-14T13:56:50+0000 lvl=info msg="Executing migration" logger=migrator id="create tag table" 2024-04-14T13:56:51.165 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Apr 14 13:56:50 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-grafana-a[41857]: t=2024-04-14T13:56:50+0000 lvl=info msg="Executing migration" logger=migrator id="add index tag.key_value" 2024-04-14T13:56:51.165 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Apr 14 13:56:50 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-grafana-a[41857]: t=2024-04-14T13:56:50+0000 lvl=info msg="Executing migration" logger=migrator id="create login attempt table" 2024-04-14T13:56:51.165 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Apr 14 13:56:50 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-grafana-a[41857]: t=2024-04-14T13:56:50+0000 lvl=info msg="Executing migration" logger=migrator id="add index login_attempt.username" 2024-04-14T13:56:51.165 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Apr 14 13:56:50 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-grafana-a[41857]: t=2024-04-14T13:56:50+0000 lvl=info msg="Executing migration" logger=migrator id="drop index IDX_login_attempt_username - v1" 2024-04-14T13:56:51.165 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Apr 14 13:56:50 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-grafana-a[41857]: t=2024-04-14T13:56:50+0000 lvl=info msg="Executing migration" logger=migrator id="Rename table login_attempt to login_attempt_tmp_qwerty - v1" 2024-04-14T13:56:51.165 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Apr 14 13:56:50 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-grafana-a[41857]: t=2024-04-14T13:56:50+0000 lvl=info msg="Executing migration" logger=migrator id="create login_attempt v2" 2024-04-14T13:56:51.165 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Apr 14 13:56:50 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-grafana-a[41857]: t=2024-04-14T13:56:50+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_login_attempt_username - v2" 2024-04-14T13:56:51.165 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Apr 14 13:56:50 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-grafana-a[41857]: t=2024-04-14T13:56:50+0000 lvl=info msg="Executing migration" logger=migrator id="copy login_attempt v1 to v2" 2024-04-14T13:56:51.165 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Apr 14 13:56:50 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-grafana-a[41857]: t=2024-04-14T13:56:50+0000 lvl=info msg="Executing migration" logger=migrator id="drop login_attempt_tmp_qwerty" 2024-04-14T13:56:51.165 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Apr 14 13:56:50 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-grafana-a[41857]: t=2024-04-14T13:56:50+0000 lvl=info msg="Executing migration" logger=migrator id="create user auth table" 2024-04-14T13:56:51.165 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Apr 14 13:56:50 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-grafana-a[41857]: t=2024-04-14T13:56:50+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_user_auth_auth_module_auth_id - v1" 2024-04-14T13:56:51.165 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Apr 14 13:56:50 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-grafana-a[41857]: t=2024-04-14T13:56:50+0000 lvl=info msg="Executing migration" logger=migrator id="alter user_auth.auth_id to length 190" 2024-04-14T13:56:51.165 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Apr 14 13:56:50 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-grafana-a[41857]: t=2024-04-14T13:56:50+0000 lvl=info msg="Executing migration" logger=migrator id="Add OAuth access token to user_auth" 2024-04-14T13:56:51.165 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Apr 14 13:56:50 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-grafana-a[41857]: t=2024-04-14T13:56:50+0000 lvl=info msg="Executing migration" logger=migrator id="Add OAuth refresh token to user_auth" 2024-04-14T13:56:51.166 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Apr 14 13:56:50 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-grafana-a[41857]: t=2024-04-14T13:56:50+0000 lvl=info msg="Executing migration" logger=migrator id="Add OAuth token type to user_auth" 2024-04-14T13:56:51.166 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Apr 14 13:56:50 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-grafana-a[41857]: t=2024-04-14T13:56:50+0000 lvl=info msg="Executing migration" logger=migrator id="Add OAuth expiry to user_auth" 2024-04-14T13:56:51.166 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Apr 14 13:56:50 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-grafana-a[41857]: t=2024-04-14T13:56:50+0000 lvl=info msg="Executing migration" logger=migrator id="Add index to user_id column in user_auth" 2024-04-14T13:56:51.166 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Apr 14 13:56:50 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-grafana-a[41857]: t=2024-04-14T13:56:50+0000 lvl=info msg="Executing migration" logger=migrator id="create server_lock table" 2024-04-14T13:56:51.166 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Apr 14 13:56:50 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-grafana-a[41857]: t=2024-04-14T13:56:50+0000 lvl=info msg="Executing migration" logger=migrator id="add index server_lock.operation_uid" 2024-04-14T13:56:51.166 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Apr 14 13:56:50 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-grafana-a[41857]: t=2024-04-14T13:56:50+0000 lvl=info msg="Executing migration" logger=migrator id="create user auth token table" 2024-04-14T13:56:51.166 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Apr 14 13:56:50 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-grafana-a[41857]: t=2024-04-14T13:56:50+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index user_auth_token.auth_token" 2024-04-14T13:56:51.166 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Apr 14 13:56:50 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-grafana-a[41857]: t=2024-04-14T13:56:50+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index user_auth_token.prev_auth_token" 2024-04-14T13:56:51.166 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Apr 14 13:56:50 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-grafana-a[41857]: t=2024-04-14T13:56:50+0000 lvl=info msg="Executing migration" logger=migrator id="add index user_auth_token.user_id" 2024-04-14T13:56:51.166 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Apr 14 13:56:50 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-grafana-a[41857]: t=2024-04-14T13:56:50+0000 lvl=info msg="Executing migration" logger=migrator id="Add revoked_at to the user auth token" 2024-04-14T13:56:51.166 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Apr 14 13:56:50 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-grafana-a[41857]: t=2024-04-14T13:56:50+0000 lvl=info msg="Executing migration" logger=migrator id="create cache_data table" 2024-04-14T13:56:51.166 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Apr 14 13:56:50 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-grafana-a[41857]: t=2024-04-14T13:56:50+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index cache_data.cache_key" 2024-04-14T13:56:51.166 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Apr 14 13:56:50 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-grafana-a[41857]: t=2024-04-14T13:56:50+0000 lvl=info msg="Executing migration" logger=migrator id="create short_url table v1" 2024-04-14T13:56:51.166 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Apr 14 13:56:50 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-grafana-a[41857]: t=2024-04-14T13:56:50+0000 lvl=info msg="Executing migration" logger=migrator id="add index short_url.org_id-uid" 2024-04-14T13:56:51.167 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Apr 14 13:56:50 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-grafana-a[41857]: t=2024-04-14T13:56:50+0000 lvl=info msg="Executing migration" logger=migrator id="delete alert_definition table" 2024-04-14T13:56:51.167 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Apr 14 13:56:50 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-grafana-a[41857]: t=2024-04-14T13:56:50+0000 lvl=info msg="Executing migration" logger=migrator id="recreate alert_definition table" 2024-04-14T13:56:51.167 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Apr 14 13:56:50 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-grafana-a[41857]: t=2024-04-14T13:56:50+0000 lvl=info msg="Executing migration" logger=migrator id="add index in alert_definition on org_id and title columns" 2024-04-14T13:56:51.167 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Apr 14 13:56:50 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-grafana-a[41857]: t=2024-04-14T13:56:50+0000 lvl=info msg="Executing migration" logger=migrator id="add index in alert_definition on org_id and uid columns" 2024-04-14T13:56:51.167 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Apr 14 13:56:50 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-grafana-a[41857]: t=2024-04-14T13:56:50+0000 lvl=info msg="Executing migration" logger=migrator id="alter alert_definition table data column to mediumtext in mysql" 2024-04-14T13:56:51.167 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Apr 14 13:56:50 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-grafana-a[41857]: t=2024-04-14T13:56:50+0000 lvl=info msg="Executing migration" logger=migrator id="drop index in alert_definition on org_id and title columns" 2024-04-14T13:56:51.167 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Apr 14 13:56:50 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-grafana-a[41857]: t=2024-04-14T13:56:50+0000 lvl=info msg="Executing migration" logger=migrator id="drop index in alert_definition on org_id and uid columns" 2024-04-14T13:56:51.167 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Apr 14 13:56:50 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-grafana-a[41857]: t=2024-04-14T13:56:50+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index in alert_definition on org_id and title columns" 2024-04-14T13:56:51.167 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Apr 14 13:56:50 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-grafana-a[41857]: t=2024-04-14T13:56:50+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index in alert_definition on org_id and uid columns" 2024-04-14T13:56:51.167 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Apr 14 13:56:50 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-grafana-a[41857]: t=2024-04-14T13:56:50+0000 lvl=info msg="Executing migration" logger=migrator id="Add column paused in alert_definition" 2024-04-14T13:56:51.167 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Apr 14 13:56:50 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-grafana-a[41857]: t=2024-04-14T13:56:50+0000 lvl=info msg="Executing migration" logger=migrator id="drop alert_definition table" 2024-04-14T13:56:51.167 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Apr 14 13:56:50 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-grafana-a[41857]: t=2024-04-14T13:56:50+0000 lvl=info msg="Executing migration" logger=migrator id="delete alert_definition_version table" 2024-04-14T13:56:51.167 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Apr 14 13:56:50 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-grafana-a[41857]: t=2024-04-14T13:56:50+0000 lvl=info msg="Executing migration" logger=migrator id="recreate alert_definition_version table" 2024-04-14T13:56:51.167 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Apr 14 13:56:50 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-grafana-a[41857]: t=2024-04-14T13:56:50+0000 lvl=info msg="Executing migration" logger=migrator id="add index in alert_definition_version table on alert_definition_id and version columns" 2024-04-14T13:56:51.167 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Apr 14 13:56:51 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-grafana-a[41857]: t=2024-04-14T13:56:51+0000 lvl=info msg="Executing migration" logger=migrator id="add index in alert_definition_version table on alert_definition_uid and version columns" 2024-04-14T13:56:51.168 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Apr 14 13:56:51 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-grafana-a[41857]: t=2024-04-14T13:56:51+0000 lvl=info msg="Executing migration" logger=migrator id="alter alert_definition_version table data column to mediumtext in mysql" 2024-04-14T13:56:51.168 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Apr 14 13:56:51 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-grafana-a[41857]: t=2024-04-14T13:56:51+0000 lvl=info msg="Executing migration" logger=migrator id="drop alert_definition_version table" 2024-04-14T13:56:51.168 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Apr 14 13:56:51 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-grafana-a[41857]: t=2024-04-14T13:56:51+0000 lvl=info msg="Executing migration" logger=migrator id="create alert_instance table" 2024-04-14T13:56:51.168 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Apr 14 13:56:51 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-grafana-a[41857]: t=2024-04-14T13:56:51+0000 lvl=info msg="Executing migration" logger=migrator id="add index in alert_instance table on def_org_id, def_uid and current_state columns" 2024-04-14T13:56:51.168 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Apr 14 13:56:51 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-grafana-a[41857]: t=2024-04-14T13:56:51+0000 lvl=info msg="Executing migration" logger=migrator id="add index in alert_instance table on def_org_id, current_state columns" 2024-04-14T13:56:51.168 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Apr 14 13:56:51 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-grafana-a[41857]: t=2024-04-14T13:56:51+0000 lvl=info msg="Executing migration" logger=migrator id="add column current_state_end to alert_instance" 2024-04-14T13:56:51.168 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Apr 14 13:56:51 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-grafana-a[41857]: t=2024-04-14T13:56:51+0000 lvl=info msg="Executing migration" logger=migrator id="remove index def_org_id, def_uid, current_state on alert_instance" 2024-04-14T13:56:51.168 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Apr 14 13:56:51 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-grafana-a[41857]: t=2024-04-14T13:56:51+0000 lvl=info msg="Executing migration" logger=migrator id="remove index def_org_id, current_state on alert_instance" 2024-04-14T13:56:51.168 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Apr 14 13:56:51 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-grafana-a[41857]: t=2024-04-14T13:56:51+0000 lvl=info msg="Executing migration" logger=migrator id="rename def_org_id to rule_org_id in alert_instance" 2024-04-14T13:56:51.168 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Apr 14 13:56:51 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-grafana-a[41857]: t=2024-04-14T13:56:51+0000 lvl=info msg="Executing migration" logger=migrator id="rename def_uid to rule_uid in alert_instance" 2024-04-14T13:56:51.168 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Apr 14 13:56:51 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-grafana-a[41857]: t=2024-04-14T13:56:51+0000 lvl=info msg="Executing migration" logger=migrator id="add index rule_org_id, rule_uid, current_state on alert_instance" 2024-04-14T13:56:51.168 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Apr 14 13:56:51 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-grafana-a[41857]: t=2024-04-14T13:56:51+0000 lvl=info msg="Executing migration" logger=migrator id="add index rule_org_id, current_state on alert_instance" 2024-04-14T13:56:51.168 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Apr 14 13:56:51 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-grafana-a[41857]: t=2024-04-14T13:56:51+0000 lvl=info msg="Executing migration" logger=migrator id="create alert_rule table" 2024-04-14T13:56:51.168 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Apr 14 13:56:51 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-grafana-a[41857]: t=2024-04-14T13:56:51+0000 lvl=info msg="Executing migration" logger=migrator id="add index in alert_rule on org_id and title columns" 2024-04-14T13:56:51.168 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Apr 14 13:56:51 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-grafana-a[41857]: t=2024-04-14T13:56:51+0000 lvl=info msg="Executing migration" logger=migrator id="add index in alert_rule on org_id and uid columns" 2024-04-14T13:56:51.169 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Apr 14 13:56:51 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-grafana-a[41857]: t=2024-04-14T13:56:51+0000 lvl=info msg="Executing migration" logger=migrator id="add index in alert_rule on org_id, namespace_uid, group_uid columns" 2024-04-14T13:56:51.169 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Apr 14 13:56:51 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-grafana-a[41857]: t=2024-04-14T13:56:51+0000 lvl=info msg="Executing migration" logger=migrator id="alter alert_rule table data column to mediumtext in mysql" 2024-04-14T13:56:51.169 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Apr 14 13:56:51 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-grafana-a[41857]: t=2024-04-14T13:56:51+0000 lvl=info msg="Executing migration" logger=migrator id="add column for to alert_rule" 2024-04-14T13:56:51.169 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Apr 14 13:56:51 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-grafana-a[41857]: t=2024-04-14T13:56:51+0000 lvl=info msg="Executing migration" logger=migrator id="add column annotations to alert_rule" 2024-04-14T13:56:51.169 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Apr 14 13:56:51 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-grafana-a[41857]: t=2024-04-14T13:56:51+0000 lvl=info msg="Executing migration" logger=migrator id="add column labels to alert_rule" 2024-04-14T13:56:51.169 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Apr 14 13:56:51 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-grafana-a[41857]: t=2024-04-14T13:56:51+0000 lvl=info msg="Executing migration" logger=migrator id="remove unique index from alert_rule on org_id, title columns" 2024-04-14T13:56:51.169 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Apr 14 13:56:51 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-grafana-a[41857]: t=2024-04-14T13:56:51+0000 lvl=info msg="Executing migration" logger=migrator id="add index in alert_rule on org_id, namespase_uid and title columns" 2024-04-14T13:56:51.169 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Apr 14 13:56:51 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-grafana-a[41857]: t=2024-04-14T13:56:51+0000 lvl=info msg="Executing migration" logger=migrator id="add dashboard_uid column to alert_rule" 2024-04-14T13:56:51.169 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Apr 14 13:56:51 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-grafana-a[41857]: t=2024-04-14T13:56:51+0000 lvl=info msg="Executing migration" logger=migrator id="add panel_id column to alert_rule" 2024-04-14T13:56:51.169 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Apr 14 13:56:51 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-grafana-a[41857]: t=2024-04-14T13:56:51+0000 lvl=info msg="Executing migration" logger=migrator id="add index in alert_rule on org_id, dashboard_uid and panel_id columns" 2024-04-14T13:56:51.169 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Apr 14 13:56:51 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-grafana-a[41857]: t=2024-04-14T13:56:51+0000 lvl=info msg="Executing migration" logger=migrator id="create alert_rule_version table" 2024-04-14T13:56:51.169 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Apr 14 13:56:51 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-grafana-a[41857]: t=2024-04-14T13:56:51+0000 lvl=info msg="Executing migration" logger=migrator id="add index in alert_rule_version table on rule_org_id, rule_uid and version columns" 2024-04-14T13:56:51.169 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Apr 14 13:56:51 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-grafana-a[41857]: t=2024-04-14T13:56:51+0000 lvl=info msg="Executing migration" logger=migrator id="add index in alert_rule_version table on rule_org_id, rule_namespace_uid and rule_group columns" 2024-04-14T13:56:51.169 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Apr 14 13:56:51 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-grafana-a[41857]: t=2024-04-14T13:56:51+0000 lvl=info msg="Executing migration" logger=migrator id="alter alert_rule_version table data column to mediumtext in mysql" 2024-04-14T13:56:51.169 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Apr 14 13:56:51 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-grafana-a[41857]: t=2024-04-14T13:56:51+0000 lvl=info msg="Executing migration" logger=migrator id="add column for to alert_rule_version" 2024-04-14T13:56:51.170 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Apr 14 13:56:51 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-grafana-a[41857]: t=2024-04-14T13:56:51+0000 lvl=info msg="Executing migration" logger=migrator id="add column annotations to alert_rule_version" 2024-04-14T13:56:51.170 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Apr 14 13:56:51 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-grafana-a[41857]: t=2024-04-14T13:56:51+0000 lvl=info msg="Executing migration" logger=migrator id="add column labels to alert_rule_version" 2024-04-14T13:56:51.170 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Apr 14 13:56:51 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-grafana-a[41857]: t=2024-04-14T13:56:51+0000 lvl=info msg="Executing migration" logger=migrator id=create_alert_configuration_table 2024-04-14T13:56:51.170 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Apr 14 13:56:51 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-grafana-a[41857]: t=2024-04-14T13:56:51+0000 lvl=info msg="Executing migration" logger=migrator id="Add column default in alert_configuration" 2024-04-14T13:56:51.170 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Apr 14 13:56:51 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-grafana-a[41857]: t=2024-04-14T13:56:51+0000 lvl=info msg="Executing migration" logger=migrator id="alert alert_configuration alertmanager_configuration column from TEXT to MEDIUMTEXT if mysql" 2024-04-14T13:56:51.170 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Apr 14 13:56:51 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-grafana-a[41857]: t=2024-04-14T13:56:51+0000 lvl=info msg="Executing migration" logger=migrator id="add column org_id in alert_configuration" 2024-04-14T13:56:51.170 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Apr 14 13:56:51 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-grafana-a[41857]: t=2024-04-14T13:56:51+0000 lvl=info msg="Executing migration" logger=migrator id="add index in alert_configuration table on org_id column" 2024-04-14T13:56:51.170 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Apr 14 13:56:51 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-grafana-a[41857]: t=2024-04-14T13:56:51+0000 lvl=info msg="Executing migration" logger=migrator id=create_ngalert_configuration_table 2024-04-14T13:56:51.170 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Apr 14 13:56:51 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-grafana-a[41857]: t=2024-04-14T13:56:51+0000 lvl=info msg="Executing migration" logger=migrator id="add index in ngalert_configuration on org_id column" 2024-04-14T13:56:51.170 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Apr 14 13:56:51 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-grafana-a[41857]: t=2024-04-14T13:56:51+0000 lvl=info msg="Executing migration" logger=migrator id="clear migration entry \"remove unified alerting data\"" 2024-04-14T13:56:51.170 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Apr 14 13:56:51 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-grafana-a[41857]: t=2024-04-14T13:56:51+0000 lvl=info msg="Executing migration" logger=migrator id="move dashboard alerts to unified alerting" 2024-04-14T13:56:51.170 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Apr 14 13:56:51 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-grafana-a[41857]: t=2024-04-14T13:56:51+0000 lvl=info msg="Executing migration" logger=migrator id="create library_element table v1" 2024-04-14T13:56:51.170 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Apr 14 13:56:51 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-grafana-a[41857]: t=2024-04-14T13:56:51+0000 lvl=info msg="Executing migration" logger=migrator id="add index library_element org_id-folder_id-name-kind" 2024-04-14T13:56:51.170 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Apr 14 13:56:51 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-grafana-a[41857]: t=2024-04-14T13:56:51+0000 lvl=info msg="Executing migration" logger=migrator id="create library_element_connection table v1" 2024-04-14T13:56:51.171 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Apr 14 13:56:51 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-grafana-a[41857]: t=2024-04-14T13:56:51+0000 lvl=info msg="Executing migration" logger=migrator id="add index library_element_connection element_id-kind-connection_id" 2024-04-14T13:56:51.171 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Apr 14 13:56:51 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-grafana-a[41857]: t=2024-04-14T13:56:51+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index library_element org_id_uid" 2024-04-14T13:56:51.171 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Apr 14 13:56:51 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-grafana-a[41857]: t=2024-04-14T13:56:51+0000 lvl=info msg="Executing migration" logger=migrator id="clone move dashboard alerts to unified alerting" 2024-04-14T13:56:51.171 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Apr 14 13:56:51 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-grafana-a[41857]: t=2024-04-14T13:56:51+0000 lvl=info msg="Executing migration" logger=migrator id="create data_keys table" 2024-04-14T13:56:51.171 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Apr 14 13:56:51 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-grafana-a[41857]: t=2024-04-14T13:56:51+0000 lvl=info msg="Executing migration" logger=migrator id="create kv_store table v1" 2024-04-14T13:56:51.171 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Apr 14 13:56:51 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-grafana-a[41857]: t=2024-04-14T13:56:51+0000 lvl=info msg="Executing migration" logger=migrator id="add index kv_store.org_id-namespace-key" 2024-04-14T13:56:51.171 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Apr 14 13:56:51 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-grafana-a[41857]: t=2024-04-14T13:56:51+0000 lvl=info msg="Executing migration" logger=migrator id="update dashboard_uid and panel_id from existing annotations" 2024-04-14T13:56:51.171 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Apr 14 13:56:51 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-grafana-a[41857]: t=2024-04-14T13:56:51+0000 lvl=info msg="Executing migration" logger=migrator id="create permission table" 2024-04-14T13:56:51.171 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Apr 14 13:56:51 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-grafana-a[41857]: t=2024-04-14T13:56:51+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index permission.role_id" 2024-04-14T13:56:51.171 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Apr 14 13:56:51 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-grafana-a[41857]: t=2024-04-14T13:56:51+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index role_id_action_scope" 2024-04-14T13:56:51.171 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Apr 14 13:56:51 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-grafana-a[41857]: t=2024-04-14T13:56:51+0000 lvl=info msg="Executing migration" logger=migrator id="create role table" 2024-04-14T13:56:51.171 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Apr 14 13:56:51 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-grafana-a[41857]: t=2024-04-14T13:56:51+0000 lvl=info msg="Executing migration" logger=migrator id="add column display_name" 2024-04-14T13:56:51.171 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Apr 14 13:56:51 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-grafana-a[41857]: t=2024-04-14T13:56:51+0000 lvl=info msg="Executing migration" logger=migrator id="add column group_name" 2024-04-14T13:56:51.171 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Apr 14 13:56:51 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-grafana-a[41857]: t=2024-04-14T13:56:51+0000 lvl=info msg="Executing migration" logger=migrator id="add index role.org_id" 2024-04-14T13:56:51.171 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Apr 14 13:56:51 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-grafana-a[41857]: t=2024-04-14T13:56:51+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index role_org_id_name" 2024-04-14T13:56:51.172 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Apr 14 13:56:51 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-grafana-a[41857]: t=2024-04-14T13:56:51+0000 lvl=info msg="Executing migration" logger=migrator id="add index role_org_id_uid" 2024-04-14T13:56:51.172 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Apr 14 13:56:51 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-grafana-a[41857]: t=2024-04-14T13:56:51+0000 lvl=info msg="Executing migration" logger=migrator id="create team role table" 2024-04-14T13:56:51.172 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Apr 14 13:56:51 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-grafana-a[41857]: t=2024-04-14T13:56:51+0000 lvl=info msg="Executing migration" logger=migrator id="add index team_role.org_id" 2024-04-14T13:56:51.172 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Apr 14 13:56:51 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-grafana-a[41857]: t=2024-04-14T13:56:51+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index team_role_org_id_team_id_role_id" 2024-04-14T13:56:51.172 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Apr 14 13:56:51 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-grafana-a[41857]: t=2024-04-14T13:56:51+0000 lvl=info msg="Executing migration" logger=migrator id="add index team_role.team_id" 2024-04-14T13:56:51.172 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Apr 14 13:56:51 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-grafana-a[41857]: t=2024-04-14T13:56:51+0000 lvl=info msg="Executing migration" logger=migrator id="create user role table" 2024-04-14T13:56:51.172 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Apr 14 13:56:51 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-grafana-a[41857]: t=2024-04-14T13:56:51+0000 lvl=info msg="Executing migration" logger=migrator id="add index user_role.org_id" 2024-04-14T13:56:51.172 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Apr 14 13:56:51 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-grafana-a[41857]: t=2024-04-14T13:56:51+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index user_role_org_id_user_id_role_id" 2024-04-14T13:56:51.172 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Apr 14 13:56:51 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-grafana-a[41857]: t=2024-04-14T13:56:51+0000 lvl=info msg="Executing migration" logger=migrator id="add index user_role.user_id" 2024-04-14T13:56:51.172 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Apr 14 13:56:51 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-grafana-a[41857]: t=2024-04-14T13:56:51+0000 lvl=info msg="Executing migration" logger=migrator id="create builtin role table" 2024-04-14T13:56:51.172 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Apr 14 13:56:51 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-grafana-a[41857]: t=2024-04-14T13:56:51+0000 lvl=info msg="Executing migration" logger=migrator id="add index builtin_role.role_id" 2024-04-14T13:56:51.172 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Apr 14 13:56:51 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-grafana-a[41857]: t=2024-04-14T13:56:51+0000 lvl=info msg="Executing migration" logger=migrator id="add index builtin_role.name" 2024-04-14T13:56:51.172 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Apr 14 13:56:51 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-grafana-a[41857]: t=2024-04-14T13:56:51+0000 lvl=info msg="Executing migration" logger=migrator id="Add column org_id to builtin_role table" 2024-04-14T13:56:51.172 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Apr 14 13:56:51 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-grafana-a[41857]: t=2024-04-14T13:56:51+0000 lvl=info msg="Executing migration" logger=migrator id="add index builtin_role.org_id" 2024-04-14T13:56:51.172 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Apr 14 13:56:51 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-grafana-a[41857]: t=2024-04-14T13:56:51+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index builtin_role_org_id_role_id_role" 2024-04-14T13:56:51.173 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Apr 14 13:56:51 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-grafana-a[41857]: t=2024-04-14T13:56:51+0000 lvl=info msg="Executing migration" logger=migrator id="Remove unique index role_org_id_uid" 2024-04-14T13:56:51.173 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Apr 14 13:56:51 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-grafana-a[41857]: t=2024-04-14T13:56:51+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index role.uid" 2024-04-14T13:56:51.173 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Apr 14 13:56:51 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-grafana-a[41857]: t=2024-04-14T13:56:51+0000 lvl=info msg="Executing migration" logger=migrator id="create seed assignment table" 2024-04-14T13:56:51.173 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Apr 14 13:56:51 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-grafana-a[41857]: t=2024-04-14T13:56:51+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index builtin_role_role_name" 2024-04-14T13:56:51.173 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Apr 14 13:56:51 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-grafana-a[41857]: t=2024-04-14T13:56:51+0000 lvl=info msg="migrations completed" logger=migrator performed=381 skipped=0 duration=523.177837ms 2024-04-14T13:56:51.173 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Apr 14 13:56:51 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-grafana-a[41857]: t=2024-04-14T13:56:51+0000 lvl=info msg="Created default organization" logger=sqlstore 2024-04-14T13:56:51.173 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Apr 14 13:56:51 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-grafana-a[41857]: t=2024-04-14T13:56:51+0000 lvl=info msg="Initialising plugins" logger=plugin.manager 2024-04-14T13:56:51.173 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Apr 14 13:56:51 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-grafana-a[41857]: t=2024-04-14T13:56:51+0000 lvl=info msg="Plugin registered" logger=plugin.manager pluginId=input 2024-04-14T13:56:51.173 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Apr 14 13:56:51 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-grafana-a[41857]: t=2024-04-14T13:56:51+0000 lvl=info msg="Plugin registered" logger=plugin.manager pluginId=vonage-status-panel 2024-04-14T13:56:51.173 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Apr 14 13:56:51 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-grafana-a[41857]: t=2024-04-14T13:56:51+0000 lvl=info msg="Plugin registered" logger=plugin.manager pluginId=grafana-piechart-panel 2024-04-14T13:56:51.447 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:56:51 smithi122 ceph-mon[25721]: pgmap v49: 33 pgs: 33 active+clean; 577 KiB data, 49 MiB used, 715 GiB / 715 GiB avail; 1023 B/s wr, 1 op/s 2024-04-14T13:56:51.447 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:56:51 smithi122 ceph-mon[25721]: osdmap e63: 8 total, 8 up, 8 in 2024-04-14T13:56:51.447 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:56:51 smithi122 ceph-mon[25721]: from='client.? 172.21.15.69:0/402342286' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "z.rgw.log","app": "rgw"}]: dispatch 2024-04-14T13:56:51.447 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:56:51 smithi122 ceph-mon[25721]: from='mgr.14448 172.21.15.69:0/453473650' entity='mgr.y' 2024-04-14T13:56:51.447 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:56:51 smithi122 ceph-mon[25721]: from='mgr.14448 172.21.15.69:0/453473650' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T13:56:51.447 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:56:51 smithi122 ceph-mon[25721]: from='mgr.14448 172.21.15.69:0/453473650' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-14T13:56:51.447 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Apr 14 13:56:51 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-grafana-a[41857]: t=2024-04-14T13:56:51+0000 lvl=info msg="Live Push Gateway initialization" logger=live.push_http 2024-04-14T13:56:51.448 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Apr 14 13:56:51 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-grafana-a[41857]: t=2024-04-14T13:56:51+0000 lvl=warn msg="[Deprecated] the datasource provisioning config is outdated. please upgrade" logger=provisioning.datasources filename=/etc/grafana/provisioning/datasources/ceph-dashboard.yml 2024-04-14T13:56:51.448 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Apr 14 13:56:51 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-grafana-a[41857]: t=2024-04-14T13:56:51+0000 lvl=info msg="inserting datasource from configuration " logger=provisioning.datasources name=Dashboard1 uid=P43CA22E17D0F9596 2024-04-14T13:56:51.448 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Apr 14 13:56:51 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-grafana-a[41857]: t=2024-04-14T13:56:51+0000 lvl=info msg="warming cache for startup" logger=ngalert 2024-04-14T13:56:51.448 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Apr 14 13:56:51 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-grafana-a[41857]: t=2024-04-14T13:56:51+0000 lvl=info msg="starting MultiOrg Alertmanager" logger=ngalert.multiorg.alertmanager 2024-04-14T13:56:51.448 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Apr 14 13:56:51 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-grafana-a[41857]: t=2024-04-14T13:56:51+0000 lvl=info msg="HTTP Server Listen" logger=http.server address=[::]:3000 protocol=https subUrl= socket= 2024-04-14T13:56:51.567 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:56:51 smithi069 ceph-mon[24818]: pgmap v49: 33 pgs: 33 active+clean; 577 KiB data, 49 MiB used, 715 GiB / 715 GiB avail; 1023 B/s wr, 1 op/s 2024-04-14T13:56:51.567 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:56:51 smithi069 ceph-mon[24818]: osdmap e63: 8 total, 8 up, 8 in 2024-04-14T13:56:51.567 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:56:51 smithi069 ceph-mon[24818]: from='client.? 172.21.15.69:0/402342286' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "z.rgw.log","app": "rgw"}]: dispatch 2024-04-14T13:56:51.567 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:56:51 smithi069 ceph-mon[24818]: from='mgr.14448 172.21.15.69:0/453473650' entity='mgr.y' 2024-04-14T13:56:51.567 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:56:51 smithi069 ceph-mon[24818]: from='mgr.14448 172.21.15.69:0/453473650' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T13:56:51.567 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:56:51 smithi069 ceph-mon[24818]: from='mgr.14448 172.21.15.69:0/453473650' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-14T13:56:51.567 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:56:51 smithi069 ceph-mon[29472]: pgmap v49: 33 pgs: 33 active+clean; 577 KiB data, 49 MiB used, 715 GiB / 715 GiB avail; 1023 B/s wr, 1 op/s 2024-04-14T13:56:51.567 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:56:51 smithi069 ceph-mon[29472]: osdmap e63: 8 total, 8 up, 8 in 2024-04-14T13:56:51.568 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:56:51 smithi069 ceph-mon[29472]: from='client.? 172.21.15.69:0/402342286' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "z.rgw.log","app": "rgw"}]: dispatch 2024-04-14T13:56:51.568 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:56:51 smithi069 ceph-mon[29472]: from='mgr.14448 172.21.15.69:0/453473650' entity='mgr.y' 2024-04-14T13:56:51.568 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:56:51 smithi069 ceph-mon[29472]: from='mgr.14448 172.21.15.69:0/453473650' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T13:56:51.568 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:56:51 smithi069 ceph-mon[29472]: from='mgr.14448 172.21.15.69:0/453473650' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-14T13:56:52.567 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:56:52 smithi069 ceph-mon[24818]: from='client.? 172.21.15.69:0/402342286' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "z.rgw.log","app": "rgw"}]': finished 2024-04-14T13:56:52.567 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:56:52 smithi069 ceph-mon[24818]: osdmap e64: 8 total, 8 up, 8 in 2024-04-14T13:56:52.567 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:56:52 smithi069 ceph-mon[29472]: from='client.? 172.21.15.69:0/402342286' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "z.rgw.log","app": "rgw"}]': finished 2024-04-14T13:56:52.567 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:56:52 smithi069 ceph-mon[29472]: osdmap e64: 8 total, 8 up, 8 in 2024-04-14T13:56:52.595 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:56:52 smithi122 ceph-mon[25721]: from='client.? 172.21.15.69:0/402342286' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "z.rgw.log","app": "rgw"}]': finished 2024-04-14T13:56:52.595 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:56:52 smithi122 ceph-mon[25721]: osdmap e64: 8 total, 8 up, 8 in 2024-04-14T13:56:53.653 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:56:53 smithi069 ceph-mon[29472]: pgmap v52: 65 pgs: 32 unknown, 33 active+clean; 579 KiB data, 49 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 2.0 KiB/s wr, 6 op/s 2024-04-14T13:56:53.653 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:56:53 smithi069 ceph-mon[29472]: osdmap e65: 8 total, 8 up, 8 in 2024-04-14T13:56:53.654 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:56:53 smithi069 ceph-mon[29472]: from='client.? 172.21.15.69:0/402342286' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "z.rgw.control","app": "rgw"}]: dispatch 2024-04-14T13:56:53.654 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:56:53 smithi069 ceph-mon[29472]: from='mgr.14448 172.21.15.69:0/453473650' entity='mgr.y' 2024-04-14T13:56:53.654 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:56:53 smithi069 ceph-mon[24818]: pgmap v52: 65 pgs: 32 unknown, 33 active+clean; 579 KiB data, 49 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 2.0 KiB/s wr, 6 op/s 2024-04-14T13:56:53.654 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:56:53 smithi069 ceph-mon[24818]: osdmap e65: 8 total, 8 up, 8 in 2024-04-14T13:56:53.654 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:56:53 smithi069 ceph-mon[24818]: from='client.? 172.21.15.69:0/402342286' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "z.rgw.control","app": "rgw"}]: dispatch 2024-04-14T13:56:53.654 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:56:53 smithi069 ceph-mon[24818]: from='mgr.14448 172.21.15.69:0/453473650' entity='mgr.y' 2024-04-14T13:56:53.783 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:56:53 smithi122 ceph-mon[25721]: pgmap v52: 65 pgs: 32 unknown, 33 active+clean; 579 KiB data, 49 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 2.0 KiB/s wr, 6 op/s 2024-04-14T13:56:53.783 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:56:53 smithi122 ceph-mon[25721]: osdmap e65: 8 total, 8 up, 8 in 2024-04-14T13:56:53.783 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:56:53 smithi122 ceph-mon[25721]: from='client.? 172.21.15.69:0/402342286' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "z.rgw.control","app": "rgw"}]: dispatch 2024-04-14T13:56:53.783 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:56:53 smithi122 ceph-mon[25721]: from='mgr.14448 172.21.15.69:0/453473650' entity='mgr.y' 2024-04-14T13:56:54.789 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:56:54 smithi069 ceph-mon[24818]: from='client.? 172.21.15.69:0/402342286' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "z.rgw.control","app": "rgw"}]': finished 2024-04-14T13:56:54.789 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:56:54 smithi069 ceph-mon[24818]: osdmap e66: 8 total, 8 up, 8 in 2024-04-14T13:56:54.789 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:56:54 smithi069 ceph-mon[24818]: from='mgr.14448 172.21.15.69:0/453473650' entity='mgr.y' 2024-04-14T13:56:54.789 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:56:54 smithi069 ceph-mon[24818]: from='mgr.14448 172.21.15.69:0/453473650' entity='mgr.y' 2024-04-14T13:56:54.789 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:56:54 smithi069 ceph-mon[24818]: from='mgr.14448 172.21.15.69:0/453473650' entity='mgr.y' 2024-04-14T13:56:54.789 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:56:54 smithi069 ceph-mon[29472]: from='client.? 172.21.15.69:0/402342286' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "z.rgw.control","app": "rgw"}]': finished 2024-04-14T13:56:54.790 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:56:54 smithi069 ceph-mon[29472]: osdmap e66: 8 total, 8 up, 8 in 2024-04-14T13:56:54.790 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:56:54 smithi069 ceph-mon[29472]: from='mgr.14448 172.21.15.69:0/453473650' entity='mgr.y' 2024-04-14T13:56:54.790 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:56:54 smithi069 ceph-mon[29472]: from='mgr.14448 172.21.15.69:0/453473650' entity='mgr.y' 2024-04-14T13:56:54.790 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:56:54 smithi069 ceph-mon[29472]: from='mgr.14448 172.21.15.69:0/453473650' entity='mgr.y' 2024-04-14T13:56:54.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:56:54 smithi122 ceph-mon[25721]: from='client.? 172.21.15.69:0/402342286' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "z.rgw.control","app": "rgw"}]': finished 2024-04-14T13:56:54.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:56:54 smithi122 ceph-mon[25721]: osdmap e66: 8 total, 8 up, 8 in 2024-04-14T13:56:54.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:56:54 smithi122 ceph-mon[25721]: from='mgr.14448 172.21.15.69:0/453473650' entity='mgr.y' 2024-04-14T13:56:54.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:56:54 smithi122 ceph-mon[25721]: from='mgr.14448 172.21.15.69:0/453473650' entity='mgr.y' 2024-04-14T13:56:54.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:56:54 smithi122 ceph-mon[25721]: from='mgr.14448 172.21.15.69:0/453473650' entity='mgr.y' 2024-04-14T13:56:55.817 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:56:55 smithi069 ceph-mon[24818]: pgmap v55: 97 pgs: 64 unknown, 33 active+clean; 579 KiB data, 49 MiB used, 715 GiB / 715 GiB avail; 7.5 KiB/s rd, 2.0 KiB/s wr, 9 op/s 2024-04-14T13:56:55.817 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:56:55 smithi069 ceph-mon[24818]: Reconfiguring alertmanager.a (dependencies changed)... 2024-04-14T13:56:55.818 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:56:55 smithi069 ceph-mon[24818]: Reconfiguring daemon alertmanager.a on smithi069 2024-04-14T13:56:55.818 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:56:55 smithi069 ceph-mon[24818]: osdmap e67: 8 total, 8 up, 8 in 2024-04-14T13:56:55.818 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:56:55 smithi069 ceph-mon[24818]: from='client.? 172.21.15.69:0/1956499905' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "z.rgw.meta","app": "rgw"}]: dispatch 2024-04-14T13:56:55.818 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:56:55 smithi069 ceph-mon[24818]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "z.rgw.meta","app": "rgw"}]: dispatch 2024-04-14T13:56:55.818 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:56:55 smithi069 ceph-mon[24818]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "z.rgw.meta","app": "rgw"}]': finished 2024-04-14T13:56:55.818 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:56:55 smithi069 ceph-mon[24818]: osdmap e68: 8 total, 8 up, 8 in 2024-04-14T13:56:55.818 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:56:55 smithi069 ceph-mon[24818]: from='client.? 172.21.15.69:0/1956499905' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "z.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-04-14T13:56:55.818 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:56:55 smithi069 ceph-mon[24818]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "z.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-04-14T13:56:55.818 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:56:55 smithi069 ceph-mon[29472]: pgmap v55: 97 pgs: 64 unknown, 33 active+clean; 579 KiB data, 49 MiB used, 715 GiB / 715 GiB avail; 7.5 KiB/s rd, 2.0 KiB/s wr, 9 op/s 2024-04-14T13:56:55.818 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:56:55 smithi069 ceph-mon[29472]: Reconfiguring alertmanager.a (dependencies changed)... 2024-04-14T13:56:55.818 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:56:55 smithi069 ceph-mon[29472]: Reconfiguring daemon alertmanager.a on smithi069 2024-04-14T13:56:55.819 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:56:55 smithi069 ceph-mon[29472]: osdmap e67: 8 total, 8 up, 8 in 2024-04-14T13:56:55.819 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:56:55 smithi069 ceph-mon[29472]: from='client.? 172.21.15.69:0/1956499905' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "z.rgw.meta","app": "rgw"}]: dispatch 2024-04-14T13:56:55.819 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:56:55 smithi069 ceph-mon[29472]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "z.rgw.meta","app": "rgw"}]: dispatch 2024-04-14T13:56:55.819 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:56:55 smithi069 ceph-mon[29472]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "z.rgw.meta","app": "rgw"}]': finished 2024-04-14T13:56:55.819 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:56:55 smithi069 ceph-mon[29472]: osdmap e68: 8 total, 8 up, 8 in 2024-04-14T13:56:55.819 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:56:55 smithi069 ceph-mon[29472]: from='client.? 172.21.15.69:0/1956499905' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "z.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-04-14T13:56:55.819 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:56:55 smithi069 ceph-mon[29472]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "z.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-04-14T13:56:55.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:56:55 smithi122 ceph-mon[25721]: pgmap v55: 97 pgs: 64 unknown, 33 active+clean; 579 KiB data, 49 MiB used, 715 GiB / 715 GiB avail; 7.5 KiB/s rd, 2.0 KiB/s wr, 9 op/s 2024-04-14T13:56:55.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:56:55 smithi122 ceph-mon[25721]: Reconfiguring alertmanager.a (dependencies changed)... 2024-04-14T13:56:55.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:56:55 smithi122 ceph-mon[25721]: Reconfiguring daemon alertmanager.a on smithi069 2024-04-14T13:56:55.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:56:55 smithi122 ceph-mon[25721]: osdmap e67: 8 total, 8 up, 8 in 2024-04-14T13:56:55.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:56:55 smithi122 ceph-mon[25721]: from='client.? 172.21.15.69:0/1956499905' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "z.rgw.meta","app": "rgw"}]: dispatch 2024-04-14T13:56:55.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:56:55 smithi122 ceph-mon[25721]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "z.rgw.meta","app": "rgw"}]: dispatch 2024-04-14T13:56:55.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:56:55 smithi122 ceph-mon[25721]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "z.rgw.meta","app": "rgw"}]': finished 2024-04-14T13:56:55.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:56:55 smithi122 ceph-mon[25721]: osdmap e68: 8 total, 8 up, 8 in 2024-04-14T13:56:55.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:56:55 smithi122 ceph-mon[25721]: from='client.? 172.21.15.69:0/1956499905' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "z.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-04-14T13:56:55.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:56:55 smithi122 ceph-mon[25721]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "z.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-04-14T13:56:57.488 INFO:teuthology.orchestra.run.smithi069.stdout:{ 2024-04-14T13:56:57.488 INFO:teuthology.orchestra.run.smithi069.stdout: "id": "ac50983a-155d-4f90-adca-1b1220292f4d", 2024-04-14T13:56:57.488 INFO:teuthology.orchestra.run.smithi069.stdout: "epoch": 1, 2024-04-14T13:56:57.488 INFO:teuthology.orchestra.run.smithi069.stdout: "predecessor_uuid": "0eee9837-2986-4a88-8277-5da8368ceedf", 2024-04-14T13:56:57.489 INFO:teuthology.orchestra.run.smithi069.stdout: "sync_status": [], 2024-04-14T13:56:57.489 INFO:teuthology.orchestra.run.smithi069.stdout: "period_map": { 2024-04-14T13:56:57.489 INFO:teuthology.orchestra.run.smithi069.stdout: "id": "ac50983a-155d-4f90-adca-1b1220292f4d", 2024-04-14T13:56:57.489 INFO:teuthology.orchestra.run.smithi069.stdout: "zonegroups": [ 2024-04-14T13:56:57.489 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-14T13:56:57.489 INFO:teuthology.orchestra.run.smithi069.stdout: "id": "ed3c12f2-b3fa-4f9e-ba8c-e639cb70a4bf", 2024-04-14T13:56:57.489 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "default", 2024-04-14T13:56:57.489 INFO:teuthology.orchestra.run.smithi069.stdout: "api_name": "default", 2024-04-14T13:56:57.489 INFO:teuthology.orchestra.run.smithi069.stdout: "is_master": "true", 2024-04-14T13:56:57.489 INFO:teuthology.orchestra.run.smithi069.stdout: "endpoints": [], 2024-04-14T13:56:57.489 INFO:teuthology.orchestra.run.smithi069.stdout: "hostnames": [], 2024-04-14T13:56:57.489 INFO:teuthology.orchestra.run.smithi069.stdout: "hostnames_s3website": [], 2024-04-14T13:56:57.489 INFO:teuthology.orchestra.run.smithi069.stdout: "master_zone": "a4450052-7331-4c8a-956b-296946e49785", 2024-04-14T13:56:57.489 INFO:teuthology.orchestra.run.smithi069.stdout: "zones": [ 2024-04-14T13:56:57.489 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-14T13:56:57.490 INFO:teuthology.orchestra.run.smithi069.stdout: "id": "a4450052-7331-4c8a-956b-296946e49785", 2024-04-14T13:56:57.490 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "z", 2024-04-14T13:56:57.490 INFO:teuthology.orchestra.run.smithi069.stdout: "endpoints": [], 2024-04-14T13:56:57.490 INFO:teuthology.orchestra.run.smithi069.stdout: "log_meta": "false", 2024-04-14T13:56:57.490 INFO:teuthology.orchestra.run.smithi069.stdout: "log_data": "false", 2024-04-14T13:56:57.490 INFO:teuthology.orchestra.run.smithi069.stdout: "bucket_index_max_shards": 11, 2024-04-14T13:56:57.490 INFO:teuthology.orchestra.run.smithi069.stdout: "read_only": "false", 2024-04-14T13:56:57.490 INFO:teuthology.orchestra.run.smithi069.stdout: "tier_type": "", 2024-04-14T13:56:57.490 INFO:teuthology.orchestra.run.smithi069.stdout: "sync_from_all": "true", 2024-04-14T13:56:57.490 INFO:teuthology.orchestra.run.smithi069.stdout: "sync_from": [], 2024-04-14T13:56:57.490 INFO:teuthology.orchestra.run.smithi069.stdout: "redirect_zone": "" 2024-04-14T13:56:57.490 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-14T13:56:57.490 INFO:teuthology.orchestra.run.smithi069.stdout: ], 2024-04-14T13:56:57.491 INFO:teuthology.orchestra.run.smithi069.stdout: "placement_targets": [ 2024-04-14T13:56:57.491 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-14T13:56:57.491 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "default-placement", 2024-04-14T13:56:57.491 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-04-14T13:56:57.491 INFO:teuthology.orchestra.run.smithi069.stdout: "storage_classes": [ 2024-04-14T13:56:57.491 INFO:teuthology.orchestra.run.smithi069.stdout: "STANDARD" 2024-04-14T13:56:57.491 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2024-04-14T13:56:57.491 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-14T13:56:57.491 INFO:teuthology.orchestra.run.smithi069.stdout: ], 2024-04-14T13:56:57.491 INFO:teuthology.orchestra.run.smithi069.stdout: "default_placement": "default-placement", 2024-04-14T13:56:57.491 INFO:teuthology.orchestra.run.smithi069.stdout: "realm_id": "b0e084dc-c5d6-4c72-bfd7-73d3bab36e46", 2024-04-14T13:56:57.491 INFO:teuthology.orchestra.run.smithi069.stdout: "sync_policy": { 2024-04-14T13:56:57.491 INFO:teuthology.orchestra.run.smithi069.stdout: "groups": [] 2024-04-14T13:56:57.492 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-14T13:56:57.492 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-14T13:56:57.492 INFO:teuthology.orchestra.run.smithi069.stdout: ], 2024-04-14T13:56:57.492 INFO:teuthology.orchestra.run.smithi069.stdout: "short_zone_ids": [ 2024-04-14T13:56:57.492 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-14T13:56:57.492 INFO:teuthology.orchestra.run.smithi069.stdout: "key": "a4450052-7331-4c8a-956b-296946e49785", 2024-04-14T13:56:57.492 INFO:teuthology.orchestra.run.smithi069.stdout: "val": 4236520483 2024-04-14T13:56:57.492 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-14T13:56:57.492 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2024-04-14T13:56:57.492 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-14T13:56:57.492 INFO:teuthology.orchestra.run.smithi069.stdout: "master_zonegroup": "ed3c12f2-b3fa-4f9e-ba8c-e639cb70a4bf", 2024-04-14T13:56:57.492 INFO:teuthology.orchestra.run.smithi069.stdout: "master_zone": "a4450052-7331-4c8a-956b-296946e49785", 2024-04-14T13:56:57.492 INFO:teuthology.orchestra.run.smithi069.stdout: "period_config": { 2024-04-14T13:56:57.492 INFO:teuthology.orchestra.run.smithi069.stdout: "bucket_quota": { 2024-04-14T13:56:57.492 INFO:teuthology.orchestra.run.smithi069.stdout: "enabled": false, 2024-04-14T13:56:57.493 INFO:teuthology.orchestra.run.smithi069.stdout: "check_on_raw": false, 2024-04-14T13:56:57.493 INFO:teuthology.orchestra.run.smithi069.stdout: "max_size": -1, 2024-04-14T13:56:57.493 INFO:teuthology.orchestra.run.smithi069.stdout: "max_size_kb": 0, 2024-04-14T13:56:57.493 INFO:teuthology.orchestra.run.smithi069.stdout: "max_objects": -1 2024-04-14T13:56:57.493 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-14T13:56:57.493 INFO:teuthology.orchestra.run.smithi069.stdout: "user_quota": { 2024-04-14T13:56:57.493 INFO:teuthology.orchestra.run.smithi069.stdout: "enabled": false, 2024-04-14T13:56:57.493 INFO:teuthology.orchestra.run.smithi069.stdout: "check_on_raw": false, 2024-04-14T13:56:57.494 INFO:teuthology.orchestra.run.smithi069.stdout: "max_size": -1, 2024-04-14T13:56:57.494 INFO:teuthology.orchestra.run.smithi069.stdout: "max_size_kb": 0, 2024-04-14T13:56:57.494 INFO:teuthology.orchestra.run.smithi069.stdout: "max_objects": -1 2024-04-14T13:56:57.494 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-14T13:56:57.494 INFO:teuthology.orchestra.run.smithi069.stdout: "user_ratelimit": { 2024-04-14T13:56:57.494 INFO:teuthology.orchestra.run.smithi069.stdout: "max_read_ops": 0, 2024-04-14T13:56:57.494 INFO:teuthology.orchestra.run.smithi069.stdout: "max_write_ops": 0, 2024-04-14T13:56:57.494 INFO:teuthology.orchestra.run.smithi069.stdout: "max_read_bytes": 0, 2024-04-14T13:56:57.494 INFO:teuthology.orchestra.run.smithi069.stdout: "max_write_bytes": 0, 2024-04-14T13:56:57.494 INFO:teuthology.orchestra.run.smithi069.stdout: "enabled": false 2024-04-14T13:56:57.494 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-14T13:56:57.494 INFO:teuthology.orchestra.run.smithi069.stdout: "bucket_ratelimit": { 2024-04-14T13:56:57.494 INFO:teuthology.orchestra.run.smithi069.stdout: "max_read_ops": 0, 2024-04-14T13:56:57.494 INFO:teuthology.orchestra.run.smithi069.stdout: "max_write_ops": 0, 2024-04-14T13:56:57.495 INFO:teuthology.orchestra.run.smithi069.stdout: "max_read_bytes": 0, 2024-04-14T13:56:57.495 INFO:teuthology.orchestra.run.smithi069.stdout: "max_write_bytes": 0, 2024-04-14T13:56:57.495 INFO:teuthology.orchestra.run.smithi069.stdout: "enabled": false 2024-04-14T13:56:57.495 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-14T13:56:57.495 INFO:teuthology.orchestra.run.smithi069.stdout: "anonymous_ratelimit": { 2024-04-14T13:56:57.495 INFO:teuthology.orchestra.run.smithi069.stdout: "max_read_ops": 0, 2024-04-14T13:56:57.495 INFO:teuthology.orchestra.run.smithi069.stdout: "max_write_ops": 0, 2024-04-14T13:56:57.495 INFO:teuthology.orchestra.run.smithi069.stdout: "max_read_bytes": 0, 2024-04-14T13:56:57.495 INFO:teuthology.orchestra.run.smithi069.stdout: "max_write_bytes": 0, 2024-04-14T13:56:57.495 INFO:teuthology.orchestra.run.smithi069.stdout: "enabled": false 2024-04-14T13:56:57.495 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-14T13:56:57.495 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-14T13:56:57.495 INFO:teuthology.orchestra.run.smithi069.stdout: "realm_id": "b0e084dc-c5d6-4c72-bfd7-73d3bab36e46", 2024-04-14T13:56:57.495 INFO:teuthology.orchestra.run.smithi069.stdout: "realm_name": "r", 2024-04-14T13:56:57.495 INFO:teuthology.orchestra.run.smithi069.stdout: "realm_epoch": 2 2024-04-14T13:56:57.496 INFO:teuthology.orchestra.run.smithi069.stdout:} 2024-04-14T13:56:57.567 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:56:57 smithi069 ceph-mon[24818]: pgmap v58: 129 pgs: 32 creating+peering, 97 active+clean; 579 KiB data, 51 MiB used, 715 GiB / 715 GiB avail; 255 B/s rd, 511 B/s wr, 1 op/s 2024-04-14T13:56:57.567 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:56:57 smithi069 ceph-mon[24818]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "z.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]': finished 2024-04-14T13:56:57.567 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:56:57 smithi069 ceph-mon[24818]: osdmap e69: 8 total, 8 up, 8 in 2024-04-14T13:56:57.567 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:56:57 smithi069 ceph-mon[24818]: from='client.? 172.21.15.69:0/1956499905' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "z.rgw.meta", "var": "pg_num_min", "val": "8"}]: dispatch 2024-04-14T13:56:57.567 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:56:57 smithi069 ceph-mon[24818]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "z.rgw.meta", "var": "pg_num_min", "val": "8"}]: dispatch 2024-04-14T13:56:57.568 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:56:57 smithi069 ceph-mon[29472]: pgmap v58: 129 pgs: 32 creating+peering, 97 active+clean; 579 KiB data, 51 MiB used, 715 GiB / 715 GiB avail; 255 B/s rd, 511 B/s wr, 1 op/s 2024-04-14T13:56:57.568 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:56:57 smithi069 ceph-mon[29472]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "z.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]': finished 2024-04-14T13:56:57.568 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:56:57 smithi069 ceph-mon[29472]: osdmap e69: 8 total, 8 up, 8 in 2024-04-14T13:56:57.568 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:56:57 smithi069 ceph-mon[29472]: from='client.? 172.21.15.69:0/1956499905' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "z.rgw.meta", "var": "pg_num_min", "val": "8"}]: dispatch 2024-04-14T13:56:57.568 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:56:57 smithi069 ceph-mon[29472]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "z.rgw.meta", "var": "pg_num_min", "val": "8"}]: dispatch 2024-04-14T13:56:57.568 INFO:journalctl@ceph.alertmanager.a.smithi069.stdout:Apr 14 13:56:57 smithi069 systemd[1]: Stopping Ceph alertmanager.a for dc326fc2-fa65-11ee-bc8e-c7b262605968... 2024-04-14T13:56:57.595 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Apr 14 13:56:57 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-x[26913]: ::ffff:172.21.15.122 - - [14/Apr/2024:13:56:57] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-04-14T13:56:57.595 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:56:57 smithi122 ceph-mon[25721]: pgmap v58: 129 pgs: 32 creating+peering, 97 active+clean; 579 KiB data, 51 MiB used, 715 GiB / 715 GiB avail; 255 B/s rd, 511 B/s wr, 1 op/s 2024-04-14T13:56:57.596 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:56:57 smithi122 ceph-mon[25721]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "z.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]': finished 2024-04-14T13:56:57.596 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:56:57 smithi122 ceph-mon[25721]: osdmap e69: 8 total, 8 up, 8 in 2024-04-14T13:56:57.596 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:56:57 smithi122 ceph-mon[25721]: from='client.? 172.21.15.69:0/1956499905' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "z.rgw.meta", "var": "pg_num_min", "val": "8"}]: dispatch 2024-04-14T13:56:57.596 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:56:57 smithi122 ceph-mon[25721]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "z.rgw.meta", "var": "pg_num_min", "val": "8"}]: dispatch 2024-04-14T13:56:58.067 INFO:journalctl@ceph.alertmanager.a.smithi069.stdout:Apr 14 13:56:57 smithi069 bash[49297]: Error: no container with name or ID "ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-alertmanager.a" found: no such container 2024-04-14T13:56:58.067 INFO:journalctl@ceph.alertmanager.a.smithi069.stdout:Apr 14 13:56:57 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-alertmanager-a[47876]: level=info ts=2024-04-14T13:56:57.875Z caller=main.go:557 msg="Received SIGTERM, exiting gracefully..." 2024-04-14T13:56:58.067 INFO:journalctl@ceph.alertmanager.a.smithi069.stdout:Apr 14 13:56:57 smithi069 podman[49310]: 2024-04-14 13:56:57.895591559 +0000 UTC m=+0.120901211 container died 943c4195589632d2acf76a3628f3c417daeb9179c031451eb537585994d25b2a (image=quay.io/prometheus/alertmanager:v0.23.0, name=ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-alertmanager-a, maintainer=The Prometheus Authors ) 2024-04-14T13:56:58.436 INFO:journalctl@ceph.alertmanager.a.smithi069.stdout:Apr 14 13:56:58 smithi069 podman[49310]: 2024-04-14 13:56:58.308324461 +0000 UTC m=+0.533634162 container cleanup 943c4195589632d2acf76a3628f3c417daeb9179c031451eb537585994d25b2a (image=quay.io/prometheus/alertmanager:v0.23.0, name=ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-alertmanager-a, maintainer=The Prometheus Authors ) 2024-04-14T13:56:58.437 INFO:journalctl@ceph.alertmanager.a.smithi069.stdout:Apr 14 13:56:58 smithi069 bash[49310]: ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-alertmanager-a 2024-04-14T13:56:58.437 INFO:journalctl@ceph.alertmanager.a.smithi069.stdout:Apr 14 13:56:58 smithi069 bash[49337]: Error: no container with name or ID "ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-alertmanager.a" found: no such container 2024-04-14T13:56:58.763 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:56:58 smithi069 ceph-mon[29472]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "z.rgw.meta", "var": "pg_num_min", "val": "8"}]': finished 2024-04-14T13:56:58.763 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:56:58 smithi069 ceph-mon[29472]: osdmap e70: 8 total, 8 up, 8 in 2024-04-14T13:56:58.764 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:56:58 smithi069 ceph-mon[24818]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "z.rgw.meta", "var": "pg_num_min", "val": "8"}]': finished 2024-04-14T13:56:58.764 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:56:58 smithi069 ceph-mon[24818]: osdmap e70: 8 total, 8 up, 8 in 2024-04-14T13:56:58.764 INFO:journalctl@ceph.alertmanager.a.smithi069.stdout:Apr 14 13:56:58 smithi069 podman[49324]: 2024-04-14 13:56:58.517768278 +0000 UTC m=+0.622266574 container remove 943c4195589632d2acf76a3628f3c417daeb9179c031451eb537585994d25b2a (image=quay.io/prometheus/alertmanager:v0.23.0, name=ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-alertmanager-a, maintainer=The Prometheus Authors ) 2024-04-14T13:56:58.765 INFO:journalctl@ceph.alertmanager.a.smithi069.stdout:Apr 14 13:56:58 smithi069 podman[49324]: 2024-04-14 13:56:58.618400446 +0000 UTC m=+0.722898752 volume remove 700ba03302c2b608a0440f2af664305d4dca3da66e4e0a99addce7dd5c818ef0 2024-04-14T13:56:58.765 INFO:journalctl@ceph.alertmanager.a.smithi069.stdout:Apr 14 13:56:58 smithi069 systemd[1]: ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968@alertmanager.a.service: Deactivated successfully. 2024-04-14T13:56:58.765 INFO:journalctl@ceph.alertmanager.a.smithi069.stdout:Apr 14 13:56:58 smithi069 systemd[1]: Stopped Ceph alertmanager.a for dc326fc2-fa65-11ee-bc8e-c7b262605968. 2024-04-14T13:56:58.765 INFO:journalctl@ceph.alertmanager.a.smithi069.stdout:Apr 14 13:56:58 smithi069 systemd[1]: Starting Ceph alertmanager.a for dc326fc2-fa65-11ee-bc8e-c7b262605968... 2024-04-14T13:56:58.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:56:58 smithi122 ceph-mon[25721]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "z.rgw.meta", "var": "pg_num_min", "val": "8"}]': finished 2024-04-14T13:56:58.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:56:58 smithi122 ceph-mon[25721]: osdmap e70: 8 total, 8 up, 8 in 2024-04-14T13:56:59.046 DEBUG:teuthology.orchestra.run.smithi069:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid dc326fc2-fa65-11ee-bc8e-c7b262605968 -e sha1=a971fd5f5719038c70b817bafacd6d5400b1d888 -- bash -c 'ceph orch apply rgw foo --realm r --zone z --placement=2 --port=8000' 2024-04-14T13:56:59.317 INFO:journalctl@ceph.alertmanager.a.smithi069.stdout:Apr 14 13:56:59 smithi069 podman[49423]: 2024-04-14 13:56:59.091371051 +0000 UTC m=+0.022193692 image pull ba2b418f427c0636d654de8757e830c80168e76482bcc46bb2138e569d6c91d4 quay.io/prometheus/alertmanager:v0.23.0 2024-04-14T13:56:59.317 INFO:journalctl@ceph.alertmanager.a.smithi069.stdout:Apr 14 13:56:59 smithi069 podman[49423]: 2024-04-14 13:56:59.238621362 +0000 UTC m=+0.169444003 volume create 6f15ef4706adf0e8f9a6fdb954c96e3cbcd62daa0b63ba5bb68d2b187f2f4372 2024-04-14T13:56:59.667 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:56:59 smithi069 ceph-mon[24818]: pgmap v61: 129 pgs: 32 creating+peering, 97 active+clean; 579 KiB data, 51 MiB used, 715 GiB / 715 GiB avail; 255 B/s rd, 511 B/s wr, 1 op/s 2024-04-14T13:56:59.668 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 14 13:56:59 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-y[25054]: ::ffff:172.21.15.122 - - [14/Apr/2024:13:56:59] "GET /metrics HTTP/1.1" 200 195240 "" "Prometheus/2.33.4" 2024-04-14T13:56:59.668 INFO:journalctl@ceph.alertmanager.a.smithi069.stdout:Apr 14 13:56:59 smithi069 podman[49423]: 2024-04-14 13:56:59.322353126 +0000 UTC m=+0.253175804 container create 148fbcb552325f752f07b182e98ec63a0e7d0dd169222d4e66f95febb6fde931 (image=quay.io/prometheus/alertmanager:v0.23.0, name=ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-alertmanager-a, maintainer=The Prometheus Authors ) 2024-04-14T13:56:59.668 INFO:journalctl@ceph.alertmanager.a.smithi069.stdout:Apr 14 13:56:59 smithi069 podman[49423]: 2024-04-14 13:56:59.666776374 +0000 UTC m=+0.597599013 container init 148fbcb552325f752f07b182e98ec63a0e7d0dd169222d4e66f95febb6fde931 (image=quay.io/prometheus/alertmanager:v0.23.0, name=ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-alertmanager-a, maintainer=The Prometheus Authors ) 2024-04-14T13:56:59.668 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:56:59 smithi069 ceph-mon[29472]: pgmap v61: 129 pgs: 32 creating+peering, 97 active+clean; 579 KiB data, 51 MiB used, 715 GiB / 715 GiB avail; 255 B/s rd, 511 B/s wr, 1 op/s 2024-04-14T13:56:59.846 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:56:59 smithi122 ceph-mon[25721]: pgmap v61: 129 pgs: 32 creating+peering, 97 active+clean; 579 KiB data, 51 MiB used, 715 GiB / 715 GiB avail; 255 B/s rd, 511 B/s wr, 1 op/s 2024-04-14T13:56:59.995 INFO:journalctl@ceph.alertmanager.a.smithi069.stdout:Apr 14 13:56:59 smithi069 podman[49423]: 2024-04-14 13:56:59.669464394 +0000 UTC m=+0.600287026 container start 148fbcb552325f752f07b182e98ec63a0e7d0dd169222d4e66f95febb6fde931 (image=quay.io/prometheus/alertmanager:v0.23.0, name=ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-alertmanager-a, maintainer=The Prometheus Authors ) 2024-04-14T13:56:59.995 INFO:journalctl@ceph.alertmanager.a.smithi069.stdout:Apr 14 13:56:59 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-alertmanager-a[49465]: level=info ts=2024-04-14T13:56:59.680Z caller=main.go:225 msg="Starting Alertmanager" version="(version=0.23.0, branch=HEAD, revision=61046b17771a57cfd4c4a51be370ab930a4d7d54)" 2024-04-14T13:56:59.995 INFO:journalctl@ceph.alertmanager.a.smithi069.stdout:Apr 14 13:56:59 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-alertmanager-a[49465]: level=info ts=2024-04-14T13:56:59.680Z caller=main.go:226 build_context="(go=go1.16.7, user=root@e21a959be8d2, date=20210825-10:48:55)" 2024-04-14T13:56:59.996 INFO:journalctl@ceph.alertmanager.a.smithi069.stdout:Apr 14 13:56:59 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-alertmanager-a[49465]: level=info ts=2024-04-14T13:56:59.681Z caller=cluster.go:184 component=cluster msg="setting advertise address explicitly" addr=172.21.15.69 port=9094 2024-04-14T13:56:59.996 INFO:journalctl@ceph.alertmanager.a.smithi069.stdout:Apr 14 13:56:59 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-alertmanager-a[49465]: level=info ts=2024-04-14T13:56:59.682Z caller=cluster.go:671 component=cluster msg="Waiting for gossip to settle..." interval=2s 2024-04-14T13:56:59.996 INFO:journalctl@ceph.alertmanager.a.smithi069.stdout:Apr 14 13:56:59 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-alertmanager-a[49465]: level=info ts=2024-04-14T13:56:59.706Z caller=coordinator.go:113 component=configuration msg="Loading configuration file" file=/etc/alertmanager/alertmanager.yml 2024-04-14T13:56:59.996 INFO:journalctl@ceph.alertmanager.a.smithi069.stdout:Apr 14 13:56:59 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-alertmanager-a[49465]: level=info ts=2024-04-14T13:56:59.706Z caller=coordinator.go:126 component=configuration msg="Completed loading of configuration file" file=/etc/alertmanager/alertmanager.yml 2024-04-14T13:56:59.996 INFO:journalctl@ceph.alertmanager.a.smithi069.stdout:Apr 14 13:56:59 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-alertmanager-a[49465]: level=info ts=2024-04-14T13:56:59.709Z caller=main.go:518 msg=Listening address=:9093 2024-04-14T13:56:59.996 INFO:journalctl@ceph.alertmanager.a.smithi069.stdout:Apr 14 13:56:59 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-alertmanager-a[49465]: level=info ts=2024-04-14T13:56:59.709Z caller=tls_config.go:191 msg="TLS is disabled." http2=false 2024-04-14T13:56:59.996 INFO:journalctl@ceph.alertmanager.a.smithi069.stdout:Apr 14 13:56:59 smithi069 bash[49423]: 148fbcb552325f752f07b182e98ec63a0e7d0dd169222d4e66f95febb6fde931 2024-04-14T13:56:59.996 INFO:journalctl@ceph.alertmanager.a.smithi069.stdout:Apr 14 13:56:59 smithi069 systemd[1]: Started Ceph alertmanager.a for dc326fc2-fa65-11ee-bc8e-c7b262605968. 2024-04-14T13:57:00.817 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:57:00 smithi069 ceph-mon[24818]: from='mgr.14448 172.21.15.69:0/453473650' entity='mgr.y' 2024-04-14T13:57:00.817 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:57:00 smithi069 ceph-mon[24818]: Reconfiguring prometheus.a (dependencies changed)... 2024-04-14T13:57:00.817 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:57:00 smithi069 ceph-mon[24818]: Reconfiguring daemon prometheus.a on smithi122 2024-04-14T13:57:00.817 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:57:00 smithi069 ceph-mon[24818]: pgmap v62: 129 pgs: 21 creating+peering, 108 active+clean; 579 KiB data, 51 MiB used, 715 GiB / 715 GiB avail; 866 B/s rd, 866 B/s wr, 2 op/s 2024-04-14T13:57:00.817 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:57:00 smithi069 ceph-mon[29472]: from='mgr.14448 172.21.15.69:0/453473650' entity='mgr.y' 2024-04-14T13:57:00.818 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:57:00 smithi069 ceph-mon[29472]: Reconfiguring prometheus.a (dependencies changed)... 2024-04-14T13:57:00.818 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:57:00 smithi069 ceph-mon[29472]: Reconfiguring daemon prometheus.a on smithi122 2024-04-14T13:57:00.818 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:57:00 smithi069 ceph-mon[29472]: pgmap v62: 129 pgs: 21 creating+peering, 108 active+clean; 579 KiB data, 51 MiB used, 715 GiB / 715 GiB avail; 866 B/s rd, 866 B/s wr, 2 op/s 2024-04-14T13:57:01.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:57:00 smithi122 ceph-mon[25721]: from='mgr.14448 172.21.15.69:0/453473650' entity='mgr.y' 2024-04-14T13:57:01.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:57:00 smithi122 ceph-mon[25721]: Reconfiguring prometheus.a (dependencies changed)... 2024-04-14T13:57:01.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:57:00 smithi122 ceph-mon[25721]: Reconfiguring daemon prometheus.a on smithi122 2024-04-14T13:57:01.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:57:00 smithi122 ceph-mon[25721]: pgmap v62: 129 pgs: 21 creating+peering, 108 active+clean; 579 KiB data, 51 MiB used, 715 GiB / 715 GiB avail; 866 B/s rd, 866 B/s wr, 2 op/s 2024-04-14T13:57:01.562 INFO:teuthology.orchestra.run.smithi069.stdout:Scheduled rgw.foo update... 2024-04-14T13:57:01.817 INFO:journalctl@ceph.alertmanager.a.smithi069.stdout:Apr 14 13:57:01 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-alertmanager-a[49465]: level=info ts=2024-04-14T13:57:01.682Z caller=cluster.go:696 component=cluster msg="gossip not settled" polls=0 before=0 now=1 elapsed=2.000088559s 2024-04-14T13:57:02.139 DEBUG:teuthology.orchestra.run.smithi069:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid dc326fc2-fa65-11ee-bc8e-c7b262605968 -e sha1=a971fd5f5719038c70b817bafacd6d5400b1d888 -- bash -c 'ceph osd pool create foo' 2024-04-14T13:57:02.733 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:57:02 smithi069 ceph-mon[24818]: from='client.24586 -' entity='client.admin' cmd=[{"prefix": "orch apply rgw", "svc_id": "foo", "realm": "r", "zone": "z", "placement": "2", "port": 8000, "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T13:57:02.733 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:57:02 smithi069 ceph-mon[24818]: Saving service rgw.foo spec with placement count:2 2024-04-14T13:57:02.733 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:57:02 smithi069 ceph-mon[24818]: from='mgr.14448 172.21.15.69:0/453473650' entity='mgr.y' 2024-04-14T13:57:02.734 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:57:02 smithi069 ceph-mon[24818]: pgmap v63: 129 pgs: 129 active+clean; 581 KiB data, 52 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1.0 KiB/s wr, 2 op/s 2024-04-14T13:57:02.734 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:57:02 smithi069 ceph-mon[29472]: from='client.24586 -' entity='client.admin' cmd=[{"prefix": "orch apply rgw", "svc_id": "foo", "realm": "r", "zone": "z", "placement": "2", "port": 8000, "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T13:57:02.734 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:57:02 smithi069 ceph-mon[29472]: Saving service rgw.foo spec with placement count:2 2024-04-14T13:57:02.734 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:57:02 smithi069 ceph-mon[29472]: from='mgr.14448 172.21.15.69:0/453473650' entity='mgr.y' 2024-04-14T13:57:02.734 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:57:02 smithi069 ceph-mon[29472]: pgmap v63: 129 pgs: 129 active+clean; 581 KiB data, 52 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1.0 KiB/s wr, 2 op/s 2024-04-14T13:57:02.828 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:57:02 smithi122 ceph-mon[25721]: from='client.24586 -' entity='client.admin' cmd=[{"prefix": "orch apply rgw", "svc_id": "foo", "realm": "r", "zone": "z", "placement": "2", "port": 8000, "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T13:57:02.828 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:57:02 smithi122 ceph-mon[25721]: Saving service rgw.foo spec with placement count:2 2024-04-14T13:57:02.828 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:57:02 smithi122 ceph-mon[25721]: from='mgr.14448 172.21.15.69:0/453473650' entity='mgr.y' 2024-04-14T13:57:02.828 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:57:02 smithi122 ceph-mon[25721]: pgmap v63: 129 pgs: 129 active+clean; 581 KiB data, 52 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1.0 KiB/s wr, 2 op/s 2024-04-14T13:57:03.095 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Apr 14 13:57:02 smithi122 systemd[1]: Stopping Ceph prometheus.a for dc326fc2-fa65-11ee-bc8e-c7b262605968... 2024-04-14T13:57:03.095 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Apr 14 13:57:02 smithi122 bash[42537]: Error: no container with name or ID "ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-prometheus.a" found: no such container 2024-04-14T13:57:03.451 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Apr 14 13:57:03 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-prometheus-a[40879]: ts=2024-04-14T13:57:03.123Z caller=main.go:775 level=warn msg="Received SIGTERM, exiting gracefully..." 2024-04-14T13:57:03.451 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Apr 14 13:57:03 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-prometheus-a[40879]: ts=2024-04-14T13:57:03.123Z caller=main.go:798 level=info msg="Stopping scrape discovery manager..." 2024-04-14T13:57:03.451 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Apr 14 13:57:03 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-prometheus-a[40879]: ts=2024-04-14T13:57:03.123Z caller=main.go:812 level=info msg="Stopping notify discovery manager..." 2024-04-14T13:57:03.451 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Apr 14 13:57:03 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-prometheus-a[40879]: ts=2024-04-14T13:57:03.123Z caller=main.go:834 level=info msg="Stopping scrape manager..." 2024-04-14T13:57:03.451 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Apr 14 13:57:03 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-prometheus-a[40879]: ts=2024-04-14T13:57:03.123Z caller=main.go:808 level=info msg="Notify discovery manager stopped" 2024-04-14T13:57:03.451 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Apr 14 13:57:03 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-prometheus-a[40879]: ts=2024-04-14T13:57:03.123Z caller=main.go:794 level=info msg="Scrape discovery manager stopped" 2024-04-14T13:57:03.451 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Apr 14 13:57:03 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-prometheus-a[40879]: ts=2024-04-14T13:57:03.124Z caller=manager.go:945 level=info component="rule manager" msg="Stopping rule manager..." 2024-04-14T13:57:03.451 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Apr 14 13:57:03 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-prometheus-a[40879]: ts=2024-04-14T13:57:03.124Z caller=manager.go:955 level=info component="rule manager" msg="Rule manager stopped" 2024-04-14T13:57:03.452 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Apr 14 13:57:03 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-prometheus-a[40879]: ts=2024-04-14T13:57:03.124Z caller=main.go:828 level=info msg="Scrape manager stopped" 2024-04-14T13:57:03.452 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Apr 14 13:57:03 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-prometheus-a[40879]: ts=2024-04-14T13:57:03.124Z caller=notifier.go:600 level=info component=notifier msg="Stopping notification manager..." 2024-04-14T13:57:03.452 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Apr 14 13:57:03 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-prometheus-a[40879]: ts=2024-04-14T13:57:03.124Z caller=main.go:1054 level=info msg="Notifier manager stopped" 2024-04-14T13:57:03.452 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Apr 14 13:57:03 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-prometheus-a[40879]: ts=2024-04-14T13:57:03.124Z caller=main.go:1066 level=info msg="See you next time!" 2024-04-14T13:57:03.452 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Apr 14 13:57:03 smithi122 podman[42550]: 2024-04-14 13:57:03.133869742 +0000 UTC m=+0.135824778 container died 18895e3c9f451c7e9d8116de15c26d864460545ae2cc8defadfdcf28fdd1a894 (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-prometheus-a, maintainer=The Prometheus Authors ) 2024-04-14T13:57:03.845 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Apr 14 13:57:03 smithi122 podman[42550]: 2024-04-14 13:57:03.608851926 +0000 UTC m=+0.610807004 container cleanup 18895e3c9f451c7e9d8116de15c26d864460545ae2cc8defadfdcf28fdd1a894 (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-prometheus-a, maintainer=The Prometheus Authors ) 2024-04-14T13:57:03.845 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Apr 14 13:57:03 smithi122 bash[42550]: ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-prometheus-a 2024-04-14T13:57:03.845 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Apr 14 13:57:03 smithi122 bash[42576]: Error: no container with name or ID "ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-prometheus.a" found: no such container 2024-04-14T13:57:04.190 INFO:teuthology.orchestra.run.smithi069.stderr:pool 'foo' created 2024-04-14T13:57:04.483 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:57:04 smithi069 ceph-mon[24818]: from='client.? 172.21.15.69:0/3031783403' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "foo"}]: dispatch 2024-04-14T13:57:04.484 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:57:04 smithi069 ceph-mon[29472]: from='client.? 172.21.15.69:0/3031783403' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "foo"}]: dispatch 2024-04-14T13:57:04.585 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:57:04 smithi122 ceph-mon[25721]: from='client.? 172.21.15.69:0/3031783403' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "foo"}]: dispatch 2024-04-14T13:57:04.765 DEBUG:teuthology.orchestra.run.smithi069:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid dc326fc2-fa65-11ee-bc8e-c7b262605968 -e sha1=a971fd5f5719038c70b817bafacd6d5400b1d888 -- bash -c 'rbd pool init foo' 2024-04-14T13:57:04.845 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Apr 14 13:57:04 smithi122 podman[42564]: 2024-04-14 13:57:04.584258465 +0000 UTC m=+1.447777627 container remove 18895e3c9f451c7e9d8116de15c26d864460545ae2cc8defadfdcf28fdd1a894 (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-prometheus-a, maintainer=The Prometheus Authors ) 2024-04-14T13:57:04.845 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Apr 14 13:57:04 smithi122 systemd[1]: ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968@prometheus.a.service: Deactivated successfully. 2024-04-14T13:57:04.845 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Apr 14 13:57:04 smithi122 systemd[1]: Stopped Ceph prometheus.a for dc326fc2-fa65-11ee-bc8e-c7b262605968. 2024-04-14T13:57:04.845 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Apr 14 13:57:04 smithi122 systemd[1]: Starting Ceph prometheus.a for dc326fc2-fa65-11ee-bc8e-c7b262605968... 2024-04-14T13:57:04.845 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Apr 14 13:57:04 smithi122 podman[42645]: 2024-04-14 13:57:04.730743113 +0000 UTC m=+0.019108564 image pull 514e6a882f6e74806a5856468489eeff8d7106095557578da96935e4d0ba4d9d quay.io/prometheus/prometheus:v2.33.4 2024-04-14T13:57:05.317 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:57:05 smithi069 ceph-mon[24818]: from='client.? 172.21.15.69:0/3031783403' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "foo"}]': finished 2024-04-14T13:57:05.685 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:57:05 smithi069 ceph-mon[24818]: osdmap e71: 8 total, 8 up, 8 in 2024-04-14T13:57:05.685 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:57:05 smithi069 ceph-mon[24818]: pgmap v65: 161 pgs: 32 unknown, 129 active+clean; 581 KiB data, 52 MiB used, 715 GiB / 715 GiB avail; 905 B/s rd, 905 B/s wr, 2 op/s 2024-04-14T13:57:05.686 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:57:05 smithi069 ceph-mon[29472]: from='client.? 172.21.15.69:0/3031783403' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "foo"}]': finished 2024-04-14T13:57:05.687 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:57:05 smithi069 ceph-mon[29472]: osdmap e71: 8 total, 8 up, 8 in 2024-04-14T13:57:05.687 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:57:05 smithi069 ceph-mon[29472]: pgmap v65: 161 pgs: 32 unknown, 129 active+clean; 581 KiB data, 52 MiB used, 715 GiB / 715 GiB avail; 905 B/s rd, 905 B/s wr, 2 op/s 2024-04-14T13:57:05.689 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:57:05 smithi122 ceph-mon[25721]: from='client.? 172.21.15.69:0/3031783403' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "foo"}]': finished 2024-04-14T13:57:05.689 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:57:05 smithi122 ceph-mon[25721]: osdmap e71: 8 total, 8 up, 8 in 2024-04-14T13:57:05.689 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:57:05 smithi122 ceph-mon[25721]: pgmap v65: 161 pgs: 32 unknown, 129 active+clean; 581 KiB data, 52 MiB used, 715 GiB / 715 GiB avail; 905 B/s rd, 905 B/s wr, 2 op/s 2024-04-14T13:57:06.095 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Apr 14 13:57:05 smithi122 podman[42645]: 2024-04-14 13:57:05.794103219 +0000 UTC m=+1.082468652 container create 59e61af03fdf3105d92fb2c73df6a42a6ef10a5eddbf11d7ef945aac315c06eb (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-prometheus-a, maintainer=The Prometheus Authors ) 2024-04-14T13:57:06.567 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:57:06 smithi069 ceph-mon[24818]: osdmap e72: 8 total, 8 up, 8 in 2024-04-14T13:57:06.567 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:57:06 smithi069 ceph-mon[24818]: osdmap e73: 8 total, 8 up, 8 in 2024-04-14T13:57:06.567 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:57:06 smithi069 ceph-mon[29472]: osdmap e72: 8 total, 8 up, 8 in 2024-04-14T13:57:06.567 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:57:06 smithi069 ceph-mon[29472]: osdmap e73: 8 total, 8 up, 8 in 2024-04-14T13:57:06.595 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:57:06 smithi122 ceph-mon[25721]: osdmap e72: 8 total, 8 up, 8 in 2024-04-14T13:57:06.595 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:57:06 smithi122 ceph-mon[25721]: osdmap e73: 8 total, 8 up, 8 in 2024-04-14T13:57:07.570 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:57:07 smithi069 ceph-mon[24818]: from='client.? 172.21.15.69:0/2842788180' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "foo","app": "rbd"}]: dispatch 2024-04-14T13:57:07.570 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:57:07 smithi069 ceph-mon[24818]: pgmap v68: 161 pgs: 10 creating+activating, 151 active+clean; 581 KiB data, 52 MiB used, 715 GiB / 715 GiB avail; 511 B/s rd, 682 B/s wr, 1 op/s 2024-04-14T13:57:07.570 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:57:07 smithi069 ceph-mon[29472]: from='client.? 172.21.15.69:0/2842788180' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "foo","app": "rbd"}]: dispatch 2024-04-14T13:57:07.571 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:57:07 smithi069 ceph-mon[29472]: pgmap v68: 161 pgs: 10 creating+activating, 151 active+clean; 581 KiB data, 52 MiB used, 715 GiB / 715 GiB avail; 511 B/s rd, 682 B/s wr, 1 op/s 2024-04-14T13:57:07.595 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:57:07 smithi122 ceph-mon[25721]: from='client.? 172.21.15.69:0/2842788180' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "foo","app": "rbd"}]: dispatch 2024-04-14T13:57:07.595 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:57:07 smithi122 ceph-mon[25721]: pgmap v68: 161 pgs: 10 creating+activating, 151 active+clean; 581 KiB data, 52 MiB used, 715 GiB / 715 GiB avail; 511 B/s rd, 682 B/s wr, 1 op/s 2024-04-14T13:57:07.937 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Apr 14 13:57:07 smithi122 podman[42645]: 2024-04-14 13:57:07.678932016 +0000 UTC m=+2.967297460 container init 59e61af03fdf3105d92fb2c73df6a42a6ef10a5eddbf11d7ef945aac315c06eb (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-prometheus-a, maintainer=The Prometheus Authors ) 2024-04-14T13:57:07.937 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Apr 14 13:57:07 smithi122 podman[42645]: 2024-04-14 13:57:07.682884084 +0000 UTC m=+2.971249521 container start 59e61af03fdf3105d92fb2c73df6a42a6ef10a5eddbf11d7ef945aac315c06eb (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-prometheus-a, maintainer=The Prometheus Authors ) 2024-04-14T13:57:07.938 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Apr 14 13:57:07 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-prometheus-a[42660]: ts=2024-04-14T13:57:07.710Z caller=main.go:475 level=info msg="No time or size retention was set so using the default time retention" duration=15d 2024-04-14T13:57:07.938 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Apr 14 13:57:07 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-prometheus-a[42660]: ts=2024-04-14T13:57:07.711Z caller=main.go:512 level=info msg="Starting Prometheus" version="(version=2.33.4, branch=HEAD, revision=83032011a5d3e6102624fe58241a374a7201fee8)" 2024-04-14T13:57:07.938 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Apr 14 13:57:07 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-prometheus-a[42660]: ts=2024-04-14T13:57:07.711Z caller=main.go:517 level=info build_context="(go=go1.17.7, user=root@d13bf69e7be8, date=20220222-16:51:28)" 2024-04-14T13:57:07.938 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Apr 14 13:57:07 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-prometheus-a[42660]: ts=2024-04-14T13:57:07.711Z caller=main.go:518 level=info host_details="(Linux 5.14.0-435.el9.x86_64 #1 SMP PREEMPT_DYNAMIC Fri Mar 29 18:46:37 UTC 2024 x86_64 smithi122 (none))" 2024-04-14T13:57:07.938 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Apr 14 13:57:07 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-prometheus-a[42660]: ts=2024-04-14T13:57:07.711Z caller=main.go:519 level=info fd_limits="(soft=1048576, hard=1048576)" 2024-04-14T13:57:07.938 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Apr 14 13:57:07 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-prometheus-a[42660]: ts=2024-04-14T13:57:07.711Z caller=main.go:520 level=info vm_limits="(soft=unlimited, hard=unlimited)" 2024-04-14T13:57:07.938 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Apr 14 13:57:07 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-prometheus-a[42660]: ts=2024-04-14T13:57:07.713Z caller=web.go:570 level=info component=web msg="Start listening for connections" address=:9095 2024-04-14T13:57:07.938 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Apr 14 13:57:07 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-prometheus-a[42660]: ts=2024-04-14T13:57:07.713Z caller=main.go:923 level=info msg="Starting TSDB ..." 2024-04-14T13:57:07.938 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Apr 14 13:57:07 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-prometheus-a[42660]: ts=2024-04-14T13:57:07.714Z caller=tls_config.go:195 level=info component=web msg="TLS is disabled." http2=false 2024-04-14T13:57:07.938 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Apr 14 13:57:07 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-prometheus-a[42660]: ts=2024-04-14T13:57:07.717Z caller=head.go:493 level=info component=tsdb msg="Replaying on-disk memory mappable chunks if any" 2024-04-14T13:57:07.938 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Apr 14 13:57:07 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-prometheus-a[42660]: ts=2024-04-14T13:57:07.717Z caller=head.go:527 level=info component=tsdb msg="On-disk memory mappable chunks replay completed" duration=2.922µs 2024-04-14T13:57:07.938 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Apr 14 13:57:07 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-prometheus-a[42660]: ts=2024-04-14T13:57:07.717Z caller=head.go:533 level=info component=tsdb msg="Replaying WAL, this may take a while" 2024-04-14T13:57:07.938 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Apr 14 13:57:07 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-prometheus-a[42660]: ts=2024-04-14T13:57:07.936Z caller=head.go:604 level=info component=tsdb msg="WAL segment loaded" segment=0 maxSegment=1 2024-04-14T13:57:07.939 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Apr 14 13:57:07 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-prometheus-a[42660]: ts=2024-04-14T13:57:07.936Z caller=head.go:604 level=info component=tsdb msg="WAL segment loaded" segment=1 maxSegment=1 2024-04-14T13:57:07.939 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Apr 14 13:57:07 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-prometheus-a[42660]: ts=2024-04-14T13:57:07.936Z caller=head.go:610 level=info component=tsdb msg="WAL replay completed" checkpoint_replay_duration=43.433µs wal_replay_duration=218.436306ms total_replay_duration=218.516257ms 2024-04-14T13:57:08.223 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Apr 14 13:57:07 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-prometheus-a[42660]: ts=2024-04-14T13:57:07.938Z caller=main.go:944 level=info fs_type=XFS_SUPER_MAGIC 2024-04-14T13:57:08.224 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Apr 14 13:57:07 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-prometheus-a[42660]: ts=2024-04-14T13:57:07.938Z caller=main.go:947 level=info msg="TSDB started" 2024-04-14T13:57:08.224 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Apr 14 13:57:07 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-prometheus-a[42660]: ts=2024-04-14T13:57:07.938Z caller=main.go:1128 level=info msg="Loading configuration file" filename=/etc/prometheus/prometheus.yml 2024-04-14T13:57:08.224 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Apr 14 13:57:07 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-prometheus-a[42660]: ts=2024-04-14T13:57:07.956Z caller=main.go:1165 level=info msg="Completed loading of configuration file" filename=/etc/prometheus/prometheus.yml totalDuration=18.065896ms db_storage=901ns remote_storage=1.362µs web_handler=410ns query_engine=817ns scrape=420.916µs scrape_sd=28.19µs notify=25.714µs notify_sd=7.439µs rules=17.31018ms 2024-04-14T13:57:08.224 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Apr 14 13:57:07 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-prometheus-a[42660]: ts=2024-04-14T13:57:07.956Z caller=main.go:896 level=info msg="Server is ready to receive web requests." 2024-04-14T13:57:08.544 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:57:08 smithi122 ceph-mon[25721]: from='client.? 172.21.15.69:0/2842788180' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "foo","app": "rbd"}]': finished 2024-04-14T13:57:08.544 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:57:08 smithi122 ceph-mon[25721]: osdmap e74: 8 total, 8 up, 8 in 2024-04-14T13:57:08.567 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:57:08 smithi069 ceph-mon[29472]: from='client.? 172.21.15.69:0/2842788180' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "foo","app": "rbd"}]': finished 2024-04-14T13:57:08.567 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:57:08 smithi069 ceph-mon[29472]: osdmap e74: 8 total, 8 up, 8 in 2024-04-14T13:57:08.567 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:57:08 smithi069 ceph-mon[24818]: from='client.? 172.21.15.69:0/2842788180' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "foo","app": "rbd"}]': finished 2024-04-14T13:57:08.567 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:57:08 smithi069 ceph-mon[24818]: osdmap e74: 8 total, 8 up, 8 in 2024-04-14T13:57:08.845 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Apr 14 13:57:08 smithi122 bash[42645]: 59e61af03fdf3105d92fb2c73df6a42a6ef10a5eddbf11d7ef945aac315c06eb 2024-04-14T13:57:08.845 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Apr 14 13:57:08 smithi122 systemd[1]: Started Ceph prometheus.a for dc326fc2-fa65-11ee-bc8e-c7b262605968. 2024-04-14T13:57:09.567 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:57:09 smithi069 ceph-mon[24818]: osdmap e75: 8 total, 8 up, 8 in 2024-04-14T13:57:09.568 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:57:09 smithi069 ceph-mon[24818]: pgmap v71: 161 pgs: 10 creating+activating, 151 active+clean; 581 KiB data, 52 MiB used, 715 GiB / 715 GiB avail 2024-04-14T13:57:09.568 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:57:09 smithi069 ceph-mon[24818]: from='mgr.14448 172.21.15.69:0/453473650' entity='mgr.y' 2024-04-14T13:57:09.568 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:57:09 smithi069 ceph-mon[24818]: from='mgr.14448 172.21.15.69:0/453473650' entity='mgr.y' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-04-14T13:57:09.568 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:57:09 smithi069 ceph-mon[24818]: from='mgr.14448 172.21.15.69:0/453473650' entity='mgr.y' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://172.21.15.69:9093"}]: dispatch 2024-04-14T13:57:09.568 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:57:09 smithi069 ceph-mon[24818]: from='mgr.14448 172.21.15.69:0/453473650' entity='mgr.y' 2024-04-14T13:57:09.568 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:57:09 smithi069 ceph-mon[24818]: from='mgr.14448 172.21.15.69:0/453473650' entity='mgr.y' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-04-14T13:57:09.568 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:57:09 smithi069 ceph-mon[24818]: from='mgr.14448 172.21.15.69:0/453473650' entity='mgr.y' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://172.21.15.122:9095"}]: dispatch 2024-04-14T13:57:09.568 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:57:09 smithi069 ceph-mon[24818]: from='mgr.14448 172.21.15.69:0/453473650' entity='mgr.y' 2024-04-14T13:57:09.568 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:57:09 smithi069 ceph-mon[24818]: from='mgr.14448 172.21.15.69:0/453473650' entity='mgr.y' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-04-14T13:57:09.568 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:57:09 smithi069 ceph-mon[24818]: from='mgr.14448 172.21.15.69:0/453473650' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://172.21.15.122:3000"}]: dispatch 2024-04-14T13:57:09.568 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:57:09 smithi069 ceph-mon[24818]: from='mgr.14448 172.21.15.69:0/453473650' entity='mgr.y' 2024-04-14T13:57:09.568 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:57:09 smithi069 ceph-mon[24818]: from='mgr.14448 172.21.15.69:0/453473650' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T13:57:09.568 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:57:09 smithi069 ceph-mon[24818]: from='mgr.14448 172.21.15.69:0/453473650' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-14T13:57:09.568 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:57:09 smithi069 ceph-mon[24818]: osdmap e76: 8 total, 8 up, 8 in 2024-04-14T13:57:09.569 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:57:09 smithi069 ceph-mon[29472]: osdmap e75: 8 total, 8 up, 8 in 2024-04-14T13:57:09.569 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:57:09 smithi069 ceph-mon[29472]: pgmap v71: 161 pgs: 10 creating+activating, 151 active+clean; 581 KiB data, 52 MiB used, 715 GiB / 715 GiB avail 2024-04-14T13:57:09.569 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:57:09 smithi069 ceph-mon[29472]: from='mgr.14448 172.21.15.69:0/453473650' entity='mgr.y' 2024-04-14T13:57:09.569 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:57:09 smithi069 ceph-mon[29472]: from='mgr.14448 172.21.15.69:0/453473650' entity='mgr.y' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-04-14T13:57:09.569 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:57:09 smithi069 ceph-mon[29472]: from='mgr.14448 172.21.15.69:0/453473650' entity='mgr.y' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://172.21.15.69:9093"}]: dispatch 2024-04-14T13:57:09.569 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:57:09 smithi069 ceph-mon[29472]: from='mgr.14448 172.21.15.69:0/453473650' entity='mgr.y' 2024-04-14T13:57:09.569 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:57:09 smithi069 ceph-mon[29472]: from='mgr.14448 172.21.15.69:0/453473650' entity='mgr.y' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-04-14T13:57:09.569 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:57:09 smithi069 ceph-mon[29472]: from='mgr.14448 172.21.15.69:0/453473650' entity='mgr.y' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://172.21.15.122:9095"}]: dispatch 2024-04-14T13:57:09.569 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:57:09 smithi069 ceph-mon[29472]: from='mgr.14448 172.21.15.69:0/453473650' entity='mgr.y' 2024-04-14T13:57:09.569 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:57:09 smithi069 ceph-mon[29472]: from='mgr.14448 172.21.15.69:0/453473650' entity='mgr.y' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-04-14T13:57:09.569 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:57:09 smithi069 ceph-mon[29472]: from='mgr.14448 172.21.15.69:0/453473650' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://172.21.15.122:3000"}]: dispatch 2024-04-14T13:57:09.569 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:57:09 smithi069 ceph-mon[29472]: from='mgr.14448 172.21.15.69:0/453473650' entity='mgr.y' 2024-04-14T13:57:09.570 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:57:09 smithi069 ceph-mon[29472]: from='mgr.14448 172.21.15.69:0/453473650' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T13:57:09.570 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:57:09 smithi069 ceph-mon[29472]: from='mgr.14448 172.21.15.69:0/453473650' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-14T13:57:09.570 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:57:09 smithi069 ceph-mon[29472]: osdmap e76: 8 total, 8 up, 8 in 2024-04-14T13:57:09.595 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:57:09 smithi122 ceph-mon[25721]: osdmap e75: 8 total, 8 up, 8 in 2024-04-14T13:57:09.595 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:57:09 smithi122 ceph-mon[25721]: pgmap v71: 161 pgs: 10 creating+activating, 151 active+clean; 581 KiB data, 52 MiB used, 715 GiB / 715 GiB avail 2024-04-14T13:57:09.595 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:57:09 smithi122 ceph-mon[25721]: from='mgr.14448 172.21.15.69:0/453473650' entity='mgr.y' 2024-04-14T13:57:09.595 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:57:09 smithi122 ceph-mon[25721]: from='mgr.14448 172.21.15.69:0/453473650' entity='mgr.y' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-04-14T13:57:09.595 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:57:09 smithi122 ceph-mon[25721]: from='mgr.14448 172.21.15.69:0/453473650' entity='mgr.y' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://172.21.15.69:9093"}]: dispatch 2024-04-14T13:57:09.595 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:57:09 smithi122 ceph-mon[25721]: from='mgr.14448 172.21.15.69:0/453473650' entity='mgr.y' 2024-04-14T13:57:09.596 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:57:09 smithi122 ceph-mon[25721]: from='mgr.14448 172.21.15.69:0/453473650' entity='mgr.y' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-04-14T13:57:09.596 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:57:09 smithi122 ceph-mon[25721]: from='mgr.14448 172.21.15.69:0/453473650' entity='mgr.y' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://172.21.15.122:9095"}]: dispatch 2024-04-14T13:57:09.596 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:57:09 smithi122 ceph-mon[25721]: from='mgr.14448 172.21.15.69:0/453473650' entity='mgr.y' 2024-04-14T13:57:09.596 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:57:09 smithi122 ceph-mon[25721]: from='mgr.14448 172.21.15.69:0/453473650' entity='mgr.y' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-04-14T13:57:09.596 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:57:09 smithi122 ceph-mon[25721]: from='mgr.14448 172.21.15.69:0/453473650' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://172.21.15.122:3000"}]: dispatch 2024-04-14T13:57:09.596 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:57:09 smithi122 ceph-mon[25721]: from='mgr.14448 172.21.15.69:0/453473650' entity='mgr.y' 2024-04-14T13:57:09.596 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:57:09 smithi122 ceph-mon[25721]: from='mgr.14448 172.21.15.69:0/453473650' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T13:57:09.596 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:57:09 smithi122 ceph-mon[25721]: from='mgr.14448 172.21.15.69:0/453473650' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-14T13:57:09.596 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:57:09 smithi122 ceph-mon[25721]: osdmap e76: 8 total, 8 up, 8 in 2024-04-14T13:57:09.941 INFO:journalctl@ceph.alertmanager.a.smithi069.stdout:Apr 14 13:57:09 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-alertmanager-a[49465]: level=info ts=2024-04-14T13:57:09.683Z caller=cluster.go:688 component=cluster msg="gossip settled; proceeding" elapsed=10.001121963s 2024-04-14T13:57:09.965 DEBUG:teuthology.orchestra.run.smithi069:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid dc326fc2-fa65-11ee-bc8e-c7b262605968 -e sha1=a971fd5f5719038c70b817bafacd6d5400b1d888 -- bash -c 'ceph orch apply iscsi foo u p' 2024-04-14T13:57:10.484 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:57:10 smithi069 ceph-mon[29472]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-04-14T13:57:10.484 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:57:10 smithi069 ceph-mon[29472]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://172.21.15.69:9093"}]: dispatch 2024-04-14T13:57:10.484 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:57:10 smithi069 ceph-mon[29472]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-04-14T13:57:10.484 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:57:10 smithi069 ceph-mon[29472]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://172.21.15.122:9095"}]: dispatch 2024-04-14T13:57:10.484 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:57:10 smithi069 ceph-mon[29472]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-04-14T13:57:10.484 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:57:10 smithi069 ceph-mon[29472]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://172.21.15.122:3000"}]: dispatch 2024-04-14T13:57:10.484 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:57:10 smithi069 ceph-mon[24818]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-04-14T13:57:10.484 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:57:10 smithi069 ceph-mon[24818]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://172.21.15.69:9093"}]: dispatch 2024-04-14T13:57:10.484 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:57:10 smithi069 ceph-mon[24818]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-04-14T13:57:10.484 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:57:10 smithi069 ceph-mon[24818]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://172.21.15.122:9095"}]: dispatch 2024-04-14T13:57:10.485 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:57:10 smithi069 ceph-mon[24818]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-04-14T13:57:10.485 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:57:10 smithi069 ceph-mon[24818]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://172.21.15.122:3000"}]: dispatch 2024-04-14T13:57:10.595 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:57:10 smithi122 ceph-mon[25721]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-04-14T13:57:10.595 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:57:10 smithi122 ceph-mon[25721]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://172.21.15.69:9093"}]: dispatch 2024-04-14T13:57:10.595 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:57:10 smithi122 ceph-mon[25721]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-04-14T13:57:10.595 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:57:10 smithi122 ceph-mon[25721]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://172.21.15.122:9095"}]: dispatch 2024-04-14T13:57:10.595 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:57:10 smithi122 ceph-mon[25721]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-04-14T13:57:10.595 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:57:10 smithi122 ceph-mon[25721]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://172.21.15.122:3000"}]: dispatch 2024-04-14T13:57:11.567 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:57:11 smithi069 ceph-mon[24818]: pgmap v73: 161 pgs: 5 creating+activating, 156 active+clean; 581 KiB data, 52 MiB used, 715 GiB / 715 GiB avail 2024-04-14T13:57:11.567 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:57:11 smithi069 ceph-mon[29472]: pgmap v73: 161 pgs: 5 creating+activating, 156 active+clean; 581 KiB data, 52 MiB used, 715 GiB / 715 GiB avail 2024-04-14T13:57:11.595 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:57:11 smithi122 ceph-mon[25721]: pgmap v73: 161 pgs: 5 creating+activating, 156 active+clean; 581 KiB data, 52 MiB used, 715 GiB / 715 GiB avail 2024-04-14T13:57:12.101 INFO:teuthology.orchestra.run.smithi069.stdout:Scheduled iscsi.foo update... 2024-04-14T13:57:13.299 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:57:13 smithi069 ceph-mon[24818]: from='client.24604 -' entity='client.admin' cmd=[{"prefix": "orch apply iscsi", "pool": "foo", "api_user": "u", "api_password": "p", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T13:57:13.300 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:57:13 smithi069 ceph-mon[24818]: Saving service iscsi.foo spec with placement count:1 2024-04-14T13:57:13.300 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:57:13 smithi069 ceph-mon[24818]: from='mgr.14448 172.21.15.69:0/453473650' entity='mgr.y' 2024-04-14T13:57:13.300 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:57:13 smithi069 ceph-mon[24818]: pgmap v74: 161 pgs: 161 active+clean; 581 KiB data, 53 MiB used, 715 GiB / 715 GiB avail; 341 B/s wr, 0 op/s 2024-04-14T13:57:13.300 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:57:13 smithi069 ceph-mon[24818]: from='mgr.14448 172.21.15.69:0/453473650' entity='mgr.y' 2024-04-14T13:57:13.300 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:57:13 smithi069 ceph-mon[29472]: from='client.24604 -' entity='client.admin' cmd=[{"prefix": "orch apply iscsi", "pool": "foo", "api_user": "u", "api_password": "p", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T13:57:13.300 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:57:13 smithi069 ceph-mon[29472]: Saving service iscsi.foo spec with placement count:1 2024-04-14T13:57:13.300 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:57:13 smithi069 ceph-mon[29472]: from='mgr.14448 172.21.15.69:0/453473650' entity='mgr.y' 2024-04-14T13:57:13.300 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:57:13 smithi069 ceph-mon[29472]: pgmap v74: 161 pgs: 161 active+clean; 581 KiB data, 53 MiB used, 715 GiB / 715 GiB avail; 341 B/s wr, 0 op/s 2024-04-14T13:57:13.300 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:57:13 smithi069 ceph-mon[29472]: from='mgr.14448 172.21.15.69:0/453473650' entity='mgr.y' 2024-04-14T13:57:13.324 DEBUG:teuthology.orchestra.run.smithi069:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid dc326fc2-fa65-11ee-bc8e-c7b262605968 -e sha1=a971fd5f5719038c70b817bafacd6d5400b1d888 -- bash -c 'sleep 180' 2024-04-14T13:57:13.345 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:57:13 smithi122 ceph-mon[25721]: from='client.24604 -' entity='client.admin' cmd=[{"prefix": "orch apply iscsi", "pool": "foo", "api_user": "u", "api_password": "p", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T13:57:13.345 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:57:13 smithi122 ceph-mon[25721]: Saving service iscsi.foo spec with placement count:1 2024-04-14T13:57:13.345 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:57:13 smithi122 ceph-mon[25721]: from='mgr.14448 172.21.15.69:0/453473650' entity='mgr.y' 2024-04-14T13:57:13.345 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:57:13 smithi122 ceph-mon[25721]: pgmap v74: 161 pgs: 161 active+clean; 581 KiB data, 53 MiB used, 715 GiB / 715 GiB avail; 341 B/s wr, 0 op/s 2024-04-14T13:57:13.345 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:57:13 smithi122 ceph-mon[25721]: from='mgr.14448 172.21.15.69:0/453473650' entity='mgr.y' 2024-04-14T13:57:15.327 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:57:15 smithi069 ceph-mon[24818]: from='mgr.14448 172.21.15.69:0/453473650' entity='mgr.y' 2024-04-14T13:57:15.327 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:57:15 smithi069 ceph-mon[24818]: from='mgr.14448 172.21.15.69:0/453473650' entity='mgr.y' 2024-04-14T13:57:15.327 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:57:15 smithi069 ceph-mon[24818]: from='mgr.14448 172.21.15.69:0/453473650' entity='mgr.y' 2024-04-14T13:57:15.327 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:57:15 smithi069 ceph-mon[24818]: from='mgr.14448 172.21.15.69:0/453473650' entity='mgr.y' 2024-04-14T13:57:15.327 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:57:15 smithi069 ceph-mon[24818]: Saving service rgw.foo spec with placement count:2 2024-04-14T13:57:15.327 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:57:15 smithi069 ceph-mon[24818]: from='mgr.14448 172.21.15.69:0/453473650' entity='mgr.y' 2024-04-14T13:57:15.327 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:57:15 smithi069 ceph-mon[24818]: from='mgr.14448 172.21.15.69:0/453473650' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi069.wesedz", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-04-14T13:57:15.327 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:57:15 smithi069 ceph-mon[24818]: from='mgr.14448 172.21.15.69:0/453473650' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi069.wesedz", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]': finished 2024-04-14T13:57:15.328 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:57:15 smithi069 ceph-mon[24818]: from='mgr.14448 172.21.15.69:0/453473650' entity='mgr.y' 2024-04-14T13:57:15.328 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:57:15 smithi069 ceph-mon[24818]: from='mgr.14448 172.21.15.69:0/453473650' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T13:57:15.328 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:57:15 smithi069 ceph-mon[24818]: Deploying daemon rgw.foo.smithi069.wesedz on smithi069 2024-04-14T13:57:15.328 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:57:15 smithi069 ceph-mon[24818]: pgmap v75: 161 pgs: 161 active+clean; 581 KiB data, 53 MiB used, 715 GiB / 715 GiB avail; 287 B/s wr, 0 op/s 2024-04-14T13:57:15.328 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:57:15 smithi069 ceph-mon[29472]: from='mgr.14448 172.21.15.69:0/453473650' entity='mgr.y' 2024-04-14T13:57:15.328 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:57:15 smithi069 ceph-mon[29472]: from='mgr.14448 172.21.15.69:0/453473650' entity='mgr.y' 2024-04-14T13:57:15.328 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:57:15 smithi069 ceph-mon[29472]: from='mgr.14448 172.21.15.69:0/453473650' entity='mgr.y' 2024-04-14T13:57:15.328 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:57:15 smithi069 ceph-mon[29472]: from='mgr.14448 172.21.15.69:0/453473650' entity='mgr.y' 2024-04-14T13:57:15.328 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:57:15 smithi069 ceph-mon[29472]: Saving service rgw.foo spec with placement count:2 2024-04-14T13:57:15.328 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:57:15 smithi069 ceph-mon[29472]: from='mgr.14448 172.21.15.69:0/453473650' entity='mgr.y' 2024-04-14T13:57:15.328 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:57:15 smithi069 ceph-mon[29472]: from='mgr.14448 172.21.15.69:0/453473650' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi069.wesedz", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-04-14T13:57:15.329 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:57:15 smithi069 ceph-mon[29472]: from='mgr.14448 172.21.15.69:0/453473650' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi069.wesedz", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]': finished 2024-04-14T13:57:15.329 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:57:15 smithi069 ceph-mon[29472]: from='mgr.14448 172.21.15.69:0/453473650' entity='mgr.y' 2024-04-14T13:57:15.329 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:57:15 smithi069 ceph-mon[29472]: from='mgr.14448 172.21.15.69:0/453473650' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T13:57:15.329 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:57:15 smithi069 ceph-mon[29472]: Deploying daemon rgw.foo.smithi069.wesedz on smithi069 2024-04-14T13:57:15.329 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:57:15 smithi069 ceph-mon[29472]: pgmap v75: 161 pgs: 161 active+clean; 581 KiB data, 53 MiB used, 715 GiB / 715 GiB avail; 287 B/s wr, 0 op/s 2024-04-14T13:57:15.345 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:57:15 smithi122 ceph-mon[25721]: from='mgr.14448 172.21.15.69:0/453473650' entity='mgr.y' 2024-04-14T13:57:15.345 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:57:15 smithi122 ceph-mon[25721]: from='mgr.14448 172.21.15.69:0/453473650' entity='mgr.y' 2024-04-14T13:57:15.345 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:57:15 smithi122 ceph-mon[25721]: from='mgr.14448 172.21.15.69:0/453473650' entity='mgr.y' 2024-04-14T13:57:15.345 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:57:15 smithi122 ceph-mon[25721]: from='mgr.14448 172.21.15.69:0/453473650' entity='mgr.y' 2024-04-14T13:57:15.345 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:57:15 smithi122 ceph-mon[25721]: Saving service rgw.foo spec with placement count:2 2024-04-14T13:57:15.345 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:57:15 smithi122 ceph-mon[25721]: from='mgr.14448 172.21.15.69:0/453473650' entity='mgr.y' 2024-04-14T13:57:15.345 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:57:15 smithi122 ceph-mon[25721]: from='mgr.14448 172.21.15.69:0/453473650' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi069.wesedz", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-04-14T13:57:15.346 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:57:15 smithi122 ceph-mon[25721]: from='mgr.14448 172.21.15.69:0/453473650' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi069.wesedz", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]': finished 2024-04-14T13:57:15.346 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:57:15 smithi122 ceph-mon[25721]: from='mgr.14448 172.21.15.69:0/453473650' entity='mgr.y' 2024-04-14T13:57:15.346 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:57:15 smithi122 ceph-mon[25721]: from='mgr.14448 172.21.15.69:0/453473650' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T13:57:15.346 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:57:15 smithi122 ceph-mon[25721]: Deploying daemon rgw.foo.smithi069.wesedz on smithi069 2024-04-14T13:57:15.346 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:57:15 smithi122 ceph-mon[25721]: pgmap v75: 161 pgs: 161 active+clean; 581 KiB data, 53 MiB used, 715 GiB / 715 GiB avail; 287 B/s wr, 0 op/s 2024-04-14T13:57:16.317 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:57:16 smithi069 ceph-mon[24818]: osdmap e77: 8 total, 8 up, 8 in 2024-04-14T13:57:16.317 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:57:16 smithi069 ceph-mon[29472]: osdmap e77: 8 total, 8 up, 8 in 2024-04-14T13:57:16.595 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:57:16 smithi122 ceph-mon[25721]: osdmap e77: 8 total, 8 up, 8 in 2024-04-14T13:57:17.443 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:57:17 smithi069 ceph-mon[24818]: pgmap v77: 161 pgs: 161 active+clean; 581 KiB data, 53 MiB used, 715 GiB / 715 GiB avail; 255 B/s wr, 0 op/s 2024-04-14T13:57:17.443 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:57:17 smithi069 ceph-mon[29472]: pgmap v77: 161 pgs: 161 active+clean; 581 KiB data, 53 MiB used, 715 GiB / 715 GiB avail; 255 B/s wr, 0 op/s 2024-04-14T13:57:17.595 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:57:17 smithi122 ceph-mon[25721]: pgmap v77: 161 pgs: 161 active+clean; 581 KiB data, 53 MiB used, 715 GiB / 715 GiB avail; 255 B/s wr, 0 op/s 2024-04-14T13:57:17.595 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Apr 14 13:57:17 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-x[26913]: ::ffff:172.21.15.122 - - [14/Apr/2024:13:57:17] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-04-14T13:57:19.067 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:57:18 smithi069 ceph-mon[24818]: from='mgr.14448 172.21.15.69:0/453473650' entity='mgr.y' 2024-04-14T13:57:19.067 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:57:18 smithi069 ceph-mon[24818]: from='mgr.14448 172.21.15.69:0/453473650' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi122.jiobyv", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-04-14T13:57:19.067 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:57:18 smithi069 ceph-mon[24818]: from='mgr.14448 172.21.15.69:0/453473650' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi122.jiobyv", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]': finished 2024-04-14T13:57:19.068 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:57:18 smithi069 ceph-mon[24818]: from='mgr.14448 172.21.15.69:0/453473650' entity='mgr.y' 2024-04-14T13:57:19.068 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:57:18 smithi069 ceph-mon[24818]: from='mgr.14448 172.21.15.69:0/453473650' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T13:57:19.068 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:57:18 smithi069 ceph-mon[24818]: Deploying daemon rgw.foo.smithi122.jiobyv on smithi122 2024-04-14T13:57:19.068 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:57:18 smithi069 ceph-mon[24818]: pgmap v78: 161 pgs: 161 active+clean; 581 KiB data, 53 MiB used, 715 GiB / 715 GiB avail; 224 B/s wr, 0 op/s 2024-04-14T13:57:19.068 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:57:18 smithi069 ceph-mon[29472]: from='mgr.14448 172.21.15.69:0/453473650' entity='mgr.y' 2024-04-14T13:57:19.068 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:57:18 smithi069 ceph-mon[29472]: from='mgr.14448 172.21.15.69:0/453473650' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi122.jiobyv", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-04-14T13:57:19.068 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:57:18 smithi069 ceph-mon[29472]: from='mgr.14448 172.21.15.69:0/453473650' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi122.jiobyv", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]': finished 2024-04-14T13:57:19.068 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:57:18 smithi069 ceph-mon[29472]: from='mgr.14448 172.21.15.69:0/453473650' entity='mgr.y' 2024-04-14T13:57:19.068 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:57:18 smithi069 ceph-mon[29472]: from='mgr.14448 172.21.15.69:0/453473650' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T13:57:19.068 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:57:18 smithi069 ceph-mon[29472]: Deploying daemon rgw.foo.smithi122.jiobyv on smithi122 2024-04-14T13:57:19.068 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:57:18 smithi069 ceph-mon[29472]: pgmap v78: 161 pgs: 161 active+clean; 581 KiB data, 53 MiB used, 715 GiB / 715 GiB avail; 224 B/s wr, 0 op/s 2024-04-14T13:57:19.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:57:18 smithi122 ceph-mon[25721]: from='mgr.14448 172.21.15.69:0/453473650' entity='mgr.y' 2024-04-14T13:57:19.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:57:18 smithi122 ceph-mon[25721]: from='mgr.14448 172.21.15.69:0/453473650' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi122.jiobyv", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-04-14T13:57:19.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:57:18 smithi122 ceph-mon[25721]: from='mgr.14448 172.21.15.69:0/453473650' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi122.jiobyv", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]': finished 2024-04-14T13:57:19.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:57:18 smithi122 ceph-mon[25721]: from='mgr.14448 172.21.15.69:0/453473650' entity='mgr.y' 2024-04-14T13:57:19.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:57:18 smithi122 ceph-mon[25721]: from='mgr.14448 172.21.15.69:0/453473650' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T13:57:19.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:57:18 smithi122 ceph-mon[25721]: Deploying daemon rgw.foo.smithi122.jiobyv on smithi122 2024-04-14T13:57:19.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:57:18 smithi122 ceph-mon[25721]: pgmap v78: 161 pgs: 161 active+clean; 581 KiB data, 53 MiB used, 715 GiB / 715 GiB avail; 224 B/s wr, 0 op/s 2024-04-14T13:57:19.817 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 14 13:57:19 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-y[25054]: ::ffff:172.21.15.122 - - [14/Apr/2024:13:57:19] "GET /metrics HTTP/1.1" 200 197961 "" "Prometheus/2.33.4" 2024-04-14T13:57:20.595 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:57:20 smithi122 ceph-mon[25721]: from='mgr.14448 172.21.15.69:0/453473650' entity='mgr.y' 2024-04-14T13:57:20.595 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:57:20 smithi122 ceph-mon[25721]: from='mgr.14448 172.21.15.69:0/453473650' entity='mgr.y' 2024-04-14T13:57:20.595 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:57:20 smithi122 ceph-mon[25721]: from='mgr.14448 172.21.15.69:0/453473650' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi069.yuumtr", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-04-14T13:57:20.595 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:57:20 smithi122 ceph-mon[25721]: from='mgr.14448 172.21.15.69:0/453473650' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi069.yuumtr", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]': finished 2024-04-14T13:57:20.595 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:57:20 smithi122 ceph-mon[25721]: from='mgr.14448 172.21.15.69:0/453473650' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T13:57:20.786 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:57:20 smithi069 ceph-mon[29472]: from='mgr.14448 172.21.15.69:0/453473650' entity='mgr.y' 2024-04-14T13:57:20.787 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:57:20 smithi069 ceph-mon[29472]: from='mgr.14448 172.21.15.69:0/453473650' entity='mgr.y' 2024-04-14T13:57:20.787 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:57:20 smithi069 ceph-mon[29472]: from='mgr.14448 172.21.15.69:0/453473650' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi069.yuumtr", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-04-14T13:57:20.787 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:57:20 smithi069 ceph-mon[29472]: from='mgr.14448 172.21.15.69:0/453473650' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi069.yuumtr", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]': finished 2024-04-14T13:57:20.787 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:57:20 smithi069 ceph-mon[29472]: from='mgr.14448 172.21.15.69:0/453473650' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T13:57:20.787 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:57:20 smithi069 ceph-mon[24818]: from='mgr.14448 172.21.15.69:0/453473650' entity='mgr.y' 2024-04-14T13:57:20.787 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:57:20 smithi069 ceph-mon[24818]: from='mgr.14448 172.21.15.69:0/453473650' entity='mgr.y' 2024-04-14T13:57:20.787 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:57:20 smithi069 ceph-mon[24818]: from='mgr.14448 172.21.15.69:0/453473650' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi069.yuumtr", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-04-14T13:57:20.787 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:57:20 smithi069 ceph-mon[24818]: from='mgr.14448 172.21.15.69:0/453473650' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi069.yuumtr", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]': finished 2024-04-14T13:57:20.787 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:57:20 smithi069 ceph-mon[24818]: from='mgr.14448 172.21.15.69:0/453473650' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T13:57:21.595 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:57:21 smithi122 ceph-mon[25721]: Checking pool "foo" exists for service iscsi.foo 2024-04-14T13:57:21.595 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:57:21 smithi122 ceph-mon[25721]: pgmap v79: 161 pgs: 161 active+clean; 584 KiB data, 55 MiB used, 715 GiB / 715 GiB avail; 68 KiB/s rd, 3.6 KiB/s wr, 144 op/s 2024-04-14T13:57:21.595 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:57:21 smithi122 ceph-mon[25721]: Deploying daemon iscsi.foo.smithi069.yuumtr on smithi069 2024-04-14T13:57:21.817 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:57:21 smithi069 ceph-mon[24818]: Checking pool "foo" exists for service iscsi.foo 2024-04-14T13:57:21.817 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:57:21 smithi069 ceph-mon[24818]: pgmap v79: 161 pgs: 161 active+clean; 584 KiB data, 55 MiB used, 715 GiB / 715 GiB avail; 68 KiB/s rd, 3.6 KiB/s wr, 144 op/s 2024-04-14T13:57:21.817 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:57:21 smithi069 ceph-mon[24818]: Deploying daemon iscsi.foo.smithi069.yuumtr on smithi069 2024-04-14T13:57:21.817 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:57:21 smithi069 ceph-mon[29472]: Checking pool "foo" exists for service iscsi.foo 2024-04-14T13:57:21.817 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:57:21 smithi069 ceph-mon[29472]: pgmap v79: 161 pgs: 161 active+clean; 584 KiB data, 55 MiB used, 715 GiB / 715 GiB avail; 68 KiB/s rd, 3.6 KiB/s wr, 144 op/s 2024-04-14T13:57:21.817 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:57:21 smithi069 ceph-mon[29472]: Deploying daemon iscsi.foo.smithi069.yuumtr on smithi069 2024-04-14T13:57:23.817 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:57:23 smithi069 ceph-mon[29472]: pgmap v80: 161 pgs: 161 active+clean; 584 KiB data, 55 MiB used, 715 GiB / 715 GiB avail; 68 KiB/s rd, 3.4 KiB/s wr, 144 op/s 2024-04-14T13:57:23.817 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:57:23 smithi069 ceph-mon[29472]: mgrmap e20: y(active, since 118s), standbys: x 2024-04-14T13:57:23.817 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:57:23 smithi069 ceph-mon[29472]: from='mgr.14448 172.21.15.69:0/453473650' entity='mgr.y' 2024-04-14T13:57:23.817 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:57:23 smithi069 ceph-mon[29472]: from='mgr.14448 172.21.15.69:0/453473650' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T13:57:23.817 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:57:23 smithi069 ceph-mon[29472]: from='mgr.14448 172.21.15.69:0/453473650' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-14T13:57:23.818 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:57:23 smithi069 ceph-mon[24818]: pgmap v80: 161 pgs: 161 active+clean; 584 KiB data, 55 MiB used, 715 GiB / 715 GiB avail; 68 KiB/s rd, 3.4 KiB/s wr, 144 op/s 2024-04-14T13:57:23.818 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:57:23 smithi069 ceph-mon[24818]: mgrmap e20: y(active, since 118s), standbys: x 2024-04-14T13:57:23.818 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:57:23 smithi069 ceph-mon[24818]: from='mgr.14448 172.21.15.69:0/453473650' entity='mgr.y' 2024-04-14T13:57:23.818 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:57:23 smithi069 ceph-mon[24818]: from='mgr.14448 172.21.15.69:0/453473650' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T13:57:23.818 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:57:23 smithi069 ceph-mon[24818]: from='mgr.14448 172.21.15.69:0/453473650' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-14T13:57:23.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:57:23 smithi122 ceph-mon[25721]: pgmap v80: 161 pgs: 161 active+clean; 584 KiB data, 55 MiB used, 715 GiB / 715 GiB avail; 68 KiB/s rd, 3.4 KiB/s wr, 144 op/s 2024-04-14T13:57:23.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:57:23 smithi122 ceph-mon[25721]: mgrmap e20: y(active, since 118s), standbys: x 2024-04-14T13:57:23.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:57:23 smithi122 ceph-mon[25721]: from='mgr.14448 172.21.15.69:0/453473650' entity='mgr.y' 2024-04-14T13:57:23.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:57:23 smithi122 ceph-mon[25721]: from='mgr.14448 172.21.15.69:0/453473650' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T13:57:23.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:57:23 smithi122 ceph-mon[25721]: from='mgr.14448 172.21.15.69:0/453473650' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-14T13:57:24.567 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:57:24 smithi069 ceph-mon[24818]: from='client.? 172.21.15.69:0/2375511018' entity='client.iscsi.foo.smithi069.yuumtr' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-04-14T13:57:24.567 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:57:24 smithi069 ceph-mon[24818]: from='client.? 172.21.15.69:0/514596082' entity='client.iscsi.foo.smithi069.yuumtr' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.69:6800/2212179823"}]: dispatch 2024-04-14T13:57:24.567 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:57:24 smithi069 ceph-mon[24818]: from='mgr.14448 172.21.15.69:0/453473650' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.5", "id": [7, 2]}]: dispatch 2024-04-14T13:57:24.567 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:57:24 smithi069 ceph-mon[24818]: from='mgr.14448 172.21.15.69:0/453473650' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.6", "id": [1, 5]}]: dispatch 2024-04-14T13:57:24.567 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:57:24 smithi069 ceph-mon[24818]: from='mgr.14448 172.21.15.69:0/453473650' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.b", "id": [1, 2]}]: dispatch 2024-04-14T13:57:24.567 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:57:24 smithi069 ceph-mon[24818]: from='mgr.14448 172.21.15.69:0/453473650' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1d", "id": [1, 2]}]: dispatch 2024-04-14T13:57:24.567 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:57:24 smithi069 ceph-mon[24818]: from='mgr.14448 172.21.15.69:0/453473650' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-04-14T13:57:24.568 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:57:24 smithi069 ceph-mon[24818]: from='mgr.14448 172.21.15.69:0/453473650' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-04-14T13:57:24.568 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:57:24 smithi069 ceph-mon[24818]: from='mgr.14448 172.21.15.69:0/453473650' entity='mgr.y' 2024-04-14T13:57:24.568 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:57:24 smithi069 ceph-mon[29472]: from='client.? 172.21.15.69:0/2375511018' entity='client.iscsi.foo.smithi069.yuumtr' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-04-14T13:57:24.568 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:57:24 smithi069 ceph-mon[29472]: from='client.? 172.21.15.69:0/514596082' entity='client.iscsi.foo.smithi069.yuumtr' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.69:6800/2212179823"}]: dispatch 2024-04-14T13:57:24.568 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:57:24 smithi069 ceph-mon[29472]: from='mgr.14448 172.21.15.69:0/453473650' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.5", "id": [7, 2]}]: dispatch 2024-04-14T13:57:24.568 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:57:24 smithi069 ceph-mon[29472]: from='mgr.14448 172.21.15.69:0/453473650' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.6", "id": [1, 5]}]: dispatch 2024-04-14T13:57:24.568 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:57:24 smithi069 ceph-mon[29472]: from='mgr.14448 172.21.15.69:0/453473650' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.b", "id": [1, 2]}]: dispatch 2024-04-14T13:57:24.568 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:57:24 smithi069 ceph-mon[29472]: from='mgr.14448 172.21.15.69:0/453473650' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1d", "id": [1, 2]}]: dispatch 2024-04-14T13:57:24.568 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:57:24 smithi069 ceph-mon[29472]: from='mgr.14448 172.21.15.69:0/453473650' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-04-14T13:57:24.568 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:57:24 smithi069 ceph-mon[29472]: from='mgr.14448 172.21.15.69:0/453473650' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-04-14T13:57:24.568 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:57:24 smithi069 ceph-mon[29472]: from='mgr.14448 172.21.15.69:0/453473650' entity='mgr.y' 2024-04-14T13:57:24.838 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:57:24 smithi122 ceph-mon[25721]: from='client.? 172.21.15.69:0/2375511018' entity='client.iscsi.foo.smithi069.yuumtr' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-04-14T13:57:24.838 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:57:24 smithi122 ceph-mon[25721]: from='client.? 172.21.15.69:0/514596082' entity='client.iscsi.foo.smithi069.yuumtr' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.69:6800/2212179823"}]: dispatch 2024-04-14T13:57:24.838 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:57:24 smithi122 ceph-mon[25721]: from='mgr.14448 172.21.15.69:0/453473650' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.5", "id": [7, 2]}]: dispatch 2024-04-14T13:57:24.838 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:57:24 smithi122 ceph-mon[25721]: from='mgr.14448 172.21.15.69:0/453473650' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.6", "id": [1, 5]}]: dispatch 2024-04-14T13:57:24.838 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:57:24 smithi122 ceph-mon[25721]: from='mgr.14448 172.21.15.69:0/453473650' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.b", "id": [1, 2]}]: dispatch 2024-04-14T13:57:24.838 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:57:24 smithi122 ceph-mon[25721]: from='mgr.14448 172.21.15.69:0/453473650' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1d", "id": [1, 2]}]: dispatch 2024-04-14T13:57:24.838 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:57:24 smithi122 ceph-mon[25721]: from='mgr.14448 172.21.15.69:0/453473650' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-04-14T13:57:24.839 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:57:24 smithi122 ceph-mon[25721]: from='mgr.14448 172.21.15.69:0/453473650' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-04-14T13:57:24.839 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:57:24 smithi122 ceph-mon[25721]: from='mgr.14448 172.21.15.69:0/453473650' entity='mgr.y' 2024-04-14T13:57:25.746 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:57:25 smithi069 ceph-mon[24818]: pgmap v81: 161 pgs: 161 active+clean; 584 KiB data, 55 MiB used, 715 GiB / 715 GiB avail; 68 KiB/s rd, 3.4 KiB/s wr, 144 op/s 2024-04-14T13:57:25.746 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:57:25 smithi069 ceph-mon[24818]: from='client.? 172.21.15.69:0/514596082' entity='client.iscsi.foo.smithi069.yuumtr' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.69:6800/2212179823"}]': finished 2024-04-14T13:57:25.746 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:57:25 smithi069 ceph-mon[24818]: from='mgr.14448 172.21.15.69:0/453473650' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.5", "id": [7, 2]}]': finished 2024-04-14T13:57:25.746 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:57:25 smithi069 ceph-mon[24818]: from='mgr.14448 172.21.15.69:0/453473650' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.6", "id": [1, 5]}]': finished 2024-04-14T13:57:25.746 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:57:25 smithi069 ceph-mon[24818]: from='mgr.14448 172.21.15.69:0/453473650' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.b", "id": [1, 2]}]': finished 2024-04-14T13:57:25.746 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:57:25 smithi069 ceph-mon[24818]: from='mgr.14448 172.21.15.69:0/453473650' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1d", "id": [1, 2]}]': finished 2024-04-14T13:57:25.746 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:57:25 smithi069 ceph-mon[24818]: osdmap e78: 8 total, 8 up, 8 in 2024-04-14T13:57:25.746 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:57:25 smithi069 ceph-mon[24818]: from='client.? 172.21.15.69:0/2993016378' entity='client.iscsi.foo.smithi069.yuumtr' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.69:0/1453816126"}]: dispatch 2024-04-14T13:57:25.746 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:57:25 smithi069 ceph-mon[24818]: from='client.? ' entity='client.iscsi.foo.smithi069.yuumtr' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.69:0/1453816126"}]: dispatch 2024-04-14T13:57:25.747 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:57:25 smithi069 ceph-mon[29472]: pgmap v81: 161 pgs: 161 active+clean; 584 KiB data, 55 MiB used, 715 GiB / 715 GiB avail; 68 KiB/s rd, 3.4 KiB/s wr, 144 op/s 2024-04-14T13:57:25.747 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:57:25 smithi069 ceph-mon[29472]: from='client.? 172.21.15.69:0/514596082' entity='client.iscsi.foo.smithi069.yuumtr' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.69:6800/2212179823"}]': finished 2024-04-14T13:57:25.747 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:57:25 smithi069 ceph-mon[29472]: from='mgr.14448 172.21.15.69:0/453473650' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.5", "id": [7, 2]}]': finished 2024-04-14T13:57:25.747 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:57:25 smithi069 ceph-mon[29472]: from='mgr.14448 172.21.15.69:0/453473650' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.6", "id": [1, 5]}]': finished 2024-04-14T13:57:25.747 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:57:25 smithi069 ceph-mon[29472]: from='mgr.14448 172.21.15.69:0/453473650' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.b", "id": [1, 2]}]': finished 2024-04-14T13:57:25.747 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:57:25 smithi069 ceph-mon[29472]: from='mgr.14448 172.21.15.69:0/453473650' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1d", "id": [1, 2]}]': finished 2024-04-14T13:57:25.747 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:57:25 smithi069 ceph-mon[29472]: osdmap e78: 8 total, 8 up, 8 in 2024-04-14T13:57:25.747 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:57:25 smithi069 ceph-mon[29472]: from='client.? 172.21.15.69:0/2993016378' entity='client.iscsi.foo.smithi069.yuumtr' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.69:0/1453816126"}]: dispatch 2024-04-14T13:57:25.747 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:57:25 smithi069 ceph-mon[29472]: from='client.? ' entity='client.iscsi.foo.smithi069.yuumtr' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.69:0/1453816126"}]: dispatch 2024-04-14T13:57:25.769 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:57:25 smithi122 ceph-mon[25721]: pgmap v81: 161 pgs: 161 active+clean; 584 KiB data, 55 MiB used, 715 GiB / 715 GiB avail; 68 KiB/s rd, 3.4 KiB/s wr, 144 op/s 2024-04-14T13:57:25.769 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:57:25 smithi122 ceph-mon[25721]: from='client.? 172.21.15.69:0/514596082' entity='client.iscsi.foo.smithi069.yuumtr' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.69:6800/2212179823"}]': finished 2024-04-14T13:57:25.769 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:57:25 smithi122 ceph-mon[25721]: from='mgr.14448 172.21.15.69:0/453473650' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.5", "id": [7, 2]}]': finished 2024-04-14T13:57:25.769 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:57:25 smithi122 ceph-mon[25721]: from='mgr.14448 172.21.15.69:0/453473650' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.6", "id": [1, 5]}]': finished 2024-04-14T13:57:25.769 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:57:25 smithi122 ceph-mon[25721]: from='mgr.14448 172.21.15.69:0/453473650' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.b", "id": [1, 2]}]': finished 2024-04-14T13:57:25.769 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:57:25 smithi122 ceph-mon[25721]: from='mgr.14448 172.21.15.69:0/453473650' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1d", "id": [1, 2]}]': finished 2024-04-14T13:57:25.769 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:57:25 smithi122 ceph-mon[25721]: osdmap e78: 8 total, 8 up, 8 in 2024-04-14T13:57:25.769 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:57:25 smithi122 ceph-mon[25721]: from='client.? 172.21.15.69:0/2993016378' entity='client.iscsi.foo.smithi069.yuumtr' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.69:0/1453816126"}]: dispatch 2024-04-14T13:57:25.769 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:57:25 smithi122 ceph-mon[25721]: from='client.? ' entity='client.iscsi.foo.smithi069.yuumtr' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.69:0/1453816126"}]: dispatch 2024-04-14T13:57:26.595 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:57:26 smithi122 ceph-mon[25721]: from='client.? ' entity='client.iscsi.foo.smithi069.yuumtr' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.69:0/1453816126"}]': finished 2024-04-14T13:57:26.595 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:57:26 smithi122 ceph-mon[25721]: osdmap e79: 8 total, 8 up, 8 in 2024-04-14T13:57:26.595 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:57:26 smithi122 ceph-mon[25721]: from='client.? 172.21.15.69:0/3400682442' entity='client.iscsi.foo.smithi069.yuumtr' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.69:0/3639638985"}]: dispatch 2024-04-14T13:57:26.725 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:57:26 smithi069 ceph-mon[24818]: from='client.? ' entity='client.iscsi.foo.smithi069.yuumtr' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.69:0/1453816126"}]': finished 2024-04-14T13:57:26.725 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:57:26 smithi069 ceph-mon[24818]: osdmap e79: 8 total, 8 up, 8 in 2024-04-14T13:57:26.725 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:57:26 smithi069 ceph-mon[24818]: from='client.? 172.21.15.69:0/3400682442' entity='client.iscsi.foo.smithi069.yuumtr' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.69:0/3639638985"}]: dispatch 2024-04-14T13:57:26.725 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:57:26 smithi069 ceph-mon[29472]: from='client.? ' entity='client.iscsi.foo.smithi069.yuumtr' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.69:0/1453816126"}]': finished 2024-04-14T13:57:26.725 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:57:26 smithi069 ceph-mon[29472]: osdmap e79: 8 total, 8 up, 8 in 2024-04-14T13:57:26.725 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:57:26 smithi069 ceph-mon[29472]: from='client.? 172.21.15.69:0/3400682442' entity='client.iscsi.foo.smithi069.yuumtr' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.69:0/3639638985"}]: dispatch 2024-04-14T13:57:27.567 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:57:27 smithi069 ceph-mon[24818]: pgmap v84: 161 pgs: 4 peering, 157 active+clean; 585 KiB data, 56 MiB used, 715 GiB / 715 GiB avail; 139 KiB/s rd, 4.4 KiB/s wr, 266 op/s 2024-04-14T13:57:27.567 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:57:27 smithi069 ceph-mon[24818]: from='client.? 172.21.15.69:0/3400682442' entity='client.iscsi.foo.smithi069.yuumtr' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.69:0/3639638985"}]': finished 2024-04-14T13:57:27.568 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:57:27 smithi069 ceph-mon[24818]: osdmap e80: 8 total, 8 up, 8 in 2024-04-14T13:57:27.568 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:57:27 smithi069 ceph-mon[24818]: from='client.? 172.21.15.69:0/1370291664' entity='client.iscsi.foo.smithi069.yuumtr' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.69:0/3170426090"}]: dispatch 2024-04-14T13:57:27.568 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:57:27 smithi069 ceph-mon[24818]: from='mgr.14448 172.21.15.69:0/453473650' entity='mgr.y' 2024-04-14T13:57:27.568 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:57:27 smithi069 ceph-mon[24818]: from='mgr.14448 172.21.15.69:0/453473650' entity='mgr.y' 2024-04-14T13:57:27.568 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:57:27 smithi069 ceph-mon[24818]: from='mgr.14448 172.21.15.69:0/453473650' entity='mgr.y' 2024-04-14T13:57:27.568 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:57:27 smithi069 ceph-mon[24818]: from='mgr.14448 172.21.15.69:0/453473650' entity='mgr.y' 2024-04-14T13:57:27.568 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:57:27 smithi069 ceph-mon[24818]: from='mgr.14448 172.21.15.69:0/453473650' entity='mgr.y' 2024-04-14T13:57:27.568 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:57:27 smithi069 ceph-mon[24818]: from='mgr.14448 172.21.15.69:0/453473650' entity='mgr.y' cmd=[{"prefix": "dashboard iscsi-gateway-list"}]: dispatch 2024-04-14T13:57:27.568 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:57:27 smithi069 ceph-mon[24818]: from='mgr.14448 172.21.15.69:0/453473650' entity='mgr.y' cmd=[{"prefix": "dashboard set-iscsi-api-ssl-verification", "value": "true"}]: dispatch 2024-04-14T13:57:27.568 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:57:27 smithi069 ceph-mon[24818]: from='mgr.14448 172.21.15.69:0/453473650' entity='mgr.y' 2024-04-14T13:57:27.568 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:57:27 smithi069 ceph-mon[24818]: from='mgr.14448 172.21.15.69:0/453473650' entity='mgr.y' cmd=[{"prefix": "dashboard iscsi-gateway-add", "name": "smithi069"}]: dispatch 2024-04-14T13:57:27.568 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:57:27 smithi069 ceph-mon[24818]: from='mgr.14448 172.21.15.69:0/453473650' entity='mgr.y' 2024-04-14T13:57:27.568 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:57:27 smithi069 ceph-mon[24818]: from='mgr.14448 172.21.15.69:0/453473650' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T13:57:27.568 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:57:27 smithi069 ceph-mon[24818]: from='mgr.14448 172.21.15.69:0/453473650' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-14T13:57:27.569 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:57:27 smithi069 ceph-mon[29472]: pgmap v84: 161 pgs: 4 peering, 157 active+clean; 585 KiB data, 56 MiB used, 715 GiB / 715 GiB avail; 139 KiB/s rd, 4.4 KiB/s wr, 266 op/s 2024-04-14T13:57:27.569 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:57:27 smithi069 ceph-mon[29472]: from='client.? 172.21.15.69:0/3400682442' entity='client.iscsi.foo.smithi069.yuumtr' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.69:0/3639638985"}]': finished 2024-04-14T13:57:27.569 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:57:27 smithi069 ceph-mon[29472]: osdmap e80: 8 total, 8 up, 8 in 2024-04-14T13:57:27.569 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:57:27 smithi069 ceph-mon[29472]: from='client.? 172.21.15.69:0/1370291664' entity='client.iscsi.foo.smithi069.yuumtr' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.69:0/3170426090"}]: dispatch 2024-04-14T13:57:27.569 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:57:27 smithi069 ceph-mon[29472]: from='mgr.14448 172.21.15.69:0/453473650' entity='mgr.y' 2024-04-14T13:57:27.569 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:57:27 smithi069 ceph-mon[29472]: from='mgr.14448 172.21.15.69:0/453473650' entity='mgr.y' 2024-04-14T13:57:27.569 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:57:27 smithi069 ceph-mon[29472]: from='mgr.14448 172.21.15.69:0/453473650' entity='mgr.y' 2024-04-14T13:57:27.569 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:57:27 smithi069 ceph-mon[29472]: from='mgr.14448 172.21.15.69:0/453473650' entity='mgr.y' 2024-04-14T13:57:27.569 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:57:27 smithi069 ceph-mon[29472]: from='mgr.14448 172.21.15.69:0/453473650' entity='mgr.y' 2024-04-14T13:57:27.569 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:57:27 smithi069 ceph-mon[29472]: from='mgr.14448 172.21.15.69:0/453473650' entity='mgr.y' cmd=[{"prefix": "dashboard iscsi-gateway-list"}]: dispatch 2024-04-14T13:57:27.569 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:57:27 smithi069 ceph-mon[29472]: from='mgr.14448 172.21.15.69:0/453473650' entity='mgr.y' cmd=[{"prefix": "dashboard set-iscsi-api-ssl-verification", "value": "true"}]: dispatch 2024-04-14T13:57:27.569 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:57:27 smithi069 ceph-mon[29472]: from='mgr.14448 172.21.15.69:0/453473650' entity='mgr.y' 2024-04-14T13:57:27.569 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:57:27 smithi069 ceph-mon[29472]: from='mgr.14448 172.21.15.69:0/453473650' entity='mgr.y' cmd=[{"prefix": "dashboard iscsi-gateway-add", "name": "smithi069"}]: dispatch 2024-04-14T13:57:27.570 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:57:27 smithi069 ceph-mon[29472]: from='mgr.14448 172.21.15.69:0/453473650' entity='mgr.y' 2024-04-14T13:57:27.570 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:57:27 smithi069 ceph-mon[29472]: from='mgr.14448 172.21.15.69:0/453473650' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T13:57:27.570 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:57:27 smithi069 ceph-mon[29472]: from='mgr.14448 172.21.15.69:0/453473650' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-14T13:57:27.595 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Apr 14 13:57:27 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-x[26913]: ::ffff:172.21.15.122 - - [14/Apr/2024:13:57:27] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-04-14T13:57:27.595 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:57:27 smithi122 ceph-mon[25721]: pgmap v84: 161 pgs: 4 peering, 157 active+clean; 585 KiB data, 56 MiB used, 715 GiB / 715 GiB avail; 139 KiB/s rd, 4.4 KiB/s wr, 266 op/s 2024-04-14T13:57:27.595 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:57:27 smithi122 ceph-mon[25721]: from='client.? 172.21.15.69:0/3400682442' entity='client.iscsi.foo.smithi069.yuumtr' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.69:0/3639638985"}]': finished 2024-04-14T13:57:27.595 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:57:27 smithi122 ceph-mon[25721]: osdmap e80: 8 total, 8 up, 8 in 2024-04-14T13:57:27.595 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:57:27 smithi122 ceph-mon[25721]: from='client.? 172.21.15.69:0/1370291664' entity='client.iscsi.foo.smithi069.yuumtr' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.69:0/3170426090"}]: dispatch 2024-04-14T13:57:27.595 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:57:27 smithi122 ceph-mon[25721]: from='mgr.14448 172.21.15.69:0/453473650' entity='mgr.y' 2024-04-14T13:57:27.595 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:57:27 smithi122 ceph-mon[25721]: from='mgr.14448 172.21.15.69:0/453473650' entity='mgr.y' 2024-04-14T13:57:27.595 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:57:27 smithi122 ceph-mon[25721]: from='mgr.14448 172.21.15.69:0/453473650' entity='mgr.y' 2024-04-14T13:57:27.596 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:57:27 smithi122 ceph-mon[25721]: from='mgr.14448 172.21.15.69:0/453473650' entity='mgr.y' 2024-04-14T13:57:27.596 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:57:27 smithi122 ceph-mon[25721]: from='mgr.14448 172.21.15.69:0/453473650' entity='mgr.y' 2024-04-14T13:57:27.596 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:57:27 smithi122 ceph-mon[25721]: from='mgr.14448 172.21.15.69:0/453473650' entity='mgr.y' cmd=[{"prefix": "dashboard iscsi-gateway-list"}]: dispatch 2024-04-14T13:57:27.596 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:57:27 smithi122 ceph-mon[25721]: from='mgr.14448 172.21.15.69:0/453473650' entity='mgr.y' cmd=[{"prefix": "dashboard set-iscsi-api-ssl-verification", "value": "true"}]: dispatch 2024-04-14T13:57:27.596 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:57:27 smithi122 ceph-mon[25721]: from='mgr.14448 172.21.15.69:0/453473650' entity='mgr.y' 2024-04-14T13:57:27.596 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:57:27 smithi122 ceph-mon[25721]: from='mgr.14448 172.21.15.69:0/453473650' entity='mgr.y' cmd=[{"prefix": "dashboard iscsi-gateway-add", "name": "smithi069"}]: dispatch 2024-04-14T13:57:27.596 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:57:27 smithi122 ceph-mon[25721]: from='mgr.14448 172.21.15.69:0/453473650' entity='mgr.y' 2024-04-14T13:57:27.596 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:57:27 smithi122 ceph-mon[25721]: from='mgr.14448 172.21.15.69:0/453473650' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T13:57:27.596 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:57:27 smithi122 ceph-mon[25721]: from='mgr.14448 172.21.15.69:0/453473650' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-14T13:57:28.716 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:57:28 smithi122 ceph-mon[25721]: Checking dashboard <-> RGW credentials 2024-04-14T13:57:28.716 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:57:28 smithi122 ceph-mon[25721]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard iscsi-gateway-list"}]: dispatch 2024-04-14T13:57:28.716 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:57:28 smithi122 ceph-mon[25721]: Adding iSCSI gateway http://:@172.21.15.69:5000 to Dashboard 2024-04-14T13:57:28.716 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:57:28 smithi122 ceph-mon[25721]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-iscsi-api-ssl-verification", "value": "true"}]: dispatch 2024-04-14T13:57:28.716 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:57:28 smithi122 ceph-mon[25721]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard iscsi-gateway-add", "name": "smithi069"}]: dispatch 2024-04-14T13:57:28.716 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:57:28 smithi122 ceph-mon[25721]: from='client.? 172.21.15.69:0/1370291664' entity='client.iscsi.foo.smithi069.yuumtr' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.69:0/3170426090"}]': finished 2024-04-14T13:57:28.716 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:57:28 smithi122 ceph-mon[25721]: osdmap e81: 8 total, 8 up, 8 in 2024-04-14T13:57:28.716 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:57:28 smithi122 ceph-mon[25721]: from='client.? 172.21.15.69:0/3123510337' entity='client.iscsi.foo.smithi069.yuumtr' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.69:0/3397430943"}]: dispatch 2024-04-14T13:57:28.739 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:57:28 smithi069 ceph-mon[29472]: Checking dashboard <-> RGW credentials 2024-04-14T13:57:28.739 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:57:28 smithi069 ceph-mon[29472]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard iscsi-gateway-list"}]: dispatch 2024-04-14T13:57:28.739 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:57:28 smithi069 ceph-mon[29472]: Adding iSCSI gateway http://:@172.21.15.69:5000 to Dashboard 2024-04-14T13:57:28.739 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:57:28 smithi069 ceph-mon[29472]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-iscsi-api-ssl-verification", "value": "true"}]: dispatch 2024-04-14T13:57:28.739 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:57:28 smithi069 ceph-mon[29472]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard iscsi-gateway-add", "name": "smithi069"}]: dispatch 2024-04-14T13:57:28.739 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:57:28 smithi069 ceph-mon[29472]: from='client.? 172.21.15.69:0/1370291664' entity='client.iscsi.foo.smithi069.yuumtr' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.69:0/3170426090"}]': finished 2024-04-14T13:57:28.739 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:57:28 smithi069 ceph-mon[29472]: osdmap e81: 8 total, 8 up, 8 in 2024-04-14T13:57:28.739 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:57:28 smithi069 ceph-mon[29472]: from='client.? 172.21.15.69:0/3123510337' entity='client.iscsi.foo.smithi069.yuumtr' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.69:0/3397430943"}]: dispatch 2024-04-14T13:57:28.739 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:57:28 smithi069 ceph-mon[24818]: Checking dashboard <-> RGW credentials 2024-04-14T13:57:28.740 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:57:28 smithi069 ceph-mon[24818]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard iscsi-gateway-list"}]: dispatch 2024-04-14T13:57:28.740 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:57:28 smithi069 ceph-mon[24818]: Adding iSCSI gateway http://:@172.21.15.69:5000 to Dashboard 2024-04-14T13:57:28.740 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:57:28 smithi069 ceph-mon[24818]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-iscsi-api-ssl-verification", "value": "true"}]: dispatch 2024-04-14T13:57:28.740 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:57:28 smithi069 ceph-mon[24818]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard iscsi-gateway-add", "name": "smithi069"}]: dispatch 2024-04-14T13:57:28.740 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:57:28 smithi069 ceph-mon[24818]: from='client.? 172.21.15.69:0/1370291664' entity='client.iscsi.foo.smithi069.yuumtr' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.69:0/3170426090"}]': finished 2024-04-14T13:57:28.740 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:57:28 smithi069 ceph-mon[24818]: osdmap e81: 8 total, 8 up, 8 in 2024-04-14T13:57:28.740 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:57:28 smithi069 ceph-mon[24818]: from='client.? 172.21.15.69:0/3123510337' entity='client.iscsi.foo.smithi069.yuumtr' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.69:0/3397430943"}]: dispatch 2024-04-14T13:57:29.701 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:57:29 smithi069 ceph-mon[24818]: pgmap v87: 161 pgs: 4 peering, 157 active+clean; 585 KiB data, 56 MiB used, 715 GiB / 715 GiB avail; 108 KiB/s rd, 255 B/s wr, 173 op/s 2024-04-14T13:57:29.701 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:57:29 smithi069 ceph-mon[24818]: from='client.? 172.21.15.69:0/3123510337' entity='client.iscsi.foo.smithi069.yuumtr' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.69:0/3397430943"}]': finished 2024-04-14T13:57:29.701 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:57:29 smithi069 ceph-mon[24818]: osdmap e82: 8 total, 8 up, 8 in 2024-04-14T13:57:29.702 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:57:29 smithi069 ceph-mon[24818]: from='client.? 172.21.15.69:0/2127092894' entity='client.iscsi.foo.smithi069.yuumtr' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.69:0/226283202"}]: dispatch 2024-04-14T13:57:29.702 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:57:29 smithi069 ceph-mon[24818]: from='client.? ' entity='client.iscsi.foo.smithi069.yuumtr' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.69:0/226283202"}]: dispatch 2024-04-14T13:57:29.702 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 14 13:57:29 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-y[25054]: ::ffff:172.21.15.122 - - [14/Apr/2024:13:57:29] "GET /metrics HTTP/1.1" 200 207229 "" "Prometheus/2.33.4" 2024-04-14T13:57:29.702 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:57:29 smithi069 ceph-mon[29472]: pgmap v87: 161 pgs: 4 peering, 157 active+clean; 585 KiB data, 56 MiB used, 715 GiB / 715 GiB avail; 108 KiB/s rd, 255 B/s wr, 173 op/s 2024-04-14T13:57:29.703 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:57:29 smithi069 ceph-mon[29472]: from='client.? 172.21.15.69:0/3123510337' entity='client.iscsi.foo.smithi069.yuumtr' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.69:0/3397430943"}]': finished 2024-04-14T13:57:29.703 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:57:29 smithi069 ceph-mon[29472]: osdmap e82: 8 total, 8 up, 8 in 2024-04-14T13:57:29.703 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:57:29 smithi069 ceph-mon[29472]: from='client.? 172.21.15.69:0/2127092894' entity='client.iscsi.foo.smithi069.yuumtr' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.69:0/226283202"}]: dispatch 2024-04-14T13:57:29.703 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:57:29 smithi069 ceph-mon[29472]: from='client.? ' entity='client.iscsi.foo.smithi069.yuumtr' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.69:0/226283202"}]: dispatch 2024-04-14T13:57:29.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:57:29 smithi122 ceph-mon[25721]: pgmap v87: 161 pgs: 4 peering, 157 active+clean; 585 KiB data, 56 MiB used, 715 GiB / 715 GiB avail; 108 KiB/s rd, 255 B/s wr, 173 op/s 2024-04-14T13:57:29.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:57:29 smithi122 ceph-mon[25721]: from='client.? 172.21.15.69:0/3123510337' entity='client.iscsi.foo.smithi069.yuumtr' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.69:0/3397430943"}]': finished 2024-04-14T13:57:29.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:57:29 smithi122 ceph-mon[25721]: osdmap e82: 8 total, 8 up, 8 in 2024-04-14T13:57:29.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:57:29 smithi122 ceph-mon[25721]: from='client.? 172.21.15.69:0/2127092894' entity='client.iscsi.foo.smithi069.yuumtr' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.69:0/226283202"}]: dispatch 2024-04-14T13:57:29.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:57:29 smithi122 ceph-mon[25721]: from='client.? ' entity='client.iscsi.foo.smithi069.yuumtr' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.69:0/226283202"}]: dispatch 2024-04-14T13:57:30.817 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:57:30 smithi069 ceph-mon[24818]: from='client.? ' entity='client.iscsi.foo.smithi069.yuumtr' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.69:0/226283202"}]': finished 2024-04-14T13:57:30.817 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:57:30 smithi069 ceph-mon[24818]: osdmap e83: 8 total, 8 up, 8 in 2024-04-14T13:57:30.817 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:57:30 smithi069 ceph-mon[24818]: from='client.? 172.21.15.69:0/1778724856' entity='client.iscsi.foo.smithi069.yuumtr' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.69:6801/2212179823"}]: dispatch 2024-04-14T13:57:30.817 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:57:30 smithi069 ceph-mon[29472]: from='client.? ' entity='client.iscsi.foo.smithi069.yuumtr' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.69:0/226283202"}]': finished 2024-04-14T13:57:30.817 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:57:30 smithi069 ceph-mon[29472]: osdmap e83: 8 total, 8 up, 8 in 2024-04-14T13:57:30.817 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:57:30 smithi069 ceph-mon[29472]: from='client.? 172.21.15.69:0/1778724856' entity='client.iscsi.foo.smithi069.yuumtr' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.69:6801/2212179823"}]: dispatch 2024-04-14T13:57:30.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:57:30 smithi122 ceph-mon[25721]: from='client.? ' entity='client.iscsi.foo.smithi069.yuumtr' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.69:0/226283202"}]': finished 2024-04-14T13:57:30.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:57:30 smithi122 ceph-mon[25721]: osdmap e83: 8 total, 8 up, 8 in 2024-04-14T13:57:30.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:57:30 smithi122 ceph-mon[25721]: from='client.? 172.21.15.69:0/1778724856' entity='client.iscsi.foo.smithi069.yuumtr' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.69:6801/2212179823"}]: dispatch 2024-04-14T13:57:31.817 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:57:31 smithi069 ceph-mon[24818]: pgmap v90: 161 pgs: 161 active+clean; 585 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 12 KiB/s rd, 511 B/s wr, 12 op/s 2024-04-14T13:57:31.817 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:57:31 smithi069 ceph-mon[24818]: from='client.? 172.21.15.69:0/1778724856' entity='client.iscsi.foo.smithi069.yuumtr' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.69:6801/2212179823"}]': finished 2024-04-14T13:57:31.817 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:57:31 smithi069 ceph-mon[24818]: osdmap e84: 8 total, 8 up, 8 in 2024-04-14T13:57:31.817 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:57:31 smithi069 ceph-mon[24818]: from='mgr.14448 172.21.15.69:0/453473650' entity='mgr.y' 2024-04-14T13:57:31.817 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:57:31 smithi069 ceph-mon[24818]: from='mgr.14448 172.21.15.69:0/453473650' entity='mgr.y' 2024-04-14T13:57:31.817 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:57:31 smithi069 ceph-mon[24818]: from='client.? 172.21.15.69:0/636586607' entity='client.iscsi.foo.smithi069.yuumtr' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.69:0/2434136662"}]: dispatch 2024-04-14T13:57:31.817 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:57:31 smithi069 ceph-mon[24818]: from='mgr.14448 172.21.15.69:0/453473650' entity='mgr.y' 2024-04-14T13:57:31.818 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:57:31 smithi069 ceph-mon[29472]: pgmap v90: 161 pgs: 161 active+clean; 585 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 12 KiB/s rd, 511 B/s wr, 12 op/s 2024-04-14T13:57:31.818 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:57:31 smithi069 ceph-mon[29472]: from='client.? 172.21.15.69:0/1778724856' entity='client.iscsi.foo.smithi069.yuumtr' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.69:6801/2212179823"}]': finished 2024-04-14T13:57:31.818 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:57:31 smithi069 ceph-mon[29472]: osdmap e84: 8 total, 8 up, 8 in 2024-04-14T13:57:31.818 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:57:31 smithi069 ceph-mon[29472]: from='mgr.14448 172.21.15.69:0/453473650' entity='mgr.y' 2024-04-14T13:57:31.818 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:57:31 smithi069 ceph-mon[29472]: from='mgr.14448 172.21.15.69:0/453473650' entity='mgr.y' 2024-04-14T13:57:31.818 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:57:31 smithi069 ceph-mon[29472]: from='client.? 172.21.15.69:0/636586607' entity='client.iscsi.foo.smithi069.yuumtr' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.69:0/2434136662"}]: dispatch 2024-04-14T13:57:31.818 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:57:31 smithi069 ceph-mon[29472]: from='mgr.14448 172.21.15.69:0/453473650' entity='mgr.y' 2024-04-14T13:57:31.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:57:31 smithi122 ceph-mon[25721]: pgmap v90: 161 pgs: 161 active+clean; 585 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 12 KiB/s rd, 511 B/s wr, 12 op/s 2024-04-14T13:57:31.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:57:31 smithi122 ceph-mon[25721]: from='client.? 172.21.15.69:0/1778724856' entity='client.iscsi.foo.smithi069.yuumtr' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.69:6801/2212179823"}]': finished 2024-04-14T13:57:31.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:57:31 smithi122 ceph-mon[25721]: osdmap e84: 8 total, 8 up, 8 in 2024-04-14T13:57:31.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:57:31 smithi122 ceph-mon[25721]: from='mgr.14448 172.21.15.69:0/453473650' entity='mgr.y' 2024-04-14T13:57:31.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:57:31 smithi122 ceph-mon[25721]: from='mgr.14448 172.21.15.69:0/453473650' entity='mgr.y' 2024-04-14T13:57:31.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:57:31 smithi122 ceph-mon[25721]: from='client.? 172.21.15.69:0/636586607' entity='client.iscsi.foo.smithi069.yuumtr' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.69:0/2434136662"}]: dispatch 2024-04-14T13:57:31.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:57:31 smithi122 ceph-mon[25721]: from='mgr.14448 172.21.15.69:0/453473650' entity='mgr.y' 2024-04-14T13:57:32.724 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:57:32 smithi069 ceph-mon[24818]: Checking dashboard <-> RGW credentials 2024-04-14T13:57:32.724 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:57:32 smithi069 ceph-mon[24818]: from='client.? 172.21.15.69:0/636586607' entity='client.iscsi.foo.smithi069.yuumtr' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.69:0/2434136662"}]': finished 2024-04-14T13:57:32.724 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:57:32 smithi069 ceph-mon[24818]: osdmap e85: 8 total, 8 up, 8 in 2024-04-14T13:57:32.724 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:57:32 smithi069 ceph-mon[24818]: from='client.? 172.21.15.69:0/2433367796' entity='client.iscsi.foo.smithi069.yuumtr' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.69:6800/2748095016"}]: dispatch 2024-04-14T13:57:32.724 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:57:32 smithi069 ceph-mon[29472]: Checking dashboard <-> RGW credentials 2024-04-14T13:57:32.724 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:57:32 smithi069 ceph-mon[29472]: from='client.? 172.21.15.69:0/636586607' entity='client.iscsi.foo.smithi069.yuumtr' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.69:0/2434136662"}]': finished 2024-04-14T13:57:32.724 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:57:32 smithi069 ceph-mon[29472]: osdmap e85: 8 total, 8 up, 8 in 2024-04-14T13:57:32.725 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:57:32 smithi069 ceph-mon[29472]: from='client.? 172.21.15.69:0/2433367796' entity='client.iscsi.foo.smithi069.yuumtr' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.69:6800/2748095016"}]: dispatch 2024-04-14T13:57:32.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:57:32 smithi122 ceph-mon[25721]: Checking dashboard <-> RGW credentials 2024-04-14T13:57:32.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:57:32 smithi122 ceph-mon[25721]: from='client.? 172.21.15.69:0/636586607' entity='client.iscsi.foo.smithi069.yuumtr' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.69:0/2434136662"}]': finished 2024-04-14T13:57:32.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:57:32 smithi122 ceph-mon[25721]: osdmap e85: 8 total, 8 up, 8 in 2024-04-14T13:57:32.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:57:32 smithi122 ceph-mon[25721]: from='client.? 172.21.15.69:0/2433367796' entity='client.iscsi.foo.smithi069.yuumtr' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.69:6800/2748095016"}]: dispatch 2024-04-14T13:57:33.726 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:57:33 smithi069 ceph-mon[24818]: pgmap v93: 161 pgs: 161 active+clean; 585 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 12 KiB/s rd, 511 B/s wr, 12 op/s 2024-04-14T13:57:33.726 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:57:33 smithi069 ceph-mon[24818]: from='client.? 172.21.15.69:0/2433367796' entity='client.iscsi.foo.smithi069.yuumtr' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.69:6800/2748095016"}]': finished 2024-04-14T13:57:33.726 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:57:33 smithi069 ceph-mon[24818]: osdmap e86: 8 total, 8 up, 8 in 2024-04-14T13:57:33.726 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:57:33 smithi069 ceph-mon[24818]: from='client.? 172.21.15.69:0/2805143198' entity='client.iscsi.foo.smithi069.yuumtr' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.69:6801/3528824566"}]: dispatch 2024-04-14T13:57:33.726 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:57:33 smithi069 ceph-mon[24818]: from='client.? ' entity='client.iscsi.foo.smithi069.yuumtr' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.69:6801/3528824566"}]: dispatch 2024-04-14T13:57:33.727 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:57:33 smithi069 ceph-mon[29472]: pgmap v93: 161 pgs: 161 active+clean; 585 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 12 KiB/s rd, 511 B/s wr, 12 op/s 2024-04-14T13:57:33.727 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:57:33 smithi069 ceph-mon[29472]: from='client.? 172.21.15.69:0/2433367796' entity='client.iscsi.foo.smithi069.yuumtr' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.69:6800/2748095016"}]': finished 2024-04-14T13:57:33.727 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:57:33 smithi069 ceph-mon[29472]: osdmap e86: 8 total, 8 up, 8 in 2024-04-14T13:57:33.727 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:57:33 smithi069 ceph-mon[29472]: from='client.? 172.21.15.69:0/2805143198' entity='client.iscsi.foo.smithi069.yuumtr' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.69:6801/3528824566"}]: dispatch 2024-04-14T13:57:33.727 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:57:33 smithi069 ceph-mon[29472]: from='client.? ' entity='client.iscsi.foo.smithi069.yuumtr' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.69:6801/3528824566"}]: dispatch 2024-04-14T13:57:33.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:57:33 smithi122 ceph-mon[25721]: pgmap v93: 161 pgs: 161 active+clean; 585 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 12 KiB/s rd, 511 B/s wr, 12 op/s 2024-04-14T13:57:33.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:57:33 smithi122 ceph-mon[25721]: from='client.? 172.21.15.69:0/2433367796' entity='client.iscsi.foo.smithi069.yuumtr' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.69:6800/2748095016"}]': finished 2024-04-14T13:57:33.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:57:33 smithi122 ceph-mon[25721]: osdmap e86: 8 total, 8 up, 8 in 2024-04-14T13:57:33.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:57:33 smithi122 ceph-mon[25721]: from='client.? 172.21.15.69:0/2805143198' entity='client.iscsi.foo.smithi069.yuumtr' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.69:6801/3528824566"}]: dispatch 2024-04-14T13:57:33.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:57:33 smithi122 ceph-mon[25721]: from='client.? ' entity='client.iscsi.foo.smithi069.yuumtr' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.69:6801/3528824566"}]: dispatch 2024-04-14T13:57:35.067 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:57:34 smithi069 ceph-mon[24818]: from='client.14766 -' entity='client.iscsi.foo.smithi069.yuumtr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T13:57:35.067 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:57:34 smithi069 ceph-mon[24818]: from='client.? ' entity='client.iscsi.foo.smithi069.yuumtr' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.69:6801/3528824566"}]': finished 2024-04-14T13:57:35.067 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:57:34 smithi069 ceph-mon[24818]: osdmap e87: 8 total, 8 up, 8 in 2024-04-14T13:57:35.067 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:57:34 smithi069 ceph-mon[24818]: from='client.? 172.21.15.69:0/2784372406' entity='client.iscsi.foo.smithi069.yuumtr' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.69:6800/3528824566"}]: dispatch 2024-04-14T13:57:35.067 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:57:34 smithi069 ceph-mon[24818]: from='client.? ' entity='client.iscsi.foo.smithi069.yuumtr' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.69:6800/3528824566"}]: dispatch 2024-04-14T13:57:35.067 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:57:34 smithi069 ceph-mon[24818]: pgmap v96: 161 pgs: 161 active+clean; 585 KiB data, 61 MiB used, 715 GiB / 715 GiB avail 2024-04-14T13:57:35.068 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:57:34 smithi069 ceph-mon[29472]: from='client.14766 -' entity='client.iscsi.foo.smithi069.yuumtr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T13:57:35.068 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:57:34 smithi069 ceph-mon[29472]: from='client.? ' entity='client.iscsi.foo.smithi069.yuumtr' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.69:6801/3528824566"}]': finished 2024-04-14T13:57:35.068 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:57:34 smithi069 ceph-mon[29472]: osdmap e87: 8 total, 8 up, 8 in 2024-04-14T13:57:35.068 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:57:34 smithi069 ceph-mon[29472]: from='client.? 172.21.15.69:0/2784372406' entity='client.iscsi.foo.smithi069.yuumtr' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.69:6800/3528824566"}]: dispatch 2024-04-14T13:57:35.068 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:57:34 smithi069 ceph-mon[29472]: from='client.? ' entity='client.iscsi.foo.smithi069.yuumtr' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.69:6800/3528824566"}]: dispatch 2024-04-14T13:57:35.068 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:57:34 smithi069 ceph-mon[29472]: pgmap v96: 161 pgs: 161 active+clean; 585 KiB data, 61 MiB used, 715 GiB / 715 GiB avail 2024-04-14T13:57:35.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:57:34 smithi122 ceph-mon[25721]: from='client.14766 -' entity='client.iscsi.foo.smithi069.yuumtr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T13:57:35.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:57:34 smithi122 ceph-mon[25721]: from='client.? ' entity='client.iscsi.foo.smithi069.yuumtr' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.69:6801/3528824566"}]': finished 2024-04-14T13:57:35.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:57:34 smithi122 ceph-mon[25721]: osdmap e87: 8 total, 8 up, 8 in 2024-04-14T13:57:35.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:57:34 smithi122 ceph-mon[25721]: from='client.? 172.21.15.69:0/2784372406' entity='client.iscsi.foo.smithi069.yuumtr' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.69:6800/3528824566"}]: dispatch 2024-04-14T13:57:35.096 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:57:34 smithi122 ceph-mon[25721]: from='client.? ' entity='client.iscsi.foo.smithi069.yuumtr' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.69:6800/3528824566"}]: dispatch 2024-04-14T13:57:35.096 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:57:34 smithi122 ceph-mon[25721]: pgmap v96: 161 pgs: 161 active+clean; 585 KiB data, 61 MiB used, 715 GiB / 715 GiB avail 2024-04-14T13:57:36.067 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:57:35 smithi069 ceph-mon[24818]: from='client.? ' entity='client.iscsi.foo.smithi069.yuumtr' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.69:6800/3528824566"}]': finished 2024-04-14T13:57:36.067 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:57:35 smithi069 ceph-mon[24818]: osdmap e88: 8 total, 8 up, 8 in 2024-04-14T13:57:36.067 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:57:35 smithi069 ceph-mon[24818]: from='client.? 172.21.15.69:0/1039551467' entity='client.iscsi.foo.smithi069.yuumtr' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.69:6801/2748095016"}]: dispatch 2024-04-14T13:57:36.067 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:57:35 smithi069 ceph-mon[24818]: from='client.? ' entity='client.iscsi.foo.smithi069.yuumtr' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.69:6801/2748095016"}]: dispatch 2024-04-14T13:57:36.067 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:57:35 smithi069 ceph-mon[29472]: from='client.? ' entity='client.iscsi.foo.smithi069.yuumtr' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.69:6800/3528824566"}]': finished 2024-04-14T13:57:36.067 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:57:35 smithi069 ceph-mon[29472]: osdmap e88: 8 total, 8 up, 8 in 2024-04-14T13:57:36.067 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:57:35 smithi069 ceph-mon[29472]: from='client.? 172.21.15.69:0/1039551467' entity='client.iscsi.foo.smithi069.yuumtr' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.69:6801/2748095016"}]: dispatch 2024-04-14T13:57:36.068 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:57:35 smithi069 ceph-mon[29472]: from='client.? ' entity='client.iscsi.foo.smithi069.yuumtr' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.69:6801/2748095016"}]: dispatch 2024-04-14T13:57:36.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:57:35 smithi122 ceph-mon[25721]: from='client.? ' entity='client.iscsi.foo.smithi069.yuumtr' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.69:6800/3528824566"}]': finished 2024-04-14T13:57:36.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:57:35 smithi122 ceph-mon[25721]: osdmap e88: 8 total, 8 up, 8 in 2024-04-14T13:57:36.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:57:35 smithi122 ceph-mon[25721]: from='client.? 172.21.15.69:0/1039551467' entity='client.iscsi.foo.smithi069.yuumtr' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.69:6801/2748095016"}]: dispatch 2024-04-14T13:57:36.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:57:35 smithi122 ceph-mon[25721]: from='client.? ' entity='client.iscsi.foo.smithi069.yuumtr' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.69:6801/2748095016"}]: dispatch 2024-04-14T13:57:37.067 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:57:36 smithi069 ceph-mon[24818]: from='client.? ' entity='client.iscsi.foo.smithi069.yuumtr' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.69:6801/2748095016"}]': finished 2024-04-14T13:57:37.067 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:57:36 smithi069 ceph-mon[24818]: osdmap e89: 8 total, 8 up, 8 in 2024-04-14T13:57:37.067 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:57:36 smithi069 ceph-mon[24818]: from='client.? 172.21.15.69:0/2335374325' entity='client.iscsi.foo.smithi069.yuumtr' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.69:0/3759166769"}]: dispatch 2024-04-14T13:57:37.067 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:57:36 smithi069 ceph-mon[24818]: pgmap v99: 161 pgs: 161 active+clean; 585 KiB data, 62 MiB used, 715 GiB / 715 GiB avail; 9.2 KiB/s rd, 9 op/s 2024-04-14T13:57:37.067 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:57:36 smithi069 ceph-mon[29472]: from='client.? ' entity='client.iscsi.foo.smithi069.yuumtr' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.69:6801/2748095016"}]': finished 2024-04-14T13:57:37.068 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:57:36 smithi069 ceph-mon[29472]: osdmap e89: 8 total, 8 up, 8 in 2024-04-14T13:57:37.068 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:57:36 smithi069 ceph-mon[29472]: from='client.? 172.21.15.69:0/2335374325' entity='client.iscsi.foo.smithi069.yuumtr' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.69:0/3759166769"}]: dispatch 2024-04-14T13:57:37.068 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:57:36 smithi069 ceph-mon[29472]: pgmap v99: 161 pgs: 161 active+clean; 585 KiB data, 62 MiB used, 715 GiB / 715 GiB avail; 9.2 KiB/s rd, 9 op/s 2024-04-14T13:57:37.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:57:36 smithi122 ceph-mon[25721]: from='client.? ' entity='client.iscsi.foo.smithi069.yuumtr' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.69:6801/2748095016"}]': finished 2024-04-14T13:57:37.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:57:36 smithi122 ceph-mon[25721]: osdmap e89: 8 total, 8 up, 8 in 2024-04-14T13:57:37.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:57:36 smithi122 ceph-mon[25721]: from='client.? 172.21.15.69:0/2335374325' entity='client.iscsi.foo.smithi069.yuumtr' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.69:0/3759166769"}]: dispatch 2024-04-14T13:57:37.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:57:36 smithi122 ceph-mon[25721]: pgmap v99: 161 pgs: 161 active+clean; 585 KiB data, 62 MiB used, 715 GiB / 715 GiB avail; 9.2 KiB/s rd, 9 op/s 2024-04-14T13:57:37.595 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Apr 14 13:57:37 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-x[26913]: ::ffff:172.21.15.122 - - [14/Apr/2024:13:57:37] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-04-14T13:57:38.067 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:57:37 smithi069 ceph-mon[24818]: from='client.? 172.21.15.69:0/2335374325' entity='client.iscsi.foo.smithi069.yuumtr' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.69:0/3759166769"}]': finished 2024-04-14T13:57:38.067 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:57:37 smithi069 ceph-mon[24818]: osdmap e90: 8 total, 8 up, 8 in 2024-04-14T13:57:38.067 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:57:37 smithi069 ceph-mon[24818]: from='client.? 172.21.15.69:0/2977024749' entity='client.iscsi.foo.smithi069.yuumtr' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.69:0/2144826942"}]: dispatch 2024-04-14T13:57:38.067 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:57:37 smithi069 ceph-mon[24818]: from='client.? ' entity='client.iscsi.foo.smithi069.yuumtr' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.69:0/2144826942"}]: dispatch 2024-04-14T13:57:38.067 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:57:37 smithi069 ceph-mon[29472]: from='client.? 172.21.15.69:0/2335374325' entity='client.iscsi.foo.smithi069.yuumtr' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.69:0/3759166769"}]': finished 2024-04-14T13:57:38.067 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:57:37 smithi069 ceph-mon[29472]: osdmap e90: 8 total, 8 up, 8 in 2024-04-14T13:57:38.068 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:57:37 smithi069 ceph-mon[29472]: from='client.? 172.21.15.69:0/2977024749' entity='client.iscsi.foo.smithi069.yuumtr' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.69:0/2144826942"}]: dispatch 2024-04-14T13:57:38.068 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:57:37 smithi069 ceph-mon[29472]: from='client.? ' entity='client.iscsi.foo.smithi069.yuumtr' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.69:0/2144826942"}]: dispatch 2024-04-14T13:57:38.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:57:37 smithi122 ceph-mon[25721]: from='client.? 172.21.15.69:0/2335374325' entity='client.iscsi.foo.smithi069.yuumtr' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.69:0/3759166769"}]': finished 2024-04-14T13:57:38.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:57:37 smithi122 ceph-mon[25721]: osdmap e90: 8 total, 8 up, 8 in 2024-04-14T13:57:38.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:57:37 smithi122 ceph-mon[25721]: from='client.? 172.21.15.69:0/2977024749' entity='client.iscsi.foo.smithi069.yuumtr' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.69:0/2144826942"}]: dispatch 2024-04-14T13:57:38.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:57:37 smithi122 ceph-mon[25721]: from='client.? ' entity='client.iscsi.foo.smithi069.yuumtr' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.69:0/2144826942"}]: dispatch 2024-04-14T13:57:39.067 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:57:38 smithi069 ceph-mon[24818]: from='client.? ' entity='client.iscsi.foo.smithi069.yuumtr' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.69:0/2144826942"}]': finished 2024-04-14T13:57:39.067 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:57:38 smithi069 ceph-mon[24818]: osdmap e91: 8 total, 8 up, 8 in 2024-04-14T13:57:39.067 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:57:38 smithi069 ceph-mon[24818]: from='client.? 172.21.15.69:0/2517171435' entity='client.iscsi.foo.smithi069.yuumtr' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.69:0/3270155078"}]: dispatch 2024-04-14T13:57:39.067 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:57:38 smithi069 ceph-mon[24818]: from='client.? ' entity='client.iscsi.foo.smithi069.yuumtr' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.69:0/3270155078"}]: dispatch 2024-04-14T13:57:39.067 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:57:38 smithi069 ceph-mon[24818]: pgmap v102: 161 pgs: 161 active+clean; 585 KiB data, 62 MiB used, 715 GiB / 715 GiB avail; 9.2 KiB/s rd, 9 op/s 2024-04-14T13:57:39.067 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:57:38 smithi069 ceph-mon[29472]: from='client.? ' entity='client.iscsi.foo.smithi069.yuumtr' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.69:0/2144826942"}]': finished 2024-04-14T13:57:39.067 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:57:38 smithi069 ceph-mon[29472]: osdmap e91: 8 total, 8 up, 8 in 2024-04-14T13:57:39.068 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:57:38 smithi069 ceph-mon[29472]: from='client.? 172.21.15.69:0/2517171435' entity='client.iscsi.foo.smithi069.yuumtr' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.69:0/3270155078"}]: dispatch 2024-04-14T13:57:39.068 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:57:38 smithi069 ceph-mon[29472]: from='client.? ' entity='client.iscsi.foo.smithi069.yuumtr' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.69:0/3270155078"}]: dispatch 2024-04-14T13:57:39.068 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:57:38 smithi069 ceph-mon[29472]: pgmap v102: 161 pgs: 161 active+clean; 585 KiB data, 62 MiB used, 715 GiB / 715 GiB avail; 9.2 KiB/s rd, 9 op/s 2024-04-14T13:57:39.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:57:38 smithi122 ceph-mon[25721]: from='client.? ' entity='client.iscsi.foo.smithi069.yuumtr' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.69:0/2144826942"}]': finished 2024-04-14T13:57:39.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:57:38 smithi122 ceph-mon[25721]: osdmap e91: 8 total, 8 up, 8 in 2024-04-14T13:57:39.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:57:38 smithi122 ceph-mon[25721]: from='client.? 172.21.15.69:0/2517171435' entity='client.iscsi.foo.smithi069.yuumtr' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.69:0/3270155078"}]: dispatch 2024-04-14T13:57:39.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:57:38 smithi122 ceph-mon[25721]: from='client.? ' entity='client.iscsi.foo.smithi069.yuumtr' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.69:0/3270155078"}]: dispatch 2024-04-14T13:57:39.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:57:38 smithi122 ceph-mon[25721]: pgmap v102: 161 pgs: 161 active+clean; 585 KiB data, 62 MiB used, 715 GiB / 715 GiB avail; 9.2 KiB/s rd, 9 op/s 2024-04-14T13:57:39.817 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 14 13:57:39 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-y[25054]: ::ffff:172.21.15.122 - - [14/Apr/2024:13:57:39] "GET /metrics HTTP/1.1" 200 208121 "" "Prometheus/2.33.4" 2024-04-14T13:57:39.817 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:57:39 smithi069 ceph-mon[24818]: from='client.? ' entity='client.iscsi.foo.smithi069.yuumtr' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.69:0/3270155078"}]': finished 2024-04-14T13:57:39.817 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:57:39 smithi069 ceph-mon[24818]: osdmap e92: 8 total, 8 up, 8 in 2024-04-14T13:57:39.817 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:57:39 smithi069 ceph-mon[24818]: from='client.? 172.21.15.69:0/2897531798' entity='client.iscsi.foo.smithi069.yuumtr' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.69:0/1250951073"}]: dispatch 2024-04-14T13:57:39.817 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:57:39 smithi069 ceph-mon[24818]: from='client.? ' entity='client.iscsi.foo.smithi069.yuumtr' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.69:0/1250951073"}]: dispatch 2024-04-14T13:57:39.818 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:57:39 smithi069 ceph-mon[29472]: from='client.? ' entity='client.iscsi.foo.smithi069.yuumtr' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.69:0/3270155078"}]': finished 2024-04-14T13:57:39.818 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:57:39 smithi069 ceph-mon[29472]: osdmap e92: 8 total, 8 up, 8 in 2024-04-14T13:57:39.818 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:57:39 smithi069 ceph-mon[29472]: from='client.? 172.21.15.69:0/2897531798' entity='client.iscsi.foo.smithi069.yuumtr' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.69:0/1250951073"}]: dispatch 2024-04-14T13:57:39.818 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:57:39 smithi069 ceph-mon[29472]: from='client.? ' entity='client.iscsi.foo.smithi069.yuumtr' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.69:0/1250951073"}]: dispatch 2024-04-14T13:57:40.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:57:39 smithi122 ceph-mon[25721]: from='client.? ' entity='client.iscsi.foo.smithi069.yuumtr' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.69:0/3270155078"}]': finished 2024-04-14T13:57:40.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:57:39 smithi122 ceph-mon[25721]: osdmap e92: 8 total, 8 up, 8 in 2024-04-14T13:57:40.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:57:39 smithi122 ceph-mon[25721]: from='client.? 172.21.15.69:0/2897531798' entity='client.iscsi.foo.smithi069.yuumtr' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.69:0/1250951073"}]: dispatch 2024-04-14T13:57:40.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:57:39 smithi122 ceph-mon[25721]: from='client.? ' entity='client.iscsi.foo.smithi069.yuumtr' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.69:0/1250951073"}]: dispatch 2024-04-14T13:57:41.067 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:57:40 smithi069 ceph-mon[24818]: from='client.? ' entity='client.iscsi.foo.smithi069.yuumtr' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.69:0/1250951073"}]': finished 2024-04-14T13:57:41.067 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:57:40 smithi069 ceph-mon[24818]: osdmap e93: 8 total, 8 up, 8 in 2024-04-14T13:57:41.067 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:57:40 smithi069 ceph-mon[24818]: pgmap v105: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T13:57:41.067 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:57:40 smithi069 ceph-mon[29472]: from='client.? ' entity='client.iscsi.foo.smithi069.yuumtr' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.69:0/1250951073"}]': finished 2024-04-14T13:57:41.067 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:57:40 smithi069 ceph-mon[29472]: osdmap e93: 8 total, 8 up, 8 in 2024-04-14T13:57:41.068 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:57:40 smithi069 ceph-mon[29472]: pgmap v105: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T13:57:41.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:57:40 smithi122 ceph-mon[25721]: from='client.? ' entity='client.iscsi.foo.smithi069.yuumtr' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.69:0/1250951073"}]': finished 2024-04-14T13:57:41.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:57:40 smithi122 ceph-mon[25721]: osdmap e93: 8 total, 8 up, 8 in 2024-04-14T13:57:41.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:57:40 smithi122 ceph-mon[25721]: pgmap v105: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T13:57:43.771 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:57:43 smithi069 ceph-mon[24818]: pgmap v106: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 910 B/s rd, 0 op/s 2024-04-14T13:57:43.771 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:57:43 smithi069 ceph-mon[29472]: pgmap v106: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 910 B/s rd, 0 op/s 2024-04-14T13:57:43.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:57:43 smithi122 ceph-mon[25721]: pgmap v106: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 910 B/s rd, 0 op/s 2024-04-14T13:57:45.567 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:57:45 smithi069 ceph-mon[24818]: from='client.14766 -' entity='client.iscsi.foo.smithi069.yuumtr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T13:57:45.567 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:57:45 smithi069 ceph-mon[24818]: pgmap v107: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 773 B/s rd, 0 op/s 2024-04-14T13:57:45.568 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:57:45 smithi069 ceph-mon[29472]: from='client.14766 -' entity='client.iscsi.foo.smithi069.yuumtr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T13:57:45.568 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:57:45 smithi069 ceph-mon[29472]: pgmap v107: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 773 B/s rd, 0 op/s 2024-04-14T13:57:45.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:57:45 smithi122 ceph-mon[25721]: from='client.14766 -' entity='client.iscsi.foo.smithi069.yuumtr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T13:57:45.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:57:45 smithi122 ceph-mon[25721]: pgmap v107: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 773 B/s rd, 0 op/s 2024-04-14T13:57:47.595 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:57:47 smithi122 ceph-mon[25721]: pgmap v108: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T13:57:47.595 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Apr 14 13:57:47 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-x[26913]: ::ffff:172.21.15.122 - - [14/Apr/2024:13:57:47] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-04-14T13:57:47.817 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:57:47 smithi069 ceph-mon[24818]: pgmap v108: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T13:57:47.817 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:57:47 smithi069 ceph-mon[29472]: pgmap v108: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T13:57:49.817 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:57:49 smithi069 ceph-mon[24818]: pgmap v109: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-04-14T13:57:49.817 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 14 13:57:49 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-y[25054]: ::ffff:172.21.15.122 - - [14/Apr/2024:13:57:49] "GET /metrics HTTP/1.1" 200 208121 "" "Prometheus/2.33.4" 2024-04-14T13:57:49.817 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:57:49 smithi069 ceph-mon[29472]: pgmap v109: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-04-14T13:57:49.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:57:49 smithi122 ceph-mon[25721]: pgmap v109: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-04-14T13:57:51.567 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:57:51 smithi069 ceph-mon[24818]: pgmap v110: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s 2024-04-14T13:57:51.567 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:57:51 smithi069 ceph-mon[29472]: pgmap v110: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s 2024-04-14T13:57:51.595 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:57:51 smithi122 ceph-mon[25721]: pgmap v110: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s 2024-04-14T13:57:53.817 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:57:53 smithi069 ceph-mon[24818]: pgmap v111: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T13:57:53.818 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:57:53 smithi069 ceph-mon[29472]: pgmap v111: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T13:57:53.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:57:53 smithi122 ceph-mon[25721]: pgmap v111: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T13:57:55.817 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:57:55 smithi069 ceph-mon[24818]: from='client.14766 -' entity='client.iscsi.foo.smithi069.yuumtr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T13:57:55.817 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:57:55 smithi069 ceph-mon[24818]: pgmap v112: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T13:57:55.817 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:57:55 smithi069 ceph-mon[29472]: from='client.14766 -' entity='client.iscsi.foo.smithi069.yuumtr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T13:57:55.817 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:57:55 smithi069 ceph-mon[29472]: pgmap v112: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T13:57:55.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:57:55 smithi122 ceph-mon[25721]: from='client.14766 -' entity='client.iscsi.foo.smithi069.yuumtr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T13:57:55.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:57:55 smithi122 ceph-mon[25721]: pgmap v112: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T13:57:57.595 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:57:57 smithi122 ceph-mon[25721]: pgmap v113: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T13:57:57.595 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Apr 14 13:57:57 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-x[26913]: ::ffff:172.21.15.122 - - [14/Apr/2024:13:57:57] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-04-14T13:57:57.817 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:57:57 smithi069 ceph-mon[24818]: pgmap v113: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T13:57:57.817 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:57:57 smithi069 ceph-mon[29472]: pgmap v113: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T13:57:59.817 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:57:59 smithi069 ceph-mon[24818]: pgmap v114: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T13:57:59.817 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 14 13:57:59 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-y[25054]: ::ffff:172.21.15.122 - - [14/Apr/2024:13:57:59] "GET /metrics HTTP/1.1" 200 208086 "" "Prometheus/2.33.4" 2024-04-14T13:57:59.817 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:57:59 smithi069 ceph-mon[29472]: pgmap v114: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T13:57:59.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:57:59 smithi122 ceph-mon[25721]: pgmap v114: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T13:58:01.567 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:58:01 smithi069 ceph-mon[24818]: pgmap v115: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T13:58:01.567 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:58:01 smithi069 ceph-mon[29472]: pgmap v115: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T13:58:01.595 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:58:01 smithi122 ceph-mon[25721]: pgmap v115: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T13:58:03.817 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:58:03 smithi069 ceph-mon[29472]: pgmap v116: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T13:58:03.817 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:58:03 smithi069 ceph-mon[24818]: pgmap v116: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T13:58:03.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:58:03 smithi122 ceph-mon[25721]: pgmap v116: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T13:58:05.817 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:58:05 smithi069 ceph-mon[24818]: from='client.14766 -' entity='client.iscsi.foo.smithi069.yuumtr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T13:58:05.817 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:58:05 smithi069 ceph-mon[24818]: pgmap v117: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T13:58:05.817 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:58:05 smithi069 ceph-mon[29472]: from='client.14766 -' entity='client.iscsi.foo.smithi069.yuumtr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T13:58:05.817 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:58:05 smithi069 ceph-mon[29472]: pgmap v117: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T13:58:05.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:58:05 smithi122 ceph-mon[25721]: from='client.14766 -' entity='client.iscsi.foo.smithi069.yuumtr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T13:58:05.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:58:05 smithi122 ceph-mon[25721]: pgmap v117: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T13:58:07.595 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:58:07 smithi122 ceph-mon[25721]: pgmap v118: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T13:58:07.595 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Apr 14 13:58:07 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-x[26913]: ::ffff:172.21.15.122 - - [14/Apr/2024:13:58:07] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-04-14T13:58:07.817 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:58:07 smithi069 ceph-mon[29472]: pgmap v118: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T13:58:07.817 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:58:07 smithi069 ceph-mon[24818]: pgmap v118: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T13:58:09.817 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:58:09 smithi069 ceph-mon[24818]: pgmap v119: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T13:58:09.817 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 14 13:58:09 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-y[25054]: ::ffff:172.21.15.122 - - [14/Apr/2024:13:58:09] "GET /metrics HTTP/1.1" 200 208044 "" "Prometheus/2.33.4" 2024-04-14T13:58:09.817 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:58:09 smithi069 ceph-mon[29472]: pgmap v119: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T13:58:09.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:58:09 smithi122 ceph-mon[25721]: pgmap v119: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T13:58:11.567 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:58:11 smithi069 ceph-mon[24818]: pgmap v120: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T13:58:11.567 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:58:11 smithi069 ceph-mon[29472]: pgmap v120: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T13:58:11.595 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:58:11 smithi122 ceph-mon[25721]: pgmap v120: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T13:58:13.817 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:58:13 smithi069 ceph-mon[24818]: pgmap v121: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T13:58:13.817 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:58:13 smithi069 ceph-mon[29472]: pgmap v121: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T13:58:13.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:58:13 smithi122 ceph-mon[25721]: pgmap v121: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T13:58:15.817 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:58:15 smithi069 ceph-mon[24818]: from='client.14766 -' entity='client.iscsi.foo.smithi069.yuumtr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T13:58:15.817 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:58:15 smithi069 ceph-mon[24818]: pgmap v122: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T13:58:15.817 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:58:15 smithi069 ceph-mon[29472]: from='client.14766 -' entity='client.iscsi.foo.smithi069.yuumtr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T13:58:15.817 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:58:15 smithi069 ceph-mon[29472]: pgmap v122: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T13:58:15.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:58:15 smithi122 ceph-mon[25721]: from='client.14766 -' entity='client.iscsi.foo.smithi069.yuumtr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T13:58:15.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:58:15 smithi122 ceph-mon[25721]: pgmap v122: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T13:58:17.595 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:58:17 smithi122 ceph-mon[25721]: pgmap v123: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T13:58:17.595 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Apr 14 13:58:17 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-x[26913]: ::ffff:172.21.15.122 - - [14/Apr/2024:13:58:17] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-04-14T13:58:17.817 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:58:17 smithi069 ceph-mon[24818]: pgmap v123: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T13:58:17.817 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:58:17 smithi069 ceph-mon[29472]: pgmap v123: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T13:58:19.818 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:58:19 smithi069 ceph-mon[29472]: pgmap v124: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T13:58:19.819 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:58:19 smithi069 ceph-mon[24818]: pgmap v124: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T13:58:19.819 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 14 13:58:19 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-y[25054]: ::ffff:172.21.15.122 - - [14/Apr/2024:13:58:19] "GET /metrics HTTP/1.1" 200 208044 "" "Prometheus/2.33.4" 2024-04-14T13:58:19.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:58:19 smithi122 ceph-mon[25721]: pgmap v124: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T13:58:21.595 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:58:21 smithi122 ceph-mon[25721]: pgmap v125: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T13:58:21.817 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:58:21 smithi069 ceph-mon[24818]: pgmap v125: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T13:58:21.817 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:58:21 smithi069 ceph-mon[29472]: pgmap v125: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T13:58:23.817 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:58:23 smithi069 ceph-mon[29472]: pgmap v126: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T13:58:23.817 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:58:23 smithi069 ceph-mon[24818]: pgmap v126: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T13:58:23.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:58:23 smithi122 ceph-mon[25721]: pgmap v126: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T13:58:24.817 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:58:24 smithi069 ceph-mon[29472]: from='mgr.14448 172.21.15.69:0/453473650' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-04-14T13:58:24.817 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:58:24 smithi069 ceph-mon[29472]: from='mgr.14448 172.21.15.69:0/453473650' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-04-14T13:58:24.817 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:58:24 smithi069 ceph-mon[24818]: from='mgr.14448 172.21.15.69:0/453473650' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-04-14T13:58:24.817 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:58:24 smithi069 ceph-mon[24818]: from='mgr.14448 172.21.15.69:0/453473650' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-04-14T13:58:24.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:58:24 smithi122 ceph-mon[25721]: from='mgr.14448 172.21.15.69:0/453473650' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-04-14T13:58:24.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:58:24 smithi122 ceph-mon[25721]: from='mgr.14448 172.21.15.69:0/453473650' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-04-14T13:58:25.817 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:58:25 smithi069 ceph-mon[29472]: from='client.14766 -' entity='client.iscsi.foo.smithi069.yuumtr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T13:58:25.818 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:58:25 smithi069 ceph-mon[29472]: pgmap v127: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T13:58:25.818 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:58:25 smithi069 ceph-mon[24818]: from='client.14766 -' entity='client.iscsi.foo.smithi069.yuumtr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T13:58:25.818 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:58:25 smithi069 ceph-mon[24818]: pgmap v127: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T13:58:25.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:58:25 smithi122 ceph-mon[25721]: from='client.14766 -' entity='client.iscsi.foo.smithi069.yuumtr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T13:58:25.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:58:25 smithi122 ceph-mon[25721]: pgmap v127: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T13:58:27.595 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:58:27 smithi122 ceph-mon[25721]: pgmap v128: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T13:58:27.595 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Apr 14 13:58:27 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-x[26913]: ::ffff:172.21.15.122 - - [14/Apr/2024:13:58:27] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-04-14T13:58:27.817 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:58:27 smithi069 ceph-mon[24818]: pgmap v128: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T13:58:27.817 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:58:27 smithi069 ceph-mon[29472]: pgmap v128: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T13:58:29.817 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:58:29 smithi069 ceph-mon[24818]: pgmap v129: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T13:58:29.817 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 14 13:58:29 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-y[25054]: ::ffff:172.21.15.122 - - [14/Apr/2024:13:58:29] "GET /metrics HTTP/1.1" 200 208019 "" "Prometheus/2.33.4" 2024-04-14T13:58:29.817 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:58:29 smithi069 ceph-mon[29472]: pgmap v129: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T13:58:29.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:58:29 smithi122 ceph-mon[25721]: pgmap v129: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T13:58:31.568 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:58:31 smithi069 ceph-mon[29472]: pgmap v130: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T13:58:31.568 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:58:31 smithi069 ceph-mon[29472]: from='mgr.14448 172.21.15.69:0/453473650' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T13:58:31.568 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:58:31 smithi069 ceph-mon[29472]: from='mgr.14448 172.21.15.69:0/453473650' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-14T13:58:31.568 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:58:31 smithi069 ceph-mon[29472]: from='mgr.14448 172.21.15.69:0/453473650' entity='mgr.y' 2024-04-14T13:58:31.569 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:58:31 smithi069 ceph-mon[24818]: pgmap v130: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T13:58:31.569 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:58:31 smithi069 ceph-mon[24818]: from='mgr.14448 172.21.15.69:0/453473650' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T13:58:31.569 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:58:31 smithi069 ceph-mon[24818]: from='mgr.14448 172.21.15.69:0/453473650' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-14T13:58:31.569 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:58:31 smithi069 ceph-mon[24818]: from='mgr.14448 172.21.15.69:0/453473650' entity='mgr.y' 2024-04-14T13:58:31.595 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:58:31 smithi122 ceph-mon[25721]: pgmap v130: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T13:58:31.595 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:58:31 smithi122 ceph-mon[25721]: from='mgr.14448 172.21.15.69:0/453473650' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T13:58:31.595 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:58:31 smithi122 ceph-mon[25721]: from='mgr.14448 172.21.15.69:0/453473650' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-14T13:58:31.595 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:58:31 smithi122 ceph-mon[25721]: from='mgr.14448 172.21.15.69:0/453473650' entity='mgr.y' 2024-04-14T13:58:33.817 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:58:33 smithi069 ceph-mon[29472]: pgmap v131: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T13:58:33.817 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:58:33 smithi069 ceph-mon[24818]: pgmap v131: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T13:58:33.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:58:33 smithi122 ceph-mon[25721]: pgmap v131: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T13:58:35.817 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:58:35 smithi069 ceph-mon[24818]: from='client.14766 -' entity='client.iscsi.foo.smithi069.yuumtr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T13:58:35.818 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:58:35 smithi069 ceph-mon[24818]: pgmap v132: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T13:58:35.818 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:58:35 smithi069 ceph-mon[29472]: from='client.14766 -' entity='client.iscsi.foo.smithi069.yuumtr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T13:58:35.818 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:58:35 smithi069 ceph-mon[29472]: pgmap v132: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T13:58:35.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:58:35 smithi122 ceph-mon[25721]: from='client.14766 -' entity='client.iscsi.foo.smithi069.yuumtr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T13:58:35.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:58:35 smithi122 ceph-mon[25721]: pgmap v132: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T13:58:37.595 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:58:37 smithi122 ceph-mon[25721]: pgmap v133: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T13:58:37.596 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Apr 14 13:58:37 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-x[26913]: ::ffff:172.21.15.122 - - [14/Apr/2024:13:58:37] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-04-14T13:58:37.817 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:58:37 smithi069 ceph-mon[24818]: pgmap v133: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T13:58:37.817 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:58:37 smithi069 ceph-mon[29472]: pgmap v133: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T13:58:39.817 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:58:39 smithi069 ceph-mon[24818]: pgmap v134: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T13:58:39.817 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 14 13:58:39 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-y[25054]: ::ffff:172.21.15.122 - - [14/Apr/2024:13:58:39] "GET /metrics HTTP/1.1" 200 208029 "" "Prometheus/2.33.4" 2024-04-14T13:58:39.818 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:58:39 smithi069 ceph-mon[29472]: pgmap v134: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T13:58:39.844 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:58:39 smithi122 ceph-mon[25721]: pgmap v134: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T13:58:41.595 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:58:41 smithi122 ceph-mon[25721]: pgmap v135: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T13:58:41.817 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:58:41 smithi069 ceph-mon[24818]: pgmap v135: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T13:58:41.817 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:58:41 smithi069 ceph-mon[29472]: pgmap v135: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T13:58:43.817 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:58:43 smithi069 ceph-mon[24818]: pgmap v136: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T13:58:43.817 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:58:43 smithi069 ceph-mon[29472]: pgmap v136: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T13:58:43.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:58:43 smithi122 ceph-mon[25721]: pgmap v136: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T13:58:45.817 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:58:45 smithi069 ceph-mon[24818]: from='client.14766 -' entity='client.iscsi.foo.smithi069.yuumtr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T13:58:45.817 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:58:45 smithi069 ceph-mon[24818]: pgmap v137: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T13:58:45.817 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:58:45 smithi069 ceph-mon[29472]: from='client.14766 -' entity='client.iscsi.foo.smithi069.yuumtr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T13:58:45.817 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:58:45 smithi069 ceph-mon[29472]: pgmap v137: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T13:58:45.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:58:45 smithi122 ceph-mon[25721]: from='client.14766 -' entity='client.iscsi.foo.smithi069.yuumtr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T13:58:45.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:58:45 smithi122 ceph-mon[25721]: pgmap v137: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T13:58:47.595 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:58:47 smithi122 ceph-mon[25721]: pgmap v138: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T13:58:47.595 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Apr 14 13:58:47 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-x[26913]: ::ffff:172.21.15.122 - - [14/Apr/2024:13:58:47] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-04-14T13:58:47.817 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:58:47 smithi069 ceph-mon[24818]: pgmap v138: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T13:58:47.817 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:58:47 smithi069 ceph-mon[29472]: pgmap v138: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T13:58:49.817 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:58:49 smithi069 ceph-mon[24818]: pgmap v139: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T13:58:49.817 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 14 13:58:49 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-y[25054]: ::ffff:172.21.15.122 - - [14/Apr/2024:13:58:49] "GET /metrics HTTP/1.1" 200 208029 "" "Prometheus/2.33.4" 2024-04-14T13:58:49.817 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:58:49 smithi069 ceph-mon[29472]: pgmap v139: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T13:58:49.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:58:49 smithi122 ceph-mon[25721]: pgmap v139: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T13:58:51.595 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:58:51 smithi122 ceph-mon[25721]: pgmap v140: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T13:58:51.817 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:58:51 smithi069 ceph-mon[24818]: pgmap v140: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T13:58:51.817 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:58:51 smithi069 ceph-mon[29472]: pgmap v140: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T13:58:53.817 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:58:53 smithi069 ceph-mon[24818]: pgmap v141: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T13:58:53.817 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:58:53 smithi069 ceph-mon[29472]: pgmap v141: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T13:58:53.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:58:53 smithi122 ceph-mon[25721]: pgmap v141: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T13:58:55.817 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:58:55 smithi069 ceph-mon[24818]: from='client.14766 -' entity='client.iscsi.foo.smithi069.yuumtr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T13:58:55.817 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:58:55 smithi069 ceph-mon[24818]: pgmap v142: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T13:58:55.817 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:58:55 smithi069 ceph-mon[29472]: from='client.14766 -' entity='client.iscsi.foo.smithi069.yuumtr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T13:58:55.818 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:58:55 smithi069 ceph-mon[29472]: pgmap v142: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T13:58:55.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:58:55 smithi122 ceph-mon[25721]: from='client.14766 -' entity='client.iscsi.foo.smithi069.yuumtr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T13:58:55.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:58:55 smithi122 ceph-mon[25721]: pgmap v142: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T13:58:57.595 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:58:57 smithi122 ceph-mon[25721]: pgmap v143: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T13:58:57.595 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Apr 14 13:58:57 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-x[26913]: ::ffff:172.21.15.122 - - [14/Apr/2024:13:58:57] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-04-14T13:58:57.817 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:58:57 smithi069 ceph-mon[24818]: pgmap v143: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T13:58:57.817 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:58:57 smithi069 ceph-mon[29472]: pgmap v143: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T13:58:59.817 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:58:59 smithi069 ceph-mon[24818]: pgmap v144: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T13:58:59.817 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 14 13:58:59 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-y[25054]: ::ffff:172.21.15.122 - - [14/Apr/2024:13:58:59] "GET /metrics HTTP/1.1" 200 208036 "" "Prometheus/2.33.4" 2024-04-14T13:58:59.817 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:58:59 smithi069 ceph-mon[29472]: pgmap v144: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T13:58:59.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:58:59 smithi122 ceph-mon[25721]: pgmap v144: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T13:59:01.595 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:59:01 smithi122 ceph-mon[25721]: pgmap v145: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T13:59:01.817 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:59:01 smithi069 ceph-mon[24818]: pgmap v145: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T13:59:01.817 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:59:01 smithi069 ceph-mon[29472]: pgmap v145: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T13:59:03.817 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:59:03 smithi069 ceph-mon[24818]: pgmap v146: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T13:59:03.817 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:59:03 smithi069 ceph-mon[29472]: pgmap v146: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T13:59:03.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:59:03 smithi122 ceph-mon[25721]: pgmap v146: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T13:59:05.817 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:59:05 smithi069 ceph-mon[24818]: from='client.14766 -' entity='client.iscsi.foo.smithi069.yuumtr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T13:59:05.817 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:59:05 smithi069 ceph-mon[24818]: pgmap v147: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T13:59:05.817 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:59:05 smithi069 ceph-mon[29472]: from='client.14766 -' entity='client.iscsi.foo.smithi069.yuumtr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T13:59:05.817 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:59:05 smithi069 ceph-mon[29472]: pgmap v147: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T13:59:05.846 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:59:05 smithi122 ceph-mon[25721]: from='client.14766 -' entity='client.iscsi.foo.smithi069.yuumtr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T13:59:05.846 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:59:05 smithi122 ceph-mon[25721]: pgmap v147: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T13:59:07.595 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:59:07 smithi122 ceph-mon[25721]: pgmap v148: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T13:59:07.595 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Apr 14 13:59:07 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-x[26913]: ::ffff:172.21.15.122 - - [14/Apr/2024:13:59:07] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-04-14T13:59:07.817 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:59:07 smithi069 ceph-mon[24818]: pgmap v148: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T13:59:07.817 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:59:07 smithi069 ceph-mon[29472]: pgmap v148: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T13:59:09.817 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:59:09 smithi069 ceph-mon[24818]: pgmap v149: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T13:59:09.817 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 14 13:59:09 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-y[25054]: ::ffff:172.21.15.122 - - [14/Apr/2024:13:59:09] "GET /metrics HTTP/1.1" 200 208039 "" "Prometheus/2.33.4" 2024-04-14T13:59:09.817 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:59:09 smithi069 ceph-mon[29472]: pgmap v149: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T13:59:09.844 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:59:09 smithi122 ceph-mon[25721]: pgmap v149: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T13:59:11.595 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:59:11 smithi122 ceph-mon[25721]: pgmap v150: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T13:59:11.817 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:59:11 smithi069 ceph-mon[24818]: pgmap v150: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T13:59:11.817 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:59:11 smithi069 ceph-mon[29472]: pgmap v150: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T13:59:13.817 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:59:13 smithi069 ceph-mon[29472]: pgmap v151: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T13:59:13.817 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:59:13 smithi069 ceph-mon[24818]: pgmap v151: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T13:59:13.844 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:59:13 smithi122 ceph-mon[25721]: pgmap v151: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T13:59:15.817 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:59:15 smithi069 ceph-mon[24818]: from='client.14766 -' entity='client.iscsi.foo.smithi069.yuumtr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T13:59:15.817 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:59:15 smithi069 ceph-mon[24818]: pgmap v152: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T13:59:15.817 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:59:15 smithi069 ceph-mon[29472]: from='client.14766 -' entity='client.iscsi.foo.smithi069.yuumtr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T13:59:15.817 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:59:15 smithi069 ceph-mon[29472]: pgmap v152: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T13:59:15.844 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:59:15 smithi122 ceph-mon[25721]: from='client.14766 -' entity='client.iscsi.foo.smithi069.yuumtr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T13:59:15.844 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:59:15 smithi122 ceph-mon[25721]: pgmap v152: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T13:59:17.595 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:59:17 smithi122 ceph-mon[25721]: pgmap v153: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T13:59:17.595 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Apr 14 13:59:17 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-x[26913]: ::ffff:172.21.15.122 - - [14/Apr/2024:13:59:17] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-04-14T13:59:17.817 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:59:17 smithi069 ceph-mon[24818]: pgmap v153: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T13:59:17.817 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:59:17 smithi069 ceph-mon[29472]: pgmap v153: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T13:59:19.816 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:59:19 smithi069 ceph-mon[24818]: pgmap v154: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T13:59:19.817 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 14 13:59:19 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-y[25054]: ::ffff:172.21.15.122 - - [14/Apr/2024:13:59:19] "GET /metrics HTTP/1.1" 200 208039 "" "Prometheus/2.33.4" 2024-04-14T13:59:19.818 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:59:19 smithi069 ceph-mon[29472]: pgmap v154: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T13:59:19.844 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:59:19 smithi122 ceph-mon[25721]: pgmap v154: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T13:59:21.595 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:59:21 smithi122 ceph-mon[25721]: pgmap v155: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T13:59:21.817 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:59:21 smithi069 ceph-mon[24818]: pgmap v155: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T13:59:21.817 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:59:21 smithi069 ceph-mon[29472]: pgmap v155: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T13:59:23.817 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:59:23 smithi069 ceph-mon[24818]: pgmap v156: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T13:59:23.817 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:59:23 smithi069 ceph-mon[29472]: pgmap v156: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T13:59:23.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:59:23 smithi122 ceph-mon[25721]: pgmap v156: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T13:59:24.817 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:59:24 smithi069 ceph-mon[24818]: from='mgr.14448 172.21.15.69:0/453473650' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-04-14T13:59:24.817 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:59:24 smithi069 ceph-mon[24818]: from='mgr.14448 172.21.15.69:0/453473650' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-04-14T13:59:24.817 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:59:24 smithi069 ceph-mon[29472]: from='mgr.14448 172.21.15.69:0/453473650' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-04-14T13:59:24.817 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:59:24 smithi069 ceph-mon[29472]: from='mgr.14448 172.21.15.69:0/453473650' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-04-14T13:59:24.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:59:24 smithi122 ceph-mon[25721]: from='mgr.14448 172.21.15.69:0/453473650' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-04-14T13:59:24.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:59:24 smithi122 ceph-mon[25721]: from='mgr.14448 172.21.15.69:0/453473650' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-04-14T13:59:25.817 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:59:25 smithi069 ceph-mon[24818]: from='client.14766 -' entity='client.iscsi.foo.smithi069.yuumtr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T13:59:25.817 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:59:25 smithi069 ceph-mon[24818]: pgmap v157: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-04-14T13:59:25.817 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:59:25 smithi069 ceph-mon[29472]: from='client.14766 -' entity='client.iscsi.foo.smithi069.yuumtr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T13:59:25.817 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:59:25 smithi069 ceph-mon[29472]: pgmap v157: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-04-14T13:59:25.844 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:59:25 smithi122 ceph-mon[25721]: from='client.14766 -' entity='client.iscsi.foo.smithi069.yuumtr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T13:59:25.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:59:25 smithi122 ceph-mon[25721]: pgmap v157: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-04-14T13:59:27.595 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:59:27 smithi122 ceph-mon[25721]: pgmap v158: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T13:59:27.596 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Apr 14 13:59:27 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-x[26913]: ::ffff:172.21.15.122 - - [14/Apr/2024:13:59:27] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-04-14T13:59:27.817 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:59:27 smithi069 ceph-mon[24818]: pgmap v158: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T13:59:27.817 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:59:27 smithi069 ceph-mon[29472]: pgmap v158: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T13:59:29.817 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:59:29 smithi069 ceph-mon[24818]: pgmap v159: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-04-14T13:59:29.817 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:59:29 smithi069 ceph-mon[29472]: pgmap v159: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-04-14T13:59:29.817 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 14 13:59:29 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-y[25054]: ::ffff:172.21.15.122 - - [14/Apr/2024:13:59:29] "GET /metrics HTTP/1.1" 200 208040 "" "Prometheus/2.33.4" 2024-04-14T13:59:29.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:59:29 smithi122 ceph-mon[25721]: pgmap v159: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-04-14T13:59:31.551 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:59:31 smithi122 ceph-mon[25721]: pgmap v160: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T13:59:31.552 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:59:31 smithi122 ceph-mon[25721]: from='mgr.14448 172.21.15.69:0/453473650' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T13:59:31.552 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:59:31 smithi122 ceph-mon[25721]: from='mgr.14448 172.21.15.69:0/453473650' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-14T13:59:31.555 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:59:31 smithi069 ceph-mon[24818]: pgmap v160: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T13:59:31.556 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:59:31 smithi069 ceph-mon[24818]: from='mgr.14448 172.21.15.69:0/453473650' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T13:59:31.556 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:59:31 smithi069 ceph-mon[24818]: from='mgr.14448 172.21.15.69:0/453473650' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-14T13:59:31.556 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:59:31 smithi069 ceph-mon[29472]: pgmap v160: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T13:59:31.556 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:59:31 smithi069 ceph-mon[29472]: from='mgr.14448 172.21.15.69:0/453473650' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T13:59:31.556 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:59:31 smithi069 ceph-mon[29472]: from='mgr.14448 172.21.15.69:0/453473650' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-14T13:59:32.817 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:59:32 smithi069 ceph-mon[24818]: from='mgr.14448 172.21.15.69:0/453473650' entity='mgr.y' 2024-04-14T13:59:32.817 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:59:32 smithi069 ceph-mon[24818]: pgmap v161: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-04-14T13:59:32.817 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:59:32 smithi069 ceph-mon[29472]: from='mgr.14448 172.21.15.69:0/453473650' entity='mgr.y' 2024-04-14T13:59:32.818 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:59:32 smithi069 ceph-mon[29472]: pgmap v161: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-04-14T13:59:32.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:59:32 smithi122 ceph-mon[25721]: from='mgr.14448 172.21.15.69:0/453473650' entity='mgr.y' 2024-04-14T13:59:32.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:59:32 smithi122 ceph-mon[25721]: pgmap v161: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-04-14T13:59:35.817 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:59:35 smithi069 ceph-mon[24818]: from='client.14766 -' entity='client.iscsi.foo.smithi069.yuumtr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T13:59:35.817 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:59:35 smithi069 ceph-mon[24818]: pgmap v162: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T13:59:35.817 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:59:35 smithi069 ceph-mon[29472]: from='client.14766 -' entity='client.iscsi.foo.smithi069.yuumtr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T13:59:35.817 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:59:35 smithi069 ceph-mon[29472]: pgmap v162: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T13:59:35.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:59:35 smithi122 ceph-mon[25721]: from='client.14766 -' entity='client.iscsi.foo.smithi069.yuumtr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T13:59:35.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:59:35 smithi122 ceph-mon[25721]: pgmap v162: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T13:59:37.595 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:59:37 smithi122 ceph-mon[25721]: pgmap v163: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T13:59:37.595 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Apr 14 13:59:37 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-x[26913]: ::ffff:172.21.15.122 - - [14/Apr/2024:13:59:37] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-04-14T13:59:37.817 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:59:37 smithi069 ceph-mon[24818]: pgmap v163: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T13:59:37.817 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:59:37 smithi069 ceph-mon[29472]: pgmap v163: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T13:59:39.817 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:59:39 smithi069 ceph-mon[24818]: pgmap v164: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T13:59:39.817 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 14 13:59:39 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-y[25054]: ::ffff:172.21.15.122 - - [14/Apr/2024:13:59:39] "GET /metrics HTTP/1.1" 200 208008 "" "Prometheus/2.33.4" 2024-04-14T13:59:39.817 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:59:39 smithi069 ceph-mon[29472]: pgmap v164: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T13:59:39.844 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:59:39 smithi122 ceph-mon[25721]: pgmap v164: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T13:59:41.595 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:59:41 smithi122 ceph-mon[25721]: pgmap v165: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T13:59:41.817 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:59:41 smithi069 ceph-mon[24818]: pgmap v165: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T13:59:41.817 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:59:41 smithi069 ceph-mon[29472]: pgmap v165: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T13:59:43.816 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:59:43 smithi069 ceph-mon[24818]: pgmap v166: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T13:59:43.817 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:59:43 smithi069 ceph-mon[29472]: pgmap v166: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T13:59:43.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:59:43 smithi122 ceph-mon[25721]: pgmap v166: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T13:59:45.817 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:59:45 smithi069 ceph-mon[24818]: from='client.14766 -' entity='client.iscsi.foo.smithi069.yuumtr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T13:59:45.817 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:59:45 smithi069 ceph-mon[24818]: pgmap v167: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T13:59:45.817 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:59:45 smithi069 ceph-mon[29472]: from='client.14766 -' entity='client.iscsi.foo.smithi069.yuumtr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T13:59:45.817 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:59:45 smithi069 ceph-mon[29472]: pgmap v167: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T13:59:45.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:59:45 smithi122 ceph-mon[25721]: from='client.14766 -' entity='client.iscsi.foo.smithi069.yuumtr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T13:59:45.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:59:45 smithi122 ceph-mon[25721]: pgmap v167: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T13:59:47.595 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:59:47 smithi122 ceph-mon[25721]: pgmap v168: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T13:59:47.595 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Apr 14 13:59:47 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-x[26913]: ::ffff:172.21.15.122 - - [14/Apr/2024:13:59:47] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-04-14T13:59:47.817 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:59:47 smithi069 ceph-mon[24818]: pgmap v168: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T13:59:47.817 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:59:47 smithi069 ceph-mon[29472]: pgmap v168: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T13:59:49.817 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:59:49 smithi069 ceph-mon[24818]: pgmap v169: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T13:59:49.817 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 14 13:59:49 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-y[25054]: ::ffff:172.21.15.122 - - [14/Apr/2024:13:59:49] "GET /metrics HTTP/1.1" 200 208008 "" "Prometheus/2.33.4" 2024-04-14T13:59:49.817 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:59:49 smithi069 ceph-mon[29472]: pgmap v169: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T13:59:49.844 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:59:49 smithi122 ceph-mon[25721]: pgmap v169: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T13:59:51.594 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:59:51 smithi122 ceph-mon[25721]: pgmap v170: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T13:59:51.817 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:59:51 smithi069 ceph-mon[24818]: pgmap v170: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T13:59:51.817 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:59:51 smithi069 ceph-mon[29472]: pgmap v170: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T13:59:53.817 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:59:53 smithi069 ceph-mon[24818]: pgmap v171: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T13:59:53.817 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:59:53 smithi069 ceph-mon[29472]: pgmap v171: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T13:59:53.844 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:59:53 smithi122 ceph-mon[25721]: pgmap v171: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T13:59:55.817 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:59:55 smithi069 ceph-mon[24818]: from='client.14766 -' entity='client.iscsi.foo.smithi069.yuumtr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T13:59:55.817 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:59:55 smithi069 ceph-mon[24818]: pgmap v172: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T13:59:55.817 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:59:55 smithi069 ceph-mon[29472]: from='client.14766 -' entity='client.iscsi.foo.smithi069.yuumtr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T13:59:55.818 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:59:55 smithi069 ceph-mon[29472]: pgmap v172: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T13:59:55.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:59:55 smithi122 ceph-mon[25721]: from='client.14766 -' entity='client.iscsi.foo.smithi069.yuumtr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T13:59:55.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:59:55 smithi122 ceph-mon[25721]: pgmap v172: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T13:59:57.595 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:59:57 smithi122 ceph-mon[25721]: pgmap v173: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T13:59:57.595 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Apr 14 13:59:57 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-x[26913]: ::ffff:172.21.15.122 - - [14/Apr/2024:13:59:57] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-04-14T13:59:57.739 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:59:57 smithi069 ceph-mon[24818]: pgmap v173: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T13:59:57.739 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:59:57 smithi069 ceph-mon[29472]: pgmap v173: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T13:59:59.817 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 13:59:59 smithi069 ceph-mon[24818]: pgmap v174: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T13:59:59.817 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 14 13:59:59 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-y[25054]: ::ffff:172.21.15.122 - - [14/Apr/2024:13:59:59] "GET /metrics HTTP/1.1" 200 207983 "" "Prometheus/2.33.4" 2024-04-14T13:59:59.817 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 13:59:59 smithi069 ceph-mon[29472]: pgmap v174: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T13:59:59.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 13:59:59 smithi122 ceph-mon[25721]: pgmap v174: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:00:00.817 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:00:00 smithi069 ceph-mon[24818]: overall HEALTH_OK 2024-04-14T14:00:00.817 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:00:00 smithi069 ceph-mon[29472]: overall HEALTH_OK 2024-04-14T14:00:00.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:00:00 smithi122 ceph-mon[25721]: overall HEALTH_OK 2024-04-14T14:00:01.817 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:00:01 smithi069 ceph-mon[24818]: pgmap v175: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:00:01.817 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:00:01 smithi069 ceph-mon[29472]: pgmap v175: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:00:01.844 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:00:01 smithi122 ceph-mon[25721]: pgmap v175: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:00:03.817 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:00:03 smithi069 ceph-mon[29472]: pgmap v176: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:00:03.817 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:00:03 smithi069 ceph-mon[24818]: pgmap v176: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:00:03.844 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:00:03 smithi122 ceph-mon[25721]: pgmap v176: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:00:05.817 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:00:05 smithi069 ceph-mon[24818]: from='client.14766 -' entity='client.iscsi.foo.smithi069.yuumtr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T14:00:05.817 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:00:05 smithi069 ceph-mon[24818]: pgmap v177: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:00:05.817 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:00:05 smithi069 ceph-mon[29472]: from='client.14766 -' entity='client.iscsi.foo.smithi069.yuumtr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T14:00:05.817 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:00:05 smithi069 ceph-mon[29472]: pgmap v177: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:00:05.844 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:00:05 smithi122 ceph-mon[25721]: from='client.14766 -' entity='client.iscsi.foo.smithi069.yuumtr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T14:00:05.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:00:05 smithi122 ceph-mon[25721]: pgmap v177: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:00:07.595 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:00:07 smithi122 ceph-mon[25721]: pgmap v178: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:00:07.595 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Apr 14 14:00:07 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-x[26913]: ::ffff:172.21.15.122 - - [14/Apr/2024:14:00:07] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-04-14T14:00:07.817 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:00:07 smithi069 ceph-mon[24818]: pgmap v178: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:00:07.817 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:00:07 smithi069 ceph-mon[29472]: pgmap v178: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:00:09.817 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:00:09 smithi069 ceph-mon[24818]: pgmap v179: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:00:09.817 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 14 14:00:09 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-y[25054]: ::ffff:172.21.15.122 - - [14/Apr/2024:14:00:09] "GET /metrics HTTP/1.1" 200 207988 "" "Prometheus/2.33.4" 2024-04-14T14:00:09.817 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:00:09 smithi069 ceph-mon[29472]: pgmap v179: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:00:09.844 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:00:09 smithi122 ceph-mon[25721]: pgmap v179: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:00:11.594 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:00:11 smithi122 ceph-mon[25721]: pgmap v180: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:00:11.817 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:00:11 smithi069 ceph-mon[24818]: pgmap v180: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:00:11.817 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:00:11 smithi069 ceph-mon[29472]: pgmap v180: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:00:13.817 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:00:13 smithi069 ceph-mon[24818]: pgmap v181: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:00:13.817 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:00:13 smithi069 ceph-mon[29472]: pgmap v181: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:00:13.844 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:00:13 smithi122 ceph-mon[25721]: pgmap v181: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:00:15.817 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:00:15 smithi069 ceph-mon[24818]: from='client.14766 -' entity='client.iscsi.foo.smithi069.yuumtr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T14:00:15.817 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:00:15 smithi069 ceph-mon[24818]: pgmap v182: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:00:15.817 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:00:15 smithi069 ceph-mon[29472]: from='client.14766 -' entity='client.iscsi.foo.smithi069.yuumtr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T14:00:15.817 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:00:15 smithi069 ceph-mon[29472]: pgmap v182: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:00:15.844 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:00:15 smithi122 ceph-mon[25721]: from='client.14766 -' entity='client.iscsi.foo.smithi069.yuumtr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T14:00:15.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:00:15 smithi122 ceph-mon[25721]: pgmap v182: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:00:16.224 DEBUG:teuthology.orchestra.run.smithi069:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid dc326fc2-fa65-11ee-bc8e-c7b262605968 -e sha1=a971fd5f5719038c70b817bafacd6d5400b1d888 -- bash -c 'ceph config set mon mon_warn_on_insecure_global_id_reclaim false --force' 2024-04-14T14:00:17.595 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:00:17 smithi122 ceph-mon[25721]: pgmap v183: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:00:17.595 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Apr 14 14:00:17 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-x[26913]: ::ffff:172.21.15.122 - - [14/Apr/2024:14:00:17] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-04-14T14:00:17.755 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:00:17 smithi069 ceph-mon[24818]: pgmap v183: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:00:17.756 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:00:17 smithi069 ceph-mon[29472]: pgmap v183: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:00:18.850 DEBUG:teuthology.orchestra.run.smithi069:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid dc326fc2-fa65-11ee-bc8e-c7b262605968 -e sha1=a971fd5f5719038c70b817bafacd6d5400b1d888 -- bash -c 'ceph config set mon mon_warn_on_insecure_global_id_reclaim_allowed false --force' 2024-04-14T14:00:19.817 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:00:19 smithi069 ceph-mon[24818]: pgmap v184: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:00:19.817 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 14 14:00:19 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-y[25054]: ::ffff:172.21.15.122 - - [14/Apr/2024:14:00:19] "GET /metrics HTTP/1.1" 200 207988 "" "Prometheus/2.33.4" 2024-04-14T14:00:19.817 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:00:19 smithi069 ceph-mon[29472]: pgmap v184: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:00:19.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:00:19 smithi122 ceph-mon[25721]: pgmap v184: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:00:21.265 DEBUG:teuthology.orchestra.run.smithi069:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid dc326fc2-fa65-11ee-bc8e-c7b262605968 -e sha1=a971fd5f5719038c70b817bafacd6d5400b1d888 -- bash -c 'ceph config set global log_to_journald false --force' 2024-04-14T14:00:21.566 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:00:21 smithi069 ceph-mon[24818]: pgmap v185: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:00:21.567 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:00:21 smithi069 ceph-mon[29472]: pgmap v185: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:00:21.647 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:00:21 smithi122 ceph-mon[25721]: pgmap v185: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:00:23.662 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:00:23 smithi069 ceph-mon[24818]: pgmap v186: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:00:23.663 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:00:23 smithi069 ceph-mon[29472]: pgmap v186: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:00:23.688 DEBUG:teuthology.orchestra.run.smithi069:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid dc326fc2-fa65-11ee-bc8e-c7b262605968 -e sha1=a971fd5f5719038c70b817bafacd6d5400b1d888 -- bash -c 'ceph orch ps' 2024-04-14T14:00:23.798 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:00:23 smithi122 ceph-mon[25721]: pgmap v186: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:00:24.567 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:00:24 smithi069 ceph-mon[24818]: from='mgr.14448 172.21.15.69:0/453473650' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-04-14T14:00:24.567 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:00:24 smithi069 ceph-mon[24818]: from='mgr.14448 172.21.15.69:0/453473650' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-04-14T14:00:24.567 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:00:24 smithi069 ceph-mon[29472]: from='mgr.14448 172.21.15.69:0/453473650' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-04-14T14:00:24.567 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:00:24 smithi069 ceph-mon[29472]: from='mgr.14448 172.21.15.69:0/453473650' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-04-14T14:00:24.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:00:24 smithi122 ceph-mon[25721]: from='mgr.14448 172.21.15.69:0/453473650' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-04-14T14:00:24.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:00:24 smithi122 ceph-mon[25721]: from='mgr.14448 172.21.15.69:0/453473650' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-04-14T14:00:25.640 INFO:teuthology.orchestra.run.smithi069.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-14T14:00:25.640 INFO:teuthology.orchestra.run.smithi069.stdout:alertmanager.a smithi069 *:9093,9094 running (3m) 2m ago 3m 17.2M - ba2b418f427c 148fbcb55232 2024-04-14T14:00:25.640 INFO:teuthology.orchestra.run.smithi069.stdout:grafana.a smithi122 *:3000 running (3m) 2m ago 3m 45.4M - 8.3.5 dad864ee21e9 6c0d806d20ea 2024-04-14T14:00:25.640 INFO:teuthology.orchestra.run.smithi069.stdout:iscsi.foo.smithi069.yuumtr smithi069 running (3m) 2m ago 3m 57.9M - 3.5 e1d6a67b021e 61bdcc2a5081 2024-04-14T14:00:25.640 INFO:teuthology.orchestra.run.smithi069.stdout:mgr.x smithi122 *:8443 running (8m) 2m ago 8m 411M - 17.2.0 e1d6a67b021e 9aaf29b023b1 2024-04-14T14:00:25.640 INFO:teuthology.orchestra.run.smithi069.stdout:mgr.y smithi069 *:9283 running (9m) 2m ago 9m 454M - 17.2.0 e1d6a67b021e 52e1363a8f98 2024-04-14T14:00:25.641 INFO:teuthology.orchestra.run.smithi069.stdout:mon.a smithi069 running (10m) 2m ago 10m 46.9M 2048M 17.2.0 e1d6a67b021e d3c553f91913 2024-04-14T14:00:25.641 INFO:teuthology.orchestra.run.smithi069.stdout:mon.b smithi122 running (8m) 2m ago 8m 42.1M 2048M 17.2.0 e1d6a67b021e 3b766bfce4be 2024-04-14T14:00:25.641 INFO:teuthology.orchestra.run.smithi069.stdout:mon.c smithi069 running (8m) 2m ago 8m 38.9M 2048M 17.2.0 e1d6a67b021e 23c078b9d1e5 2024-04-14T14:00:25.641 INFO:teuthology.orchestra.run.smithi069.stdout:node-exporter.a smithi069 *:9100 running (4m) 2m ago 4m 10.3M - 1dbe0e931976 73e900b404df 2024-04-14T14:00:25.641 INFO:teuthology.orchestra.run.smithi069.stdout:node-exporter.b smithi122 *:9100 running (4m) 2m ago 4m 18.2M - 1dbe0e931976 e57f3d5931fb 2024-04-14T14:00:25.641 INFO:teuthology.orchestra.run.smithi069.stdout:osd.0 smithi069 running (8m) 2m ago 8m 49.8M 4016M 17.2.0 e1d6a67b021e 46654654b6b6 2024-04-14T14:00:25.641 INFO:teuthology.orchestra.run.smithi069.stdout:osd.1 smithi069 running (7m) 2m ago 7m 47.8M 4016M 17.2.0 e1d6a67b021e cdba54a59609 2024-04-14T14:00:25.641 INFO:teuthology.orchestra.run.smithi069.stdout:osd.2 smithi069 running (7m) 2m ago 7m 44.5M 4016M 17.2.0 e1d6a67b021e bc37a1a9cba0 2024-04-14T14:00:25.641 INFO:teuthology.orchestra.run.smithi069.stdout:osd.3 smithi069 running (6m) 2m ago 6m 47.6M 4016M 17.2.0 e1d6a67b021e 0925ba3da801 2024-04-14T14:00:25.641 INFO:teuthology.orchestra.run.smithi069.stdout:osd.4 smithi122 running (6m) 2m ago 6m 46.5M 4272M 17.2.0 e1d6a67b021e c00975caf771 2024-04-14T14:00:25.641 INFO:teuthology.orchestra.run.smithi069.stdout:osd.5 smithi122 running (6m) 2m ago 6m 47.0M 4272M 17.2.0 e1d6a67b021e 5f52d217ceb8 2024-04-14T14:00:25.641 INFO:teuthology.orchestra.run.smithi069.stdout:osd.6 smithi122 running (5m) 2m ago 5m 43.2M 4272M 17.2.0 e1d6a67b021e a2acfec61475 2024-04-14T14:00:25.641 INFO:teuthology.orchestra.run.smithi069.stdout:osd.7 smithi122 running (5m) 2m ago 5m 48.9M 4272M 17.2.0 e1d6a67b021e 62411cd06801 2024-04-14T14:00:25.642 INFO:teuthology.orchestra.run.smithi069.stdout:prometheus.a smithi122 *:9095 running (3m) 2m ago 4m 42.4M - 514e6a882f6e 59e61af03fdf 2024-04-14T14:00:25.642 INFO:teuthology.orchestra.run.smithi069.stdout:rgw.foo.smithi069.wesedz smithi069 *:8000 running (3m) 2m ago 3m 71.6M - 17.2.0 e1d6a67b021e 16de491440f3 2024-04-14T14:00:25.642 INFO:teuthology.orchestra.run.smithi069.stdout:rgw.foo.smithi122.jiobyv smithi122 *:8000 running (3m) 2m ago 3m 68.3M - 17.2.0 e1d6a67b021e 6d16de445dba 2024-04-14T14:00:25.642 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:00:25 smithi069 ceph-mon[24818]: from='client.14766 -' entity='client.iscsi.foo.smithi069.yuumtr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T14:00:25.642 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:00:25 smithi069 ceph-mon[24818]: pgmap v187: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:00:25.642 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:00:25 smithi069 ceph-mon[29472]: from='client.14766 -' entity='client.iscsi.foo.smithi069.yuumtr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T14:00:25.642 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:00:25 smithi069 ceph-mon[29472]: pgmap v187: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:00:25.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:00:25 smithi122 ceph-mon[25721]: from='client.14766 -' entity='client.iscsi.foo.smithi069.yuumtr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T14:00:25.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:00:25 smithi122 ceph-mon[25721]: pgmap v187: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:00:26.231 DEBUG:teuthology.orchestra.run.smithi069:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid dc326fc2-fa65-11ee-bc8e-c7b262605968 -e sha1=a971fd5f5719038c70b817bafacd6d5400b1d888 -- bash -c 'ceph versions' 2024-04-14T14:00:27.595 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:00:27 smithi122 ceph-mon[25721]: from='client.14967 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T14:00:27.595 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:00:27 smithi122 ceph-mon[25721]: pgmap v188: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:00:27.595 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Apr 14 14:00:27 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-x[26913]: ::ffff:172.21.15.122 - - [14/Apr/2024:14:00:27] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-04-14T14:00:27.670 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:00:27 smithi069 ceph-mon[29472]: from='client.14967 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T14:00:27.671 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:00:27 smithi069 ceph-mon[29472]: pgmap v188: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:00:27.671 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:00:27 smithi069 ceph-mon[24818]: from='client.14967 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T14:00:27.671 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:00:27 smithi069 ceph-mon[24818]: pgmap v188: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:00:28.115 INFO:teuthology.orchestra.run.smithi069.stdout:{ 2024-04-14T14:00:28.115 INFO:teuthology.orchestra.run.smithi069.stdout: "mon": { 2024-04-14T14:00:28.115 INFO:teuthology.orchestra.run.smithi069.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 3 2024-04-14T14:00:28.115 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-14T14:00:28.115 INFO:teuthology.orchestra.run.smithi069.stdout: "mgr": { 2024-04-14T14:00:28.115 INFO:teuthology.orchestra.run.smithi069.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-04-14T14:00:28.115 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-14T14:00:28.115 INFO:teuthology.orchestra.run.smithi069.stdout: "osd": { 2024-04-14T14:00:28.116 INFO:teuthology.orchestra.run.smithi069.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 8 2024-04-14T14:00:28.116 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-14T14:00:28.116 INFO:teuthology.orchestra.run.smithi069.stdout: "mds": {}, 2024-04-14T14:00:28.116 INFO:teuthology.orchestra.run.smithi069.stdout: "rgw": { 2024-04-14T14:00:28.116 INFO:teuthology.orchestra.run.smithi069.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-04-14T14:00:28.116 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-14T14:00:28.116 INFO:teuthology.orchestra.run.smithi069.stdout: "overall": { 2024-04-14T14:00:28.116 INFO:teuthology.orchestra.run.smithi069.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 15 2024-04-14T14:00:28.116 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-14T14:00:28.116 INFO:teuthology.orchestra.run.smithi069.stdout:} 2024-04-14T14:00:28.728 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:00:28 smithi069 ceph-mon[29472]: from='client.? 172.21.15.69:0/841862578' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T14:00:28.729 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:00:28 smithi069 ceph-mon[24818]: from='client.? 172.21.15.69:0/841862578' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T14:00:28.730 DEBUG:teuthology.orchestra.run.smithi069:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid dc326fc2-fa65-11ee-bc8e-c7b262605968 -e sha1=a971fd5f5719038c70b817bafacd6d5400b1d888 -- bash -c 'ceph -s' 2024-04-14T14:00:28.846 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:00:28 smithi122 ceph-mon[25721]: from='client.? 172.21.15.69:0/841862578' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T14:00:29.720 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:00:29 smithi069 ceph-mon[24818]: pgmap v189: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:00:29.721 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 14 14:00:29 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-y[25054]: ::ffff:172.21.15.122 - - [14/Apr/2024:14:00:29] "GET /metrics HTTP/1.1" 200 207987 "" "Prometheus/2.33.4" 2024-04-14T14:00:29.721 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:00:29 smithi069 ceph-mon[29472]: pgmap v189: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:00:29.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:00:29 smithi122 ceph-mon[25721]: pgmap v189: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:00:30.667 INFO:teuthology.orchestra.run.smithi069.stdout: cluster: 2024-04-14T14:00:30.667 INFO:teuthology.orchestra.run.smithi069.stdout: id: dc326fc2-fa65-11ee-bc8e-c7b262605968 2024-04-14T14:00:30.668 INFO:teuthology.orchestra.run.smithi069.stdout: health: HEALTH_OK 2024-04-14T14:00:30.668 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-14T14:00:30.668 INFO:teuthology.orchestra.run.smithi069.stdout: services: 2024-04-14T14:00:30.668 INFO:teuthology.orchestra.run.smithi069.stdout: mon: 3 daemons, quorum a,c,b (age 8m) 2024-04-14T14:00:30.668 INFO:teuthology.orchestra.run.smithi069.stdout: mgr: y(active, since 5m), standbys: x 2024-04-14T14:00:30.668 INFO:teuthology.orchestra.run.smithi069.stdout: osd: 8 osds: 8 up (since 5m), 8 in (since 5m) 2024-04-14T14:00:30.668 INFO:teuthology.orchestra.run.smithi069.stdout: rgw: 2 daemons active (2 hosts, 1 zones) 2024-04-14T14:00:30.668 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-14T14:00:30.668 INFO:teuthology.orchestra.run.smithi069.stdout: data: 2024-04-14T14:00:30.668 INFO:teuthology.orchestra.run.smithi069.stdout: pools: 6 pools, 161 pgs 2024-04-14T14:00:30.668 INFO:teuthology.orchestra.run.smithi069.stdout: objects: 209 objects, 585 KiB 2024-04-14T14:00:30.668 INFO:teuthology.orchestra.run.smithi069.stdout: usage: 63 MiB used, 715 GiB / 715 GiB avail 2024-04-14T14:00:30.668 INFO:teuthology.orchestra.run.smithi069.stdout: pgs: 161 active+clean 2024-04-14T14:00:30.668 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-14T14:00:30.668 INFO:teuthology.orchestra.run.smithi069.stdout: io: 2024-04-14T14:00:30.669 INFO:teuthology.orchestra.run.smithi069.stdout: client: 853 B/s rd, 0 op/s rd, 0 op/s wr 2024-04-14T14:00:30.669 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-14T14:00:31.261 DEBUG:teuthology.orchestra.run.smithi069:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid dc326fc2-fa65-11ee-bc8e-c7b262605968 -e sha1=a971fd5f5719038c70b817bafacd6d5400b1d888 -- bash -c 'ceph orch ls' 2024-04-14T14:00:31.567 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:00:31 smithi069 ceph-mon[24818]: pgmap v190: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:00:31.567 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:00:31 smithi069 ceph-mon[24818]: from='client.? 172.21.15.69:0/3561780641' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch 2024-04-14T14:00:31.567 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:00:31 smithi069 ceph-mon[29472]: pgmap v190: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:00:31.567 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:00:31 smithi069 ceph-mon[29472]: from='client.? 172.21.15.69:0/3561780641' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch 2024-04-14T14:00:31.611 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:00:31 smithi122 ceph-mon[25721]: pgmap v190: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:00:31.612 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:00:31 smithi122 ceph-mon[25721]: from='client.? 172.21.15.69:0/3561780641' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch 2024-04-14T14:00:32.690 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:00:32 smithi069 ceph-mon[24818]: from='mgr.14448 172.21.15.69:0/453473650' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T14:00:32.690 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:00:32 smithi069 ceph-mon[24818]: from='mgr.14448 172.21.15.69:0/453473650' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-14T14:00:32.690 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:00:32 smithi069 ceph-mon[24818]: from='mgr.14448 172.21.15.69:0/453473650' entity='mgr.y' 2024-04-14T14:00:32.690 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:00:32 smithi069 ceph-mon[29472]: from='mgr.14448 172.21.15.69:0/453473650' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T14:00:32.690 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:00:32 smithi069 ceph-mon[29472]: from='mgr.14448 172.21.15.69:0/453473650' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-14T14:00:32.690 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:00:32 smithi069 ceph-mon[29472]: from='mgr.14448 172.21.15.69:0/453473650' entity='mgr.y' 2024-04-14T14:00:32.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:00:32 smithi122 ceph-mon[25721]: from='mgr.14448 172.21.15.69:0/453473650' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T14:00:32.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:00:32 smithi122 ceph-mon[25721]: from='mgr.14448 172.21.15.69:0/453473650' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-14T14:00:32.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:00:32 smithi122 ceph-mon[25721]: from='mgr.14448 172.21.15.69:0/453473650' entity='mgr.y' 2024-04-14T14:00:33.156 INFO:teuthology.orchestra.run.smithi069.stdout:NAME PORTS RUNNING REFRESHED AGE PLACEMENT 2024-04-14T14:00:33.157 INFO:teuthology.orchestra.run.smithi069.stdout:alertmanager ?:9093,9094 1/1 3m ago 5m smithi069=a;count:1 2024-04-14T14:00:33.157 INFO:teuthology.orchestra.run.smithi069.stdout:grafana ?:3000 1/1 3m ago 5m smithi122=a;count:1 2024-04-14T14:00:33.157 INFO:teuthology.orchestra.run.smithi069.stdout:iscsi.foo 1/1 3m ago 3m count:1 2024-04-14T14:00:33.157 INFO:teuthology.orchestra.run.smithi069.stdout:mgr 2/2 3m ago 8m smithi069=y;smithi122=x;count:2 2024-04-14T14:00:33.157 INFO:teuthology.orchestra.run.smithi069.stdout:mon 3/3 3m ago 8m smithi069:172.21.15.69=a;smithi069:[v2:172.21.15.69:3301,v1:172.21.15.69:6790]=c;smithi122:172.21.15.122=b;count:3 2024-04-14T14:00:33.157 INFO:teuthology.orchestra.run.smithi069.stdout:node-exporter ?:9100 2/2 3m ago 5m smithi069=a;smithi122=b;count:2 2024-04-14T14:00:33.157 INFO:teuthology.orchestra.run.smithi069.stdout:osd 8 3m ago - 2024-04-14T14:00:33.157 INFO:teuthology.orchestra.run.smithi069.stdout:prometheus ?:9095 1/1 3m ago 5m smithi122=a;count:1 2024-04-14T14:00:33.157 INFO:teuthology.orchestra.run.smithi069.stdout:rgw.foo ?:8000 2/2 3m ago 3m count:2 2024-04-14T14:00:33.595 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:00:33 smithi122 ceph-mon[25721]: pgmap v191: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:00:33.698 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:00:33 smithi069 ceph-mon[24818]: pgmap v191: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:00:33.699 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:00:33 smithi069 ceph-mon[29472]: pgmap v191: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:00:33.875 DEBUG:teuthology.orchestra.run.smithi069:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid dc326fc2-fa65-11ee-bc8e-c7b262605968 -e sha1=a971fd5f5719038c70b817bafacd6d5400b1d888 -- bash -c 'ceph orch daemon redeploy "mgr.$(ceph mgr dump -f json | jq .standbys | jq .[] | jq -r .name)" --image quay.ceph.io/ceph-ci/ceph:$sha1' 2024-04-14T14:00:34.493 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:00:34 smithi069 ceph-mon[29472]: from='client.14985 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T14:00:34.493 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:00:34 smithi069 ceph-mon[24818]: from='client.14985 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T14:00:34.844 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:00:34 smithi122 ceph-mon[25721]: from='client.14985 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T14:00:35.405 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:00:35 smithi069 ceph-mon[24818]: from='client.14766 -' entity='client.iscsi.foo.smithi069.yuumtr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T14:00:35.406 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:00:35 smithi069 ceph-mon[24818]: pgmap v192: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:00:35.406 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:00:35 smithi069 ceph-mon[29472]: from='client.14766 -' entity='client.iscsi.foo.smithi069.yuumtr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T14:00:35.406 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:00:35 smithi069 ceph-mon[29472]: pgmap v192: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:00:35.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:00:35 smithi122 ceph-mon[25721]: from='client.14766 -' entity='client.iscsi.foo.smithi069.yuumtr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T14:00:35.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:00:35 smithi122 ceph-mon[25721]: pgmap v192: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:00:36.154 INFO:teuthology.orchestra.run.smithi069.stdout:Scheduled to redeploy mgr.x on host 'smithi122' 2024-04-14T14:00:36.470 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:00:36 smithi069 ceph-mon[29472]: from='client.? 172.21.15.69:0/2415191276' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-04-14T14:00:36.470 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:00:36 smithi069 ceph-mon[29472]: from='mgr.14448 172.21.15.69:0/453473650' entity='mgr.y' 2024-04-14T14:00:36.470 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:00:36 smithi069 ceph-mon[29472]: from='mgr.14448 172.21.15.69:0/453473650' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T14:00:36.470 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:00:36 smithi069 ceph-mon[29472]: from='mgr.14448 172.21.15.69:0/453473650' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-14T14:00:36.470 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:00:36 smithi069 ceph-mon[29472]: from='mgr.14448 172.21.15.69:0/453473650' entity='mgr.y' 2024-04-14T14:00:36.470 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:00:36 smithi069 ceph-mon[29472]: from='mgr.14448 172.21.15.69:0/453473650' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-14T14:00:36.470 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:00:36 smithi069 ceph-mon[29472]: from='mgr.14448 172.21.15.69:0/453473650' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-14T14:00:36.470 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:00:36 smithi069 ceph-mon[29472]: from='mgr.14448 172.21.15.69:0/453473650' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T14:00:36.471 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:00:36 smithi069 ceph-mon[24818]: from='client.? 172.21.15.69:0/2415191276' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-04-14T14:00:36.471 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:00:36 smithi069 ceph-mon[24818]: from='mgr.14448 172.21.15.69:0/453473650' entity='mgr.y' 2024-04-14T14:00:36.471 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:00:36 smithi069 ceph-mon[24818]: from='mgr.14448 172.21.15.69:0/453473650' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T14:00:36.471 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:00:36 smithi069 ceph-mon[24818]: from='mgr.14448 172.21.15.69:0/453473650' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-14T14:00:36.471 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:00:36 smithi069 ceph-mon[24818]: from='mgr.14448 172.21.15.69:0/453473650' entity='mgr.y' 2024-04-14T14:00:36.471 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:00:36 smithi069 ceph-mon[24818]: from='mgr.14448 172.21.15.69:0/453473650' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-14T14:00:36.471 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:00:36 smithi069 ceph-mon[24818]: from='mgr.14448 172.21.15.69:0/453473650' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-14T14:00:36.471 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:00:36 smithi069 ceph-mon[24818]: from='mgr.14448 172.21.15.69:0/453473650' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T14:00:36.595 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:00:36 smithi122 ceph-mon[25721]: from='client.? 172.21.15.69:0/2415191276' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-04-14T14:00:36.595 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:00:36 smithi122 ceph-mon[25721]: from='mgr.14448 172.21.15.69:0/453473650' entity='mgr.y' 2024-04-14T14:00:36.595 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:00:36 smithi122 ceph-mon[25721]: from='mgr.14448 172.21.15.69:0/453473650' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T14:00:36.595 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:00:36 smithi122 ceph-mon[25721]: from='mgr.14448 172.21.15.69:0/453473650' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-14T14:00:36.595 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:00:36 smithi122 ceph-mon[25721]: from='mgr.14448 172.21.15.69:0/453473650' entity='mgr.y' 2024-04-14T14:00:36.595 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:00:36 smithi122 ceph-mon[25721]: from='mgr.14448 172.21.15.69:0/453473650' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-14T14:00:36.595 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:00:36 smithi122 ceph-mon[25721]: from='mgr.14448 172.21.15.69:0/453473650' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-14T14:00:36.595 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:00:36 smithi122 ceph-mon[25721]: from='mgr.14448 172.21.15.69:0/453473650' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T14:00:36.736 DEBUG:teuthology.orchestra.run.smithi069:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid dc326fc2-fa65-11ee-bc8e-c7b262605968 -e sha1=a971fd5f5719038c70b817bafacd6d5400b1d888 -- bash -c 'ceph orch ps --refresh' 2024-04-14T14:00:37.594 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Apr 14 14:00:37 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-x[26913]: ::ffff:172.21.15.122 - - [14/Apr/2024:14:00:37] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-04-14T14:00:37.595 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:00:37 smithi122 ceph-mon[25721]: from='client.14997 -' entity='client.admin' cmd=[{"prefix": "orch daemon redeploy", "name": "mgr.x", "image": "quay.ceph.io/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T14:00:37.595 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:00:37 smithi122 ceph-mon[25721]: Schedule redeploy daemon mgr.x 2024-04-14T14:00:37.595 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:00:37 smithi122 ceph-mon[25721]: Deploying daemon mgr.x on smithi122 2024-04-14T14:00:37.595 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:00:37 smithi122 ceph-mon[25721]: pgmap v193: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:00:37.817 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:00:37 smithi069 ceph-mon[24818]: from='client.14997 -' entity='client.admin' cmd=[{"prefix": "orch daemon redeploy", "name": "mgr.x", "image": "quay.ceph.io/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T14:00:37.817 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:00:37 smithi069 ceph-mon[24818]: Schedule redeploy daemon mgr.x 2024-04-14T14:00:37.817 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:00:37 smithi069 ceph-mon[24818]: Deploying daemon mgr.x on smithi122 2024-04-14T14:00:37.817 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:00:37 smithi069 ceph-mon[24818]: pgmap v193: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:00:37.817 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:00:37 smithi069 ceph-mon[29472]: from='client.14997 -' entity='client.admin' cmd=[{"prefix": "orch daemon redeploy", "name": "mgr.x", "image": "quay.ceph.io/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T14:00:37.817 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:00:37 smithi069 ceph-mon[29472]: Schedule redeploy daemon mgr.x 2024-04-14T14:00:37.817 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:00:37 smithi069 ceph-mon[29472]: Deploying daemon mgr.x on smithi122 2024-04-14T14:00:37.817 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:00:37 smithi069 ceph-mon[29472]: pgmap v193: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:00:38.748 INFO:teuthology.orchestra.run.smithi069.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-14T14:00:38.748 INFO:teuthology.orchestra.run.smithi069.stdout:alertmanager.a smithi069 *:9093,9094 running (3m) 3m ago 4m 17.2M - ba2b418f427c 148fbcb55232 2024-04-14T14:00:38.748 INFO:teuthology.orchestra.run.smithi069.stdout:grafana.a smithi122 *:3000 running (3m) 3m ago 3m 45.4M - 8.3.5 dad864ee21e9 6c0d806d20ea 2024-04-14T14:00:38.748 INFO:teuthology.orchestra.run.smithi069.stdout:iscsi.foo.smithi069.yuumtr smithi069 running (3m) 3m ago 3m 57.9M - 3.5 e1d6a67b021e 61bdcc2a5081 2024-04-14T14:00:38.748 INFO:teuthology.orchestra.run.smithi069.stdout:mgr.x smithi122 *:8443 running (8m) 3m ago 8m 411M - 17.2.0 e1d6a67b021e 9aaf29b023b1 2024-04-14T14:00:38.749 INFO:teuthology.orchestra.run.smithi069.stdout:mgr.y smithi069 *:9283 running (10m) 3m ago 10m 454M - 17.2.0 e1d6a67b021e 52e1363a8f98 2024-04-14T14:00:38.749 INFO:teuthology.orchestra.run.smithi069.stdout:mon.a smithi069 running (10m) 3m ago 10m 46.9M 2048M 17.2.0 e1d6a67b021e d3c553f91913 2024-04-14T14:00:38.749 INFO:teuthology.orchestra.run.smithi069.stdout:mon.b smithi122 running (8m) 3m ago 8m 42.1M 2048M 17.2.0 e1d6a67b021e 3b766bfce4be 2024-04-14T14:00:38.749 INFO:teuthology.orchestra.run.smithi069.stdout:mon.c smithi069 running (8m) 3m ago 8m 38.9M 2048M 17.2.0 e1d6a67b021e 23c078b9d1e5 2024-04-14T14:00:38.749 INFO:teuthology.orchestra.run.smithi069.stdout:node-exporter.a smithi069 *:9100 running (4m) 3m ago 4m 10.3M - 1dbe0e931976 73e900b404df 2024-04-14T14:00:38.749 INFO:teuthology.orchestra.run.smithi069.stdout:node-exporter.b smithi122 *:9100 running (4m) 3m ago 4m 18.2M - 1dbe0e931976 e57f3d5931fb 2024-04-14T14:00:38.749 INFO:teuthology.orchestra.run.smithi069.stdout:osd.0 smithi069 running (8m) 3m ago 8m 49.8M 4016M 17.2.0 e1d6a67b021e 46654654b6b6 2024-04-14T14:00:38.749 INFO:teuthology.orchestra.run.smithi069.stdout:osd.1 smithi069 running (7m) 3m ago 7m 47.8M 4016M 17.2.0 e1d6a67b021e cdba54a59609 2024-04-14T14:00:38.749 INFO:teuthology.orchestra.run.smithi069.stdout:osd.2 smithi069 running (7m) 3m ago 7m 44.5M 4016M 17.2.0 e1d6a67b021e bc37a1a9cba0 2024-04-14T14:00:38.749 INFO:teuthology.orchestra.run.smithi069.stdout:osd.3 smithi069 running (7m) 3m ago 7m 47.6M 4016M 17.2.0 e1d6a67b021e 0925ba3da801 2024-04-14T14:00:38.749 INFO:teuthology.orchestra.run.smithi069.stdout:osd.4 smithi122 running (6m) 3m ago 6m 46.5M 4272M 17.2.0 e1d6a67b021e c00975caf771 2024-04-14T14:00:38.749 INFO:teuthology.orchestra.run.smithi069.stdout:osd.5 smithi122 running (6m) 3m ago 6m 47.0M 4272M 17.2.0 e1d6a67b021e 5f52d217ceb8 2024-04-14T14:00:38.749 INFO:teuthology.orchestra.run.smithi069.stdout:osd.6 smithi122 running (5m) 3m ago 5m 43.2M 4272M 17.2.0 e1d6a67b021e a2acfec61475 2024-04-14T14:00:38.749 INFO:teuthology.orchestra.run.smithi069.stdout:osd.7 smithi122 running (5m) 3m ago 5m 48.9M 4272M 17.2.0 e1d6a67b021e 62411cd06801 2024-04-14T14:00:38.749 INFO:teuthology.orchestra.run.smithi069.stdout:prometheus.a smithi122 *:9095 running (3m) 3m ago 4m 42.4M - 514e6a882f6e 59e61af03fdf 2024-04-14T14:00:38.749 INFO:teuthology.orchestra.run.smithi069.stdout:rgw.foo.smithi069.wesedz smithi069 *:8000 running (3m) 3m ago 3m 71.6M - 17.2.0 e1d6a67b021e 16de491440f3 2024-04-14T14:00:38.750 INFO:teuthology.orchestra.run.smithi069.stdout:rgw.foo.smithi122.jiobyv smithi122 *:8000 running (3m) 3m ago 3m 68.3M - 17.2.0 e1d6a67b021e 6d16de445dba 2024-04-14T14:00:39.392 DEBUG:teuthology.orchestra.run.smithi069:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid dc326fc2-fa65-11ee-bc8e-c7b262605968 -e sha1=a971fd5f5719038c70b817bafacd6d5400b1d888 -- bash -c 'sleep 180' 2024-04-14T14:00:39.701 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:00:39 smithi069 ceph-mon[24818]: pgmap v194: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:00:39.701 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 14 14:00:39 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-y[25054]: ::ffff:172.21.15.122 - - [14/Apr/2024:14:00:39] "GET /metrics HTTP/1.1" 200 207970 "" "Prometheus/2.33.4" 2024-04-14T14:00:39.702 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:00:39 smithi069 ceph-mon[29472]: pgmap v194: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:00:39.844 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:00:39 smithi122 ceph-mon[25721]: pgmap v194: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:00:40.723 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:00:40 smithi069 ceph-mon[24818]: from='client.15003 -' entity='client.admin' cmd=[{"prefix": "orch ps", "refresh": true, "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T14:00:40.723 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:00:40 smithi069 ceph-mon[29472]: from='client.15003 -' entity='client.admin' cmd=[{"prefix": "orch ps", "refresh": true, "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T14:00:40.844 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:00:40 smithi122 ceph-mon[25721]: from='client.15003 -' entity='client.admin' cmd=[{"prefix": "orch ps", "refresh": true, "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T14:00:41.817 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:00:41 smithi069 ceph-mon[24818]: pgmap v195: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:00:41.817 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:00:41 smithi069 ceph-mon[29472]: pgmap v195: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:00:41.844 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:00:41 smithi122 ceph-mon[25721]: pgmap v195: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:00:43.709 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:00:43 smithi069 ceph-mon[24818]: pgmap v196: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:00:43.709 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:00:43 smithi069 ceph-mon[29472]: pgmap v196: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:00:43.844 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:00:43 smithi122 ceph-mon[25721]: pgmap v196: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:00:45.817 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:00:45 smithi069 ceph-mon[24818]: from='client.14766 -' entity='client.iscsi.foo.smithi069.yuumtr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T14:00:45.817 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:00:45 smithi069 ceph-mon[24818]: pgmap v197: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:00:45.817 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:00:45 smithi069 ceph-mon[29472]: from='client.14766 -' entity='client.iscsi.foo.smithi069.yuumtr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T14:00:45.817 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:00:45 smithi069 ceph-mon[29472]: pgmap v197: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:00:45.844 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:00:45 smithi122 ceph-mon[25721]: from='client.14766 -' entity='client.iscsi.foo.smithi069.yuumtr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T14:00:45.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:00:45 smithi122 ceph-mon[25721]: pgmap v197: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:00:47.594 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:00:47 smithi122 ceph-mon[25721]: pgmap v198: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:00:47.595 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Apr 14 14:00:47 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-x[26913]: ::ffff:172.21.15.122 - - [14/Apr/2024:14:00:47] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-04-14T14:00:47.817 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:00:47 smithi069 ceph-mon[29472]: pgmap v198: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:00:47.817 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:00:47 smithi069 ceph-mon[24818]: pgmap v198: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:00:49.817 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:00:49 smithi069 ceph-mon[24818]: pgmap v199: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:00:49.817 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 14 14:00:49 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-y[25054]: ::ffff:172.21.15.122 - - [14/Apr/2024:14:00:49] "GET /metrics HTTP/1.1" 200 207970 "" "Prometheus/2.33.4" 2024-04-14T14:00:49.817 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:00:49 smithi069 ceph-mon[29472]: pgmap v199: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:00:49.844 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:00:49 smithi122 ceph-mon[25721]: pgmap v199: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:00:51.646 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:00:51 smithi122 ceph-mon[25721]: pgmap v200: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:00:51.817 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:00:51 smithi069 ceph-mon[24818]: pgmap v200: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:00:51.817 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:00:51 smithi069 ceph-mon[29472]: pgmap v200: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:00:53.719 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:00:53 smithi069 ceph-mon[24818]: pgmap v201: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:00:53.720 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:00:53 smithi069 ceph-mon[29472]: pgmap v201: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:00:53.844 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:00:53 smithi122 ceph-mon[25721]: pgmap v201: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:00:55.817 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:00:55 smithi069 ceph-mon[24818]: from='client.14766 -' entity='client.iscsi.foo.smithi069.yuumtr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T14:00:55.817 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:00:55 smithi069 ceph-mon[24818]: pgmap v202: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:00:55.817 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:00:55 smithi069 ceph-mon[29472]: from='client.14766 -' entity='client.iscsi.foo.smithi069.yuumtr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T14:00:55.817 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:00:55 smithi069 ceph-mon[29472]: pgmap v202: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:00:55.844 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:00:55 smithi122 ceph-mon[25721]: from='client.14766 -' entity='client.iscsi.foo.smithi069.yuumtr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T14:00:55.844 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:00:55 smithi122 ceph-mon[25721]: pgmap v202: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:00:57.594 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:00:57 smithi122 ceph-mon[25721]: pgmap v203: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:00:57.595 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Apr 14 14:00:57 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-x[26913]: ::ffff:172.21.15.122 - - [14/Apr/2024:14:00:57] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-04-14T14:00:57.817 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:00:57 smithi069 ceph-mon[24818]: pgmap v203: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:00:57.817 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:00:57 smithi069 ceph-mon[29472]: pgmap v203: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:00:59.817 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:00:59 smithi069 ceph-mon[24818]: pgmap v204: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:00:59.817 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 14 14:00:59 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-y[25054]: ::ffff:172.21.15.122 - - [14/Apr/2024:14:00:59] "GET /metrics HTTP/1.1" 200 207984 "" "Prometheus/2.33.4" 2024-04-14T14:00:59.817 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:00:59 smithi069 ceph-mon[29472]: pgmap v204: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:00:59.844 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:00:59 smithi122 ceph-mon[25721]: pgmap v204: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:01:01.594 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:01:01 smithi122 ceph-mon[25721]: pgmap v205: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:01:01.817 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:01:01 smithi069 ceph-mon[24818]: pgmap v205: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:01:01.817 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:01:01 smithi069 ceph-mon[29472]: pgmap v205: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:01:03.728 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:01:03 smithi069 ceph-mon[24818]: pgmap v206: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:01:03.728 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:01:03 smithi069 ceph-mon[29472]: pgmap v206: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:01:03.844 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:01:03 smithi122 ceph-mon[25721]: pgmap v206: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:01:05.468 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Apr 14 14:01:05 smithi122 systemd[1]: Stopping Ceph mgr.x for dc326fc2-fa65-11ee-bc8e-c7b262605968... 2024-04-14T14:01:05.469 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Apr 14 14:01:05 smithi122 bash[45324]: Error: no container with name or ID "ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr.x" found: no such container 2024-04-14T14:01:05.815 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:01:05 smithi122 ceph-mon[25721]: from='client.14766 -' entity='client.iscsi.foo.smithi069.yuumtr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T14:01:05.815 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:01:05 smithi122 ceph-mon[25721]: pgmap v207: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:01:05.815 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Apr 14 14:01:05 smithi122 podman[45337]: 2024-04-14 14:01:05.559202727 +0000 UTC m=+0.131908240 container died 9aaf29b023b15d873a3c639dd2ab368d275242c3251b5d983ab68475c2451b3e (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-x, io.k8s.display-name=CentOS Stream 8, CEPH_POINT_RELEASE=-17.2.0, io.openshift.tags=base centos centos-stream, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, vcs-type=git, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, RELEASE=HEAD, GIT_REPO=https://github.com/ceph/ceph-container.git, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, version=8, distribution-scope=public, GIT_BRANCH=HEAD, name=centos-stream, GIT_CLEAN=True, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, vendor=Red Hat, Inc., io.buildah.version=1.19.8, architecture=x86_64, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., build-date=2022-05-03T08:36:31.336870, release=754, ceph=True, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, io.openshift.expose-services=, com.redhat.component=centos-stream-container, maintainer=Guillaume Abrioux , url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb) 2024-04-14T14:01:05.816 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:01:05 smithi069 ceph-mon[24818]: from='client.14766 -' entity='client.iscsi.foo.smithi069.yuumtr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T14:01:05.817 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:01:05 smithi069 ceph-mon[24818]: pgmap v207: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:01:05.817 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:01:05 smithi069 ceph-mon[29472]: from='client.14766 -' entity='client.iscsi.foo.smithi069.yuumtr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T14:01:05.817 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:01:05 smithi069 ceph-mon[29472]: pgmap v207: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:01:06.094 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Apr 14 14:01:05 smithi122 podman[45337]: 2024-04-14 14:01:05.90479804 +0000 UTC m=+0.477503611 container cleanup 9aaf29b023b15d873a3c639dd2ab368d275242c3251b5d983ab68475c2451b3e (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-x, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, version=8, architecture=x86_64, GIT_BRANCH=HEAD, io.openshift.tags=base centos centos-stream, GIT_REPO=https://github.com/ceph/ceph-container.git, distribution-scope=public, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, vcs-type=git, release=754, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., vendor=Red Hat, Inc., CEPH_POINT_RELEASE=-17.2.0, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, maintainer=Guillaume Abrioux , name=centos-stream, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, GIT_CLEAN=True, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, io.buildah.version=1.19.8, ceph=True, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, RELEASE=HEAD, io.openshift.expose-services=, io.k8s.display-name=CentOS Stream 8, build-date=2022-05-03T08:36:31.336870, com.redhat.component=centos-stream-container) 2024-04-14T14:01:06.095 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Apr 14 14:01:05 smithi122 bash[45337]: ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-x 2024-04-14T14:01:06.095 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Apr 14 14:01:05 smithi122 bash[45364]: Error: no container with name or ID "ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr.x" found: no such container 2024-04-14T14:01:06.587 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Apr 14 14:01:06 smithi122 podman[45351]: 2024-04-14 14:01:06.18098599 +0000 UTC m=+0.618801106 container remove 9aaf29b023b15d873a3c639dd2ab368d275242c3251b5d983ab68475c2451b3e (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-x, GIT_CLEAN=True, GIT_REPO=https://github.com/ceph/ceph-container.git, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., CEPH_POINT_RELEASE=-17.2.0, architecture=x86_64, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, io.openshift.tags=base centos centos-stream, vcs-type=git, io.openshift.expose-services=, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, build-date=2022-05-03T08:36:31.336870, com.redhat.component=centos-stream-container, version=8, distribution-scope=public, io.k8s.display-name=CentOS Stream 8, release=754, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, ceph=True, io.buildah.version=1.19.8, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, RELEASE=HEAD, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, name=centos-stream, vendor=Red Hat, Inc.) 2024-04-14T14:01:06.588 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Apr 14 14:01:06 smithi122 systemd[1]: ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968@mgr.x.service: Main process exited, code=exited, status=143/n/a 2024-04-14T14:01:06.588 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Apr 14 14:01:06 smithi122 systemd[1]: ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968@mgr.x.service: Failed with result 'exit-code'. 2024-04-14T14:01:06.588 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Apr 14 14:01:06 smithi122 systemd[1]: Stopped Ceph mgr.x for dc326fc2-fa65-11ee-bc8e-c7b262605968. 2024-04-14T14:01:06.588 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Apr 14 14:01:06 smithi122 systemd[1]: ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968@mgr.x.service: Consumed 16.528s CPU time. 2024-04-14T14:01:06.588 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Apr 14 14:01:06 smithi122 systemd[1]: Starting Ceph mgr.x for dc326fc2-fa65-11ee-bc8e-c7b262605968... 2024-04-14T14:01:06.845 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Apr 14 14:01:06 smithi122 podman[45472]: 2024-04-14 14:01:06.489184701 +0000 UTC m=+0.017970160 image pull b96c509eef841460b3271f5fdd331860da87b004524f4ed59414aa3680183b53 quay.ceph.io/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888 2024-04-14T14:01:06.845 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Apr 14 14:01:06 smithi122 podman[45472]: 2024-04-14 14:01:06.590725106 +0000 UTC m=+0.119510580 container create 813744c2b615145e5f46698e8c4e176ebcb682d3e3810002ac9c8baf37da0e71 (image=quay.ceph.io/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888, name=ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-x, RELEASE=main-a971fd5, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.build-date=20240403, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0) 2024-04-14T14:01:06.845 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Apr 14 14:01:06 smithi122 podman[45472]: 2024-04-14 14:01:06.818260691 +0000 UTC m=+0.347046123 container init 813744c2b615145e5f46698e8c4e176ebcb682d3e3810002ac9c8baf37da0e71 (image=quay.ceph.io/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888, name=ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-x, GIT_BRANCH=HEAD, ceph=True, RELEASE=main-a971fd5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240403, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, io.buildah.version=1.33.5) 2024-04-14T14:01:06.845 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Apr 14 14:01:06 smithi122 podman[45472]: 2024-04-14 14:01:06.820806645 +0000 UTC m=+0.349592077 container start 813744c2b615145e5f46698e8c4e176ebcb682d3e3810002ac9c8baf37da0e71 (image=quay.ceph.io/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888, name=ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-x, org.label-schema.license=GPLv2, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240403, ceph=True, RELEASE=main-a971fd5, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-14T14:01:07.345 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Apr 14 14:01:06 smithi122 ceph-mgr[45491]: -- 172.21.15.122:0/58209193 <== mon.2 v2:172.21.15.122:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x56217ad525a0 con 0x56217ad27000 2024-04-14T14:01:07.345 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Apr 14 14:01:06 smithi122 bash[45472]: 813744c2b615145e5f46698e8c4e176ebcb682d3e3810002ac9c8baf37da0e71 2024-04-14T14:01:07.345 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Apr 14 14:01:06 smithi122 systemd[1]: Started Ceph mgr.x for dc326fc2-fa65-11ee-bc8e-c7b262605968. 2024-04-14T14:01:07.567 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:01:07 smithi069 ceph-mon[24818]: pgmap v208: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:01:07.567 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:01:07 smithi069 ceph-mon[24818]: from='mgr.14448 172.21.15.69:0/453473650' entity='mgr.y' 2024-04-14T14:01:07.567 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:01:07 smithi069 ceph-mon[24818]: from='mgr.14448 172.21.15.69:0/453473650' entity='mgr.y' 2024-04-14T14:01:07.567 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:01:07 smithi069 ceph-mon[24818]: from='mgr.14448 172.21.15.69:0/453473650' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T14:01:07.567 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:01:07 smithi069 ceph-mon[24818]: from='mgr.14448 172.21.15.69:0/453473650' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-14T14:01:07.567 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:01:07 smithi069 ceph-mon[24818]: from='mgr.14448 172.21.15.69:0/453473650' entity='mgr.y' 2024-04-14T14:01:07.567 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:01:07 smithi069 ceph-mon[29472]: pgmap v208: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:01:07.567 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:01:07 smithi069 ceph-mon[29472]: from='mgr.14448 172.21.15.69:0/453473650' entity='mgr.y' 2024-04-14T14:01:07.567 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:01:07 smithi069 ceph-mon[29472]: from='mgr.14448 172.21.15.69:0/453473650' entity='mgr.y' 2024-04-14T14:01:07.567 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:01:07 smithi069 ceph-mon[29472]: from='mgr.14448 172.21.15.69:0/453473650' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T14:01:07.568 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:01:07 smithi069 ceph-mon[29472]: from='mgr.14448 172.21.15.69:0/453473650' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-14T14:01:07.568 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:01:07 smithi069 ceph-mon[29472]: from='mgr.14448 172.21.15.69:0/453473650' entity='mgr.y' 2024-04-14T14:01:07.665 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Apr 14 14:01:07 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-x[45487]: 2024-04-14T14:01:07.406+0000 7f4697f7f200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-14T14:01:07.665 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Apr 14 14:01:07 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-x[45487]: 2024-04-14T14:01:07.503+0000 7f4697f7f200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-14T14:01:07.665 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:01:07 smithi122 ceph-mon[25721]: pgmap v208: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:01:07.665 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:01:07 smithi122 ceph-mon[25721]: from='mgr.14448 172.21.15.69:0/453473650' entity='mgr.y' 2024-04-14T14:01:07.665 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:01:07 smithi122 ceph-mon[25721]: from='mgr.14448 172.21.15.69:0/453473650' entity='mgr.y' 2024-04-14T14:01:07.665 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:01:07 smithi122 ceph-mon[25721]: from='mgr.14448 172.21.15.69:0/453473650' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T14:01:07.666 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:01:07 smithi122 ceph-mon[25721]: from='mgr.14448 172.21.15.69:0/453473650' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-14T14:01:07.666 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:01:07 smithi122 ceph-mon[25721]: from='mgr.14448 172.21.15.69:0/453473650' entity='mgr.y' 2024-04-14T14:01:08.345 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Apr 14 14:01:08 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-x[45487]: 2024-04-14T14:01:08.190+0000 7f4697f7f200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-14T14:01:08.345 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Apr 14 14:01:08 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-x[45487]: 2024-04-14T14:01:08.290+0000 7f4697f7f200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-14T14:01:08.636 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Apr 14 14:01:08 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-x[45487]: 2024-04-14T14:01:08.443+0000 7f4697f7f200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-14T14:01:08.637 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Apr 14 14:01:08 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-x[45487]: 2024-04-14T14:01:08.546+0000 7f4697f7f200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-14T14:01:08.954 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Apr 14 14:01:08 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-x[45487]: 2024-04-14T14:01:08.635+0000 7f4697f7f200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-14T14:01:08.954 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Apr 14 14:01:08 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-x[45487]: 2024-04-14T14:01:08.952+0000 7f4697f7f200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-14T14:01:09.270 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Apr 14 14:01:09 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-x[45487]: 2024-04-14T14:01:09.074+0000 7f4697f7f200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-14T14:01:09.270 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Apr 14 14:01:09 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-x[45487]: 2024-04-14T14:01:09.163+0000 7f4697f7f200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-14T14:01:09.271 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Apr 14 14:01:09 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-x[45487]: 2024-04-14T14:01:09.268+0000 7f4697f7f200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-14T14:01:09.594 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:01:09 smithi122 ceph-mon[25721]: pgmap v209: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:01:09.595 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Apr 14 14:01:09 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-x[45487]: 2024-04-14T14:01:09.454+0000 7f4697f7f200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-14T14:01:09.780 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:01:09 smithi069 ceph-mon[24818]: pgmap v209: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:01:09.780 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 14 14:01:09 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-y[25054]: ::ffff:172.21.15.122 - - [14/Apr/2024:14:01:09] "GET /metrics HTTP/1.1" 200 207983 "" "Prometheus/2.33.4" 2024-04-14T14:01:09.780 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:01:09 smithi069 ceph-mon[29472]: pgmap v209: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:01:10.829 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Apr 14 14:01:10 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-x[45487]: 2024-04-14T14:01:10.474+0000 7f4697f7f200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-14T14:01:10.829 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Apr 14 14:01:10 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-x[45487]: 2024-04-14T14:01:10.608+0000 7f4697f7f200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-14T14:01:11.095 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Apr 14 14:01:10 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-x[45487]: 2024-04-14T14:01:10.951+0000 7f4697f7f200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-14T14:01:11.095 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Apr 14 14:01:11 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-x[45487]: 2024-04-14T14:01:11.042+0000 7f4697f7f200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-14T14:01:11.594 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:01:11 smithi122 ceph-mon[25721]: pgmap v210: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:01:11.595 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:01:11 smithi122 ceph-mon[25721]: from='mgr.14448 172.21.15.69:0/453473650' entity='mgr.y' 2024-04-14T14:01:11.595 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:01:11 smithi122 ceph-mon[25721]: from='mgr.14448 172.21.15.69:0/453473650' entity='mgr.y' 2024-04-14T14:01:11.595 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:01:11 smithi122 ceph-mon[25721]: from='mgr.14448 172.21.15.69:0/453473650' entity='mgr.y' 2024-04-14T14:01:11.595 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Apr 14 14:01:11 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-x[45487]: 2024-04-14T14:01:11.249+0000 7f4697f7f200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-14T14:01:11.595 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Apr 14 14:01:11 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-x[45487]: 2024-04-14T14:01:11.436+0000 7f4697f7f200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-14T14:01:11.817 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:01:11 smithi069 ceph-mon[24818]: pgmap v210: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:01:11.817 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:01:11 smithi069 ceph-mon[24818]: from='mgr.14448 172.21.15.69:0/453473650' entity='mgr.y' 2024-04-14T14:01:11.817 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:01:11 smithi069 ceph-mon[24818]: from='mgr.14448 172.21.15.69:0/453473650' entity='mgr.y' 2024-04-14T14:01:11.817 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:01:11 smithi069 ceph-mon[24818]: from='mgr.14448 172.21.15.69:0/453473650' entity='mgr.y' 2024-04-14T14:01:11.817 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:01:11 smithi069 ceph-mon[29472]: pgmap v210: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:01:11.817 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:01:11 smithi069 ceph-mon[29472]: from='mgr.14448 172.21.15.69:0/453473650' entity='mgr.y' 2024-04-14T14:01:11.817 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:01:11 smithi069 ceph-mon[29472]: from='mgr.14448 172.21.15.69:0/453473650' entity='mgr.y' 2024-04-14T14:01:11.817 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:01:11 smithi069 ceph-mon[29472]: from='mgr.14448 172.21.15.69:0/453473650' entity='mgr.y' 2024-04-14T14:01:11.985 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Apr 14 14:01:11 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-x[45487]: 2024-04-14T14:01:11.726+0000 7f4697f7f200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-14T14:01:12.344 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Apr 14 14:01:11 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-x[45487]: 2024-04-14T14:01:11.983+0000 7f4697f7f200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-14T14:01:12.345 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Apr 14 14:01:12 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-x[45487]: 2024-04-14T14:01:12.224+0000 7f4697f7f200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-14T14:01:13.344 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Apr 14 14:01:13 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-x[45487]: 2024-04-14T14:01:13.029+0000 7f4697f7f200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-14T14:01:13.733 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:01:13 smithi069 ceph-mon[24818]: pgmap v211: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:01:13.733 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:01:13 smithi069 ceph-mon[29472]: pgmap v211: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:01:13.734 INFO:journalctl@ceph.alertmanager.a.smithi069.stdout:Apr 14 14:01:13 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-alertmanager-a[49465]: level=warn ts=2024-04-14T14:01:13.506Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.122:8443/api/prometheus_receiver\": dial tcp 172.21.15.122:8443: connect: connection refused" 2024-04-14T14:01:13.734 INFO:journalctl@ceph.alertmanager.a.smithi069.stdout:Apr 14 14:01:13 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-alertmanager-a[49465]: level=warn ts=2024-04-14T14:01:13.519Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.69:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.69 because it doesn't contain any IP SANs" 2024-04-14T14:01:13.844 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:01:13 smithi122 ceph-mon[25721]: pgmap v211: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:01:13.845 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Apr 14 14:01:13 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-x[45487]: 2024-04-14T14:01:13.600+0000 7f4697f7f200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-14T14:01:13.845 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Apr 14 14:01:13 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-x[45487]: 2024-04-14T14:01:13.700+0000 7f4697f7f200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-14T14:01:14.317 INFO:journalctl@ceph.alertmanager.a.smithi069.stdout:Apr 14 14:01:14 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-alertmanager-a[49465]: level=warn ts=2024-04-14T14:01:14.044Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=2 err="Post \"https://172.21.15.122:8443/api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.122 because it doesn't contain any IP SANs" 2024-04-14T14:01:14.345 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Apr 14 14:01:13 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-x[45487]: 2024-04-14T14:01:13.918+0000 7f4697f7f200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-14T14:01:14.345 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Apr 14 14:01:13 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-x[45487]: [14/Apr/2024:14:01:13] ENGINE Bus STARTING 2024-04-14T14:01:14.345 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Apr 14 14:01:14 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-x[45487]: CherryPy Checker: 2024-04-14T14:01:14.345 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Apr 14 14:01:14 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-x[45487]: The Application mounted at '' has an empty config. 2024-04-14T14:01:14.345 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Apr 14 14:01:14 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-x[45487]: 2024-04-14T14:01:14.345 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Apr 14 14:01:14 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-x[45487]: [14/Apr/2024:14:01:14] ENGINE Serving on http://:::9283 2024-04-14T14:01:14.345 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Apr 14 14:01:14 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-x[45487]: [14/Apr/2024:14:01:14] ENGINE Bus STARTED 2024-04-14T14:01:14.771 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:01:14 smithi069 ceph-mon[24818]: Standby manager daemon x restarted 2024-04-14T14:01:14.771 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:01:14 smithi069 ceph-mon[24818]: Standby manager daemon x started 2024-04-14T14:01:14.772 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:01:14 smithi069 ceph-mon[24818]: from='mgr.? 172.21.15.122:0/1892077634' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-04-14T14:01:14.772 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:01:14 smithi069 ceph-mon[24818]: from='mgr.? 172.21.15.122:0/1892077634' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-14T14:01:14.772 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:01:14 smithi069 ceph-mon[24818]: from='mgr.? 172.21.15.122:0/1892077634' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-04-14T14:01:14.772 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:01:14 smithi069 ceph-mon[24818]: from='mgr.? 172.21.15.122:0/1892077634' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-14T14:01:14.772 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:01:14 smithi069 ceph-mon[29472]: Standby manager daemon x restarted 2024-04-14T14:01:14.772 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:01:14 smithi069 ceph-mon[29472]: Standby manager daemon x started 2024-04-14T14:01:14.772 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:01:14 smithi069 ceph-mon[29472]: from='mgr.? 172.21.15.122:0/1892077634' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-04-14T14:01:14.772 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:01:14 smithi069 ceph-mon[29472]: from='mgr.? 172.21.15.122:0/1892077634' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-14T14:01:14.772 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:01:14 smithi069 ceph-mon[29472]: from='mgr.? 172.21.15.122:0/1892077634' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-04-14T14:01:14.772 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:01:14 smithi069 ceph-mon[29472]: from='mgr.? 172.21.15.122:0/1892077634' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-14T14:01:14.844 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:01:14 smithi122 ceph-mon[25721]: Standby manager daemon x restarted 2024-04-14T14:01:14.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:01:14 smithi122 ceph-mon[25721]: Standby manager daemon x started 2024-04-14T14:01:14.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:01:14 smithi122 ceph-mon[25721]: from='mgr.? 172.21.15.122:0/1892077634' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-04-14T14:01:14.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:01:14 smithi122 ceph-mon[25721]: from='mgr.? 172.21.15.122:0/1892077634' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-14T14:01:14.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:01:14 smithi122 ceph-mon[25721]: from='mgr.? 172.21.15.122:0/1892077634' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-04-14T14:01:14.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:01:14 smithi122 ceph-mon[25721]: from='mgr.? 172.21.15.122:0/1892077634' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-14T14:01:15.817 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:01:15 smithi069 ceph-mon[29472]: from='client.14766 -' entity='client.iscsi.foo.smithi069.yuumtr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T14:01:15.817 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:01:15 smithi069 ceph-mon[29472]: pgmap v212: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:01:15.817 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:01:15 smithi069 ceph-mon[29472]: mgrmap e21: y(active, since 5m), standbys: x 2024-04-14T14:01:15.817 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:01:15 smithi069 ceph-mon[24818]: from='client.14766 -' entity='client.iscsi.foo.smithi069.yuumtr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T14:01:15.817 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:01:15 smithi069 ceph-mon[24818]: pgmap v212: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:01:15.817 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:01:15 smithi069 ceph-mon[24818]: mgrmap e21: y(active, since 5m), standbys: x 2024-04-14T14:01:15.844 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:01:15 smithi122 ceph-mon[25721]: from='client.14766 -' entity='client.iscsi.foo.smithi069.yuumtr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T14:01:15.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:01:15 smithi122 ceph-mon[25721]: pgmap v212: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:01:15.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:01:15 smithi122 ceph-mon[25721]: mgrmap e21: y(active, since 5m), standbys: x 2024-04-14T14:01:17.595 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:01:17 smithi122 ceph-mon[25721]: pgmap v213: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:01:17.595 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Apr 14 14:01:17 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-x[45487]: ::ffff:172.21.15.122 - - [14/Apr/2024:14:01:17] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-04-14T14:01:17.816 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:01:17 smithi069 ceph-mon[24818]: pgmap v213: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:01:17.817 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:01:17 smithi069 ceph-mon[29472]: pgmap v213: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:01:19.817 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:01:19 smithi069 ceph-mon[24818]: pgmap v214: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:01:19.817 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 14 14:01:19 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-y[25054]: ::ffff:172.21.15.122 - - [14/Apr/2024:14:01:19] "GET /metrics HTTP/1.1" 200 207983 "" "Prometheus/2.33.4" 2024-04-14T14:01:19.817 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:01:19 smithi069 ceph-mon[29472]: pgmap v214: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:01:19.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:01:19 smithi122 ceph-mon[25721]: pgmap v214: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:01:21.647 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:01:21 smithi122 ceph-mon[25721]: pgmap v215: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:01:21.816 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:01:21 smithi069 ceph-mon[24818]: pgmap v215: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:01:21.817 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:01:21 smithi069 ceph-mon[29472]: pgmap v215: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:01:23.744 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:01:23 smithi069 ceph-mon[24818]: pgmap v216: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:01:23.744 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:01:23 smithi069 ceph-mon[29472]: pgmap v216: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:01:23.744 INFO:journalctl@ceph.alertmanager.a.smithi069.stdout:Apr 14 14:01:23 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-alertmanager-a[49465]: level=error ts=2024-04-14T14:01:23.505Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=6 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://172.21.15.69:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.69 because it doesn't contain any IP SANs; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://172.21.15.122:8443/api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.122 because it doesn't contain any IP SANs" 2024-04-14T14:01:23.745 INFO:journalctl@ceph.alertmanager.a.smithi069.stdout:Apr 14 14:01:23 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-alertmanager-a[49465]: level=warn ts=2024-04-14T14:01:23.508Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.122:8443/api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.122 because it doesn't contain any IP SANs" 2024-04-14T14:01:23.745 INFO:journalctl@ceph.alertmanager.a.smithi069.stdout:Apr 14 14:01:23 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-alertmanager-a[49465]: level=warn ts=2024-04-14T14:01:23.508Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.69:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.69 because it doesn't contain any IP SANs" 2024-04-14T14:01:23.801 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:01:23 smithi122 ceph-mon[25721]: pgmap v216: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:01:24.567 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:01:24 smithi069 ceph-mon[24818]: from='mgr.14448 172.21.15.69:0/453473650' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-04-14T14:01:24.567 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:01:24 smithi069 ceph-mon[24818]: from='mgr.14448 172.21.15.69:0/453473650' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-04-14T14:01:24.567 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:01:24 smithi069 ceph-mon[29472]: from='mgr.14448 172.21.15.69:0/453473650' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-04-14T14:01:24.567 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:01:24 smithi069 ceph-mon[29472]: from='mgr.14448 172.21.15.69:0/453473650' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-04-14T14:01:24.844 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:01:24 smithi122 ceph-mon[25721]: from='mgr.14448 172.21.15.69:0/453473650' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-04-14T14:01:24.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:01:24 smithi122 ceph-mon[25721]: from='mgr.14448 172.21.15.69:0/453473650' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-04-14T14:01:25.817 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:01:25 smithi069 ceph-mon[24818]: from='client.14766 -' entity='client.iscsi.foo.smithi069.yuumtr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T14:01:25.817 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:01:25 smithi069 ceph-mon[24818]: pgmap v217: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:01:25.817 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:01:25 smithi069 ceph-mon[29472]: from='client.14766 -' entity='client.iscsi.foo.smithi069.yuumtr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T14:01:25.817 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:01:25 smithi069 ceph-mon[29472]: pgmap v217: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:01:25.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:01:25 smithi122 ceph-mon[25721]: from='client.14766 -' entity='client.iscsi.foo.smithi069.yuumtr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T14:01:25.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:01:25 smithi122 ceph-mon[25721]: pgmap v217: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:01:27.595 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:01:27 smithi122 ceph-mon[25721]: pgmap v218: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:01:27.595 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Apr 14 14:01:27 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-x[45487]: ::ffff:172.21.15.122 - - [14/Apr/2024:14:01:27] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-04-14T14:01:27.816 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:01:27 smithi069 ceph-mon[24818]: pgmap v218: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:01:27.817 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:01:27 smithi069 ceph-mon[29472]: pgmap v218: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:01:29.816 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:01:29 smithi069 ceph-mon[24818]: pgmap v219: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:01:29.817 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:01:29 smithi069 ceph-mon[29472]: pgmap v219: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:01:29.817 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 14 14:01:29 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-y[25054]: ::ffff:172.21.15.122 - - [14/Apr/2024:14:01:29] "GET /metrics HTTP/1.1" 200 208001 "" "Prometheus/2.33.4" 2024-04-14T14:01:29.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:01:29 smithi122 ceph-mon[25721]: pgmap v219: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:01:31.817 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:01:31 smithi069 ceph-mon[24818]: pgmap v220: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:01:31.817 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:01:31 smithi069 ceph-mon[29472]: pgmap v220: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:01:31.844 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:01:31 smithi122 ceph-mon[25721]: pgmap v220: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:01:33.595 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:01:33 smithi122 ceph-mon[25721]: pgmap v221: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:01:33.749 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:01:33 smithi069 ceph-mon[24818]: pgmap v221: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:01:33.750 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:01:33 smithi069 ceph-mon[29472]: pgmap v221: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:01:33.750 INFO:journalctl@ceph.alertmanager.a.smithi069.stdout:Apr 14 14:01:33 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-alertmanager-a[49465]: level=error ts=2024-04-14T14:01:33.507Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=6 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://172.21.15.122:8443/api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.122 because it doesn't contain any IP SANs; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://172.21.15.69:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.69 because it doesn't contain any IP SANs" 2024-04-14T14:01:33.750 INFO:journalctl@ceph.alertmanager.a.smithi069.stdout:Apr 14 14:01:33 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-alertmanager-a[49465]: level=warn ts=2024-04-14T14:01:33.509Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.69:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.69 because it doesn't contain any IP SANs" 2024-04-14T14:01:33.750 INFO:journalctl@ceph.alertmanager.a.smithi069.stdout:Apr 14 14:01:33 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-alertmanager-a[49465]: level=warn ts=2024-04-14T14:01:33.509Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.122:8443/api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.122 because it doesn't contain any IP SANs" 2024-04-14T14:01:35.817 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:01:35 smithi069 ceph-mon[24818]: from='client.14766 -' entity='client.iscsi.foo.smithi069.yuumtr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T14:01:35.817 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:01:35 smithi069 ceph-mon[24818]: pgmap v222: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:01:35.817 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:01:35 smithi069 ceph-mon[29472]: from='client.14766 -' entity='client.iscsi.foo.smithi069.yuumtr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T14:01:35.817 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:01:35 smithi069 ceph-mon[29472]: pgmap v222: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:01:35.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:01:35 smithi122 ceph-mon[25721]: from='client.14766 -' entity='client.iscsi.foo.smithi069.yuumtr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T14:01:35.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:01:35 smithi122 ceph-mon[25721]: pgmap v222: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:01:37.595 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:01:37 smithi122 ceph-mon[25721]: pgmap v223: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:01:37.595 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Apr 14 14:01:37 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-x[45487]: ::ffff:172.21.15.122 - - [14/Apr/2024:14:01:37] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-04-14T14:01:37.817 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:01:37 smithi069 ceph-mon[24818]: pgmap v223: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:01:37.817 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:01:37 smithi069 ceph-mon[29472]: pgmap v223: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:01:39.817 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:01:39 smithi069 ceph-mon[24818]: pgmap v224: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:01:39.817 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:01:39 smithi069 ceph-mon[29472]: pgmap v224: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:01:39.817 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 14 14:01:39 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-y[25054]: ::ffff:172.21.15.122 - - [14/Apr/2024:14:01:39] "GET /metrics HTTP/1.1" 200 207999 "" "Prometheus/2.33.4" 2024-04-14T14:01:39.844 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:01:39 smithi122 ceph-mon[25721]: pgmap v224: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:01:41.817 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:01:41 smithi069 ceph-mon[24818]: pgmap v225: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:01:41.817 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:01:41 smithi069 ceph-mon[29472]: pgmap v225: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:01:41.844 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:01:41 smithi122 ceph-mon[25721]: pgmap v225: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:01:43.752 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:01:43 smithi069 ceph-mon[24818]: pgmap v226: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:01:43.752 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:01:43 smithi069 ceph-mon[29472]: pgmap v226: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:01:43.753 INFO:journalctl@ceph.alertmanager.a.smithi069.stdout:Apr 14 14:01:43 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-alertmanager-a[49465]: level=error ts=2024-04-14T14:01:43.508Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=6 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://172.21.15.122:8443/api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.122 because it doesn't contain any IP SANs; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://172.21.15.69:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.69 because it doesn't contain any IP SANs" 2024-04-14T14:01:43.753 INFO:journalctl@ceph.alertmanager.a.smithi069.stdout:Apr 14 14:01:43 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-alertmanager-a[49465]: level=warn ts=2024-04-14T14:01:43.510Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.69:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.69 because it doesn't contain any IP SANs" 2024-04-14T14:01:43.753 INFO:journalctl@ceph.alertmanager.a.smithi069.stdout:Apr 14 14:01:43 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-alertmanager-a[49465]: level=warn ts=2024-04-14T14:01:43.510Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.122:8443/api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.122 because it doesn't contain any IP SANs" 2024-04-14T14:01:43.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:01:43 smithi122 ceph-mon[25721]: pgmap v226: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:01:45.817 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:01:45 smithi069 ceph-mon[24818]: from='client.14766 -' entity='client.iscsi.foo.smithi069.yuumtr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T14:01:45.817 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:01:45 smithi069 ceph-mon[24818]: pgmap v227: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:01:45.817 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:01:45 smithi069 ceph-mon[29472]: from='client.14766 -' entity='client.iscsi.foo.smithi069.yuumtr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T14:01:45.817 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:01:45 smithi069 ceph-mon[29472]: pgmap v227: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:01:45.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:01:45 smithi122 ceph-mon[25721]: from='client.14766 -' entity='client.iscsi.foo.smithi069.yuumtr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T14:01:45.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:01:45 smithi122 ceph-mon[25721]: pgmap v227: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:01:47.594 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:01:47 smithi122 ceph-mon[25721]: pgmap v228: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:01:47.595 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Apr 14 14:01:47 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-x[45487]: ::ffff:172.21.15.122 - - [14/Apr/2024:14:01:47] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-04-14T14:01:47.817 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:01:47 smithi069 ceph-mon[24818]: pgmap v228: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:01:47.817 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:01:47 smithi069 ceph-mon[29472]: pgmap v228: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:01:49.817 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:01:49 smithi069 ceph-mon[24818]: pgmap v229: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:01:49.818 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 14 14:01:49 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-y[25054]: ::ffff:172.21.15.122 - - [14/Apr/2024:14:01:49] "GET /metrics HTTP/1.1" 200 207999 "" "Prometheus/2.33.4" 2024-04-14T14:01:49.818 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:01:49 smithi069 ceph-mon[29472]: pgmap v229: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:01:49.844 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:01:49 smithi122 ceph-mon[25721]: pgmap v229: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:01:51.647 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:01:51 smithi122 ceph-mon[25721]: pgmap v230: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:01:51.817 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:01:51 smithi069 ceph-mon[24818]: pgmap v230: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:01:51.817 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:01:51 smithi069 ceph-mon[29472]: pgmap v230: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:01:53.755 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:01:53 smithi069 ceph-mon[24818]: pgmap v231: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:01:53.756 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:01:53 smithi069 ceph-mon[29472]: pgmap v231: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:01:53.756 INFO:journalctl@ceph.alertmanager.a.smithi069.stdout:Apr 14 14:01:53 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-alertmanager-a[49465]: level=error ts=2024-04-14T14:01:53.509Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=6 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://172.21.15.69:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.69 because it doesn't contain any IP SANs; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://172.21.15.122:8443/api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.122 because it doesn't contain any IP SANs" 2024-04-14T14:01:53.756 INFO:journalctl@ceph.alertmanager.a.smithi069.stdout:Apr 14 14:01:53 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-alertmanager-a[49465]: level=warn ts=2024-04-14T14:01:53.511Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.69:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.69 because it doesn't contain any IP SANs" 2024-04-14T14:01:53.756 INFO:journalctl@ceph.alertmanager.a.smithi069.stdout:Apr 14 14:01:53 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-alertmanager-a[49465]: level=warn ts=2024-04-14T14:01:53.511Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.122:8443/api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.122 because it doesn't contain any IP SANs" 2024-04-14T14:01:53.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:01:53 smithi122 ceph-mon[25721]: pgmap v231: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:01:55.817 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:01:55 smithi069 ceph-mon[24818]: from='client.14766 -' entity='client.iscsi.foo.smithi069.yuumtr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T14:01:55.817 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:01:55 smithi069 ceph-mon[24818]: pgmap v232: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:01:55.817 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:01:55 smithi069 ceph-mon[29472]: from='client.14766 -' entity='client.iscsi.foo.smithi069.yuumtr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T14:01:55.817 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:01:55 smithi069 ceph-mon[29472]: pgmap v232: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:01:55.844 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:01:55 smithi122 ceph-mon[25721]: from='client.14766 -' entity='client.iscsi.foo.smithi069.yuumtr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T14:01:55.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:01:55 smithi122 ceph-mon[25721]: pgmap v232: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:01:57.595 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:01:57 smithi122 ceph-mon[25721]: pgmap v233: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:01:57.595 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Apr 14 14:01:57 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-x[45487]: ::ffff:172.21.15.122 - - [14/Apr/2024:14:01:57] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-04-14T14:01:57.817 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:01:57 smithi069 ceph-mon[24818]: pgmap v233: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:01:57.817 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:01:57 smithi069 ceph-mon[29472]: pgmap v233: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:01:59.817 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:01:59 smithi069 ceph-mon[24818]: pgmap v234: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:01:59.817 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 14 14:01:59 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-y[25054]: ::ffff:172.21.15.122 - - [14/Apr/2024:14:01:59] "GET /metrics HTTP/1.1" 200 208003 "" "Prometheus/2.33.4" 2024-04-14T14:01:59.817 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:01:59 smithi069 ceph-mon[29472]: pgmap v234: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:01:59.844 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:01:59 smithi122 ceph-mon[25721]: pgmap v234: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:02:01.817 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:02:01 smithi069 ceph-mon[24818]: pgmap v235: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:02:01.817 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:02:01 smithi069 ceph-mon[29472]: pgmap v235: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:02:01.844 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:02:01 smithi122 ceph-mon[25721]: pgmap v235: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:02:03.762 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:02:03 smithi069 ceph-mon[24818]: pgmap v236: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:02:03.763 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:02:03 smithi069 ceph-mon[29472]: pgmap v236: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:02:03.763 INFO:journalctl@ceph.alertmanager.a.smithi069.stdout:Apr 14 14:02:03 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-alertmanager-a[49465]: level=error ts=2024-04-14T14:02:03.509Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=6 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://172.21.15.69:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.69 because it doesn't contain any IP SANs; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://172.21.15.122:8443/api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.122 because it doesn't contain any IP SANs" 2024-04-14T14:02:03.763 INFO:journalctl@ceph.alertmanager.a.smithi069.stdout:Apr 14 14:02:03 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-alertmanager-a[49465]: level=warn ts=2024-04-14T14:02:03.511Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.69:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.69 because it doesn't contain any IP SANs" 2024-04-14T14:02:03.764 INFO:journalctl@ceph.alertmanager.a.smithi069.stdout:Apr 14 14:02:03 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-alertmanager-a[49465]: level=warn ts=2024-04-14T14:02:03.512Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.122:8443/api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.122 because it doesn't contain any IP SANs" 2024-04-14T14:02:03.844 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:02:03 smithi122 ceph-mon[25721]: pgmap v236: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:02:05.817 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:02:05 smithi069 ceph-mon[24818]: from='client.14766 -' entity='client.iscsi.foo.smithi069.yuumtr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T14:02:05.817 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:02:05 smithi069 ceph-mon[24818]: pgmap v237: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:02:05.817 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:02:05 smithi069 ceph-mon[29472]: from='client.14766 -' entity='client.iscsi.foo.smithi069.yuumtr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T14:02:05.817 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:02:05 smithi069 ceph-mon[29472]: pgmap v237: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:02:05.844 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:02:05 smithi122 ceph-mon[25721]: from='client.14766 -' entity='client.iscsi.foo.smithi069.yuumtr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T14:02:05.844 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:02:05 smithi122 ceph-mon[25721]: pgmap v237: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:02:07.595 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:02:07 smithi122 ceph-mon[25721]: pgmap v238: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:02:07.595 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Apr 14 14:02:07 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-x[45487]: ::ffff:172.21.15.122 - - [14/Apr/2024:14:02:07] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-04-14T14:02:07.817 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:02:07 smithi069 ceph-mon[24818]: pgmap v238: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:02:07.817 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:02:07 smithi069 ceph-mon[29472]: pgmap v238: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:02:09.816 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:02:09 smithi069 ceph-mon[24818]: pgmap v239: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:02:09.817 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 14 14:02:09 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-y[25054]: ::ffff:172.21.15.122 - - [14/Apr/2024:14:02:09] "GET /metrics HTTP/1.1" 200 208002 "" "Prometheus/2.33.4" 2024-04-14T14:02:09.817 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:02:09 smithi069 ceph-mon[29472]: pgmap v239: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:02:09.844 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:02:09 smithi122 ceph-mon[25721]: pgmap v239: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:02:11.515 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:02:11 smithi122 ceph-mon[25721]: pgmap v240: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:02:11.515 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:02:11 smithi122 ceph-mon[25721]: from='mgr.14448 172.21.15.69:0/453473650' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T14:02:11.515 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:02:11 smithi122 ceph-mon[25721]: from='mgr.14448 172.21.15.69:0/453473650' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-14T14:02:11.553 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:02:11 smithi069 ceph-mon[24818]: pgmap v240: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:02:11.553 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:02:11 smithi069 ceph-mon[24818]: from='mgr.14448 172.21.15.69:0/453473650' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T14:02:11.553 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:02:11 smithi069 ceph-mon[24818]: from='mgr.14448 172.21.15.69:0/453473650' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-14T14:02:11.553 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:02:11 smithi069 ceph-mon[29472]: pgmap v240: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:02:11.554 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:02:11 smithi069 ceph-mon[29472]: from='mgr.14448 172.21.15.69:0/453473650' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T14:02:11.554 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:02:11 smithi069 ceph-mon[29472]: from='mgr.14448 172.21.15.69:0/453473650' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-14T14:02:12.794 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:02:12 smithi122 ceph-mon[25721]: from='mgr.14448 172.21.15.69:0/453473650' entity='mgr.y' 2024-04-14T14:02:12.816 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:02:12 smithi069 ceph-mon[24818]: from='mgr.14448 172.21.15.69:0/453473650' entity='mgr.y' 2024-04-14T14:02:12.817 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:02:12 smithi069 ceph-mon[29472]: from='mgr.14448 172.21.15.69:0/453473650' entity='mgr.y' 2024-04-14T14:02:13.774 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:02:13 smithi069 ceph-mon[24818]: pgmap v241: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:02:13.774 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:02:13 smithi069 ceph-mon[29472]: pgmap v241: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:02:13.774 INFO:journalctl@ceph.alertmanager.a.smithi069.stdout:Apr 14 14:02:13 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-alertmanager-a[49465]: level=error ts=2024-04-14T14:02:13.510Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=6 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://172.21.15.69:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.69 because it doesn't contain any IP SANs; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://172.21.15.122:8443/api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.122 because it doesn't contain any IP SANs" 2024-04-14T14:02:13.774 INFO:journalctl@ceph.alertmanager.a.smithi069.stdout:Apr 14 14:02:13 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-alertmanager-a[49465]: level=warn ts=2024-04-14T14:02:13.512Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.122:8443/api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.122 because it doesn't contain any IP SANs" 2024-04-14T14:02:13.775 INFO:journalctl@ceph.alertmanager.a.smithi069.stdout:Apr 14 14:02:13 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-alertmanager-a[49465]: level=warn ts=2024-04-14T14:02:13.512Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.69:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.69 because it doesn't contain any IP SANs" 2024-04-14T14:02:13.830 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:02:13 smithi122 ceph-mon[25721]: pgmap v241: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:02:15.813 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:02:15 smithi122 ceph-mon[25721]: from='client.14766 -' entity='client.iscsi.foo.smithi069.yuumtr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T14:02:15.813 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:02:15 smithi122 ceph-mon[25721]: pgmap v242: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:02:15.813 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:02:15 smithi122 ceph-mon[25721]: from='mgr.14448 172.21.15.69:0/453473650' entity='mgr.y' 2024-04-14T14:02:15.816 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:02:15 smithi069 ceph-mon[24818]: from='client.14766 -' entity='client.iscsi.foo.smithi069.yuumtr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T14:02:15.816 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:02:15 smithi069 ceph-mon[24818]: pgmap v242: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:02:15.816 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:02:15 smithi069 ceph-mon[24818]: from='mgr.14448 172.21.15.69:0/453473650' entity='mgr.y' 2024-04-14T14:02:15.817 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:02:15 smithi069 ceph-mon[29472]: from='client.14766 -' entity='client.iscsi.foo.smithi069.yuumtr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T14:02:15.817 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:02:15 smithi069 ceph-mon[29472]: pgmap v242: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:02:15.817 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:02:15 smithi069 ceph-mon[29472]: from='mgr.14448 172.21.15.69:0/453473650' entity='mgr.y' 2024-04-14T14:02:17.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:02:16 smithi122 ceph-mon[25721]: from='mgr.14448 172.21.15.69:0/453473650' entity='mgr.y' 2024-04-14T14:02:17.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:02:16 smithi122 ceph-mon[25721]: pgmap v243: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:02:17.317 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:02:16 smithi069 ceph-mon[24818]: from='mgr.14448 172.21.15.69:0/453473650' entity='mgr.y' 2024-04-14T14:02:17.317 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:02:16 smithi069 ceph-mon[24818]: pgmap v243: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:02:17.317 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:02:16 smithi069 ceph-mon[29472]: from='mgr.14448 172.21.15.69:0/453473650' entity='mgr.y' 2024-04-14T14:02:17.317 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:02:16 smithi069 ceph-mon[29472]: pgmap v243: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:02:17.595 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Apr 14 14:02:17 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-x[45487]: ::ffff:172.21.15.122 - - [14/Apr/2024:14:02:17] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-04-14T14:02:19.817 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:02:19 smithi069 ceph-mon[24818]: pgmap v244: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:02:19.817 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 14 14:02:19 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-y[25054]: ::ffff:172.21.15.122 - - [14/Apr/2024:14:02:19] "GET /metrics HTTP/1.1" 200 208002 "" "Prometheus/2.33.4" 2024-04-14T14:02:19.817 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:02:19 smithi069 ceph-mon[29472]: pgmap v244: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:02:19.844 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:02:19 smithi122 ceph-mon[25721]: pgmap v244: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:02:21.647 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:02:21 smithi122 ceph-mon[25721]: pgmap v245: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:02:21.817 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:02:21 smithi069 ceph-mon[24818]: pgmap v245: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:02:21.817 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:02:21 smithi069 ceph-mon[29472]: pgmap v245: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:02:23.776 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:02:23 smithi069 ceph-mon[24818]: pgmap v246: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:02:23.777 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:02:23 smithi069 ceph-mon[29472]: pgmap v246: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:02:23.777 INFO:journalctl@ceph.alertmanager.a.smithi069.stdout:Apr 14 14:02:23 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-alertmanager-a[49465]: level=error ts=2024-04-14T14:02:23.511Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=6 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://172.21.15.122:8443/api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.122 because it doesn't contain any IP SANs; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://172.21.15.69:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.69 because it doesn't contain any IP SANs" 2024-04-14T14:02:23.777 INFO:journalctl@ceph.alertmanager.a.smithi069.stdout:Apr 14 14:02:23 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-alertmanager-a[49465]: level=warn ts=2024-04-14T14:02:23.513Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.69:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.69 because it doesn't contain any IP SANs" 2024-04-14T14:02:23.777 INFO:journalctl@ceph.alertmanager.a.smithi069.stdout:Apr 14 14:02:23 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-alertmanager-a[49465]: level=warn ts=2024-04-14T14:02:23.513Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.122:8443/api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.122 because it doesn't contain any IP SANs" 2024-04-14T14:02:23.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:02:23 smithi122 ceph-mon[25721]: pgmap v246: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:02:24.817 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:02:24 smithi069 ceph-mon[24818]: from='mgr.14448 172.21.15.69:0/453473650' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-04-14T14:02:24.817 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:02:24 smithi069 ceph-mon[24818]: from='mgr.14448 172.21.15.69:0/453473650' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-04-14T14:02:24.817 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:02:24 smithi069 ceph-mon[29472]: from='mgr.14448 172.21.15.69:0/453473650' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-04-14T14:02:24.817 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:02:24 smithi069 ceph-mon[29472]: from='mgr.14448 172.21.15.69:0/453473650' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-04-14T14:02:24.844 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:02:24 smithi122 ceph-mon[25721]: from='mgr.14448 172.21.15.69:0/453473650' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-04-14T14:02:24.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:02:24 smithi122 ceph-mon[25721]: from='mgr.14448 172.21.15.69:0/453473650' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-04-14T14:02:25.817 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:02:25 smithi069 ceph-mon[24818]: from='client.14766 -' entity='client.iscsi.foo.smithi069.yuumtr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T14:02:25.817 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:02:25 smithi069 ceph-mon[24818]: pgmap v247: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:02:25.817 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:02:25 smithi069 ceph-mon[29472]: from='client.14766 -' entity='client.iscsi.foo.smithi069.yuumtr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T14:02:25.817 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:02:25 smithi069 ceph-mon[29472]: pgmap v247: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:02:25.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:02:25 smithi122 ceph-mon[25721]: from='client.14766 -' entity='client.iscsi.foo.smithi069.yuumtr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T14:02:25.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:02:25 smithi122 ceph-mon[25721]: pgmap v247: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:02:27.595 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:02:27 smithi122 ceph-mon[25721]: pgmap v248: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:02:27.595 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Apr 14 14:02:27 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-x[45487]: ::ffff:172.21.15.122 - - [14/Apr/2024:14:02:27] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-04-14T14:02:27.816 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:02:27 smithi069 ceph-mon[24818]: pgmap v248: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:02:27.817 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:02:27 smithi069 ceph-mon[29472]: pgmap v248: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:02:29.817 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:02:29 smithi069 ceph-mon[24818]: pgmap v249: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:02:29.817 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 14 14:02:29 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-y[25054]: ::ffff:172.21.15.122 - - [14/Apr/2024:14:02:29] "GET /metrics HTTP/1.1" 200 208000 "" "Prometheus/2.33.4" 2024-04-14T14:02:29.817 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:02:29 smithi069 ceph-mon[29472]: pgmap v249: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:02:29.844 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:02:29 smithi122 ceph-mon[25721]: pgmap v249: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:02:31.817 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:02:31 smithi069 ceph-mon[24818]: pgmap v250: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:02:31.817 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:02:31 smithi069 ceph-mon[29472]: pgmap v250: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:02:31.844 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:02:31 smithi122 ceph-mon[25721]: pgmap v250: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:02:33.784 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:02:33 smithi069 ceph-mon[24818]: pgmap v251: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:02:33.784 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:02:33 smithi069 ceph-mon[29472]: pgmap v251: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:02:33.784 INFO:journalctl@ceph.alertmanager.a.smithi069.stdout:Apr 14 14:02:33 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-alertmanager-a[49465]: level=error ts=2024-04-14T14:02:33.512Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=6 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://172.21.15.122:8443/api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.122 because it doesn't contain any IP SANs; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://172.21.15.69:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.69 because it doesn't contain any IP SANs" 2024-04-14T14:02:33.785 INFO:journalctl@ceph.alertmanager.a.smithi069.stdout:Apr 14 14:02:33 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-alertmanager-a[49465]: level=warn ts=2024-04-14T14:02:33.514Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.69:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.69 because it doesn't contain any IP SANs" 2024-04-14T14:02:33.785 INFO:journalctl@ceph.alertmanager.a.smithi069.stdout:Apr 14 14:02:33 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-alertmanager-a[49465]: level=warn ts=2024-04-14T14:02:33.514Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.122:8443/api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.122 because it doesn't contain any IP SANs" 2024-04-14T14:02:33.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:02:33 smithi122 ceph-mon[25721]: pgmap v251: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:02:35.817 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:02:35 smithi069 ceph-mon[24818]: from='client.14766 -' entity='client.iscsi.foo.smithi069.yuumtr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T14:02:35.817 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:02:35 smithi069 ceph-mon[24818]: pgmap v252: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:02:35.817 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:02:35 smithi069 ceph-mon[29472]: from='client.14766 -' entity='client.iscsi.foo.smithi069.yuumtr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T14:02:35.817 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:02:35 smithi069 ceph-mon[29472]: pgmap v252: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:02:35.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:02:35 smithi122 ceph-mon[25721]: from='client.14766 -' entity='client.iscsi.foo.smithi069.yuumtr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T14:02:35.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:02:35 smithi122 ceph-mon[25721]: pgmap v252: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:02:37.594 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:02:37 smithi122 ceph-mon[25721]: pgmap v253: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:02:37.595 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Apr 14 14:02:37 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-x[45487]: ::ffff:172.21.15.122 - - [14/Apr/2024:14:02:37] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-04-14T14:02:37.816 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:02:37 smithi069 ceph-mon[24818]: pgmap v253: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:02:37.817 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:02:37 smithi069 ceph-mon[29472]: pgmap v253: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:02:39.816 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:02:39 smithi069 ceph-mon[24818]: pgmap v254: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:02:39.817 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 14 14:02:39 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-y[25054]: ::ffff:172.21.15.122 - - [14/Apr/2024:14:02:39] "GET /metrics HTTP/1.1" 200 207995 "" "Prometheus/2.33.4" 2024-04-14T14:02:39.817 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:02:39 smithi069 ceph-mon[29472]: pgmap v254: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:02:39.844 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:02:39 smithi122 ceph-mon[25721]: pgmap v254: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:02:41.817 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:02:41 smithi069 ceph-mon[24818]: pgmap v255: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:02:41.817 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:02:41 smithi069 ceph-mon[29472]: pgmap v255: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:02:41.844 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:02:41 smithi122 ceph-mon[25721]: pgmap v255: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:02:43.789 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:02:43 smithi069 ceph-mon[29472]: pgmap v256: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:02:43.789 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:02:43 smithi069 ceph-mon[24818]: pgmap v256: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:02:43.789 INFO:journalctl@ceph.alertmanager.a.smithi069.stdout:Apr 14 14:02:43 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-alertmanager-a[49465]: level=error ts=2024-04-14T14:02:43.512Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=6 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://172.21.15.122:8443/api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.122 because it doesn't contain any IP SANs; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://172.21.15.69:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.69 because it doesn't contain any IP SANs" 2024-04-14T14:02:43.790 INFO:journalctl@ceph.alertmanager.a.smithi069.stdout:Apr 14 14:02:43 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-alertmanager-a[49465]: level=warn ts=2024-04-14T14:02:43.514Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.69:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.69 because it doesn't contain any IP SANs" 2024-04-14T14:02:43.790 INFO:journalctl@ceph.alertmanager.a.smithi069.stdout:Apr 14 14:02:43 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-alertmanager-a[49465]: level=warn ts=2024-04-14T14:02:43.514Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.122:8443/api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.122 because it doesn't contain any IP SANs" 2024-04-14T14:02:43.844 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:02:43 smithi122 ceph-mon[25721]: pgmap v256: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:02:45.817 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:02:45 smithi069 ceph-mon[24818]: from='client.14766 -' entity='client.iscsi.foo.smithi069.yuumtr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T14:02:45.817 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:02:45 smithi069 ceph-mon[24818]: pgmap v257: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:02:45.817 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:02:45 smithi069 ceph-mon[29472]: from='client.14766 -' entity='client.iscsi.foo.smithi069.yuumtr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T14:02:45.817 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:02:45 smithi069 ceph-mon[29472]: pgmap v257: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:02:45.844 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:02:45 smithi122 ceph-mon[25721]: from='client.14766 -' entity='client.iscsi.foo.smithi069.yuumtr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T14:02:45.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:02:45 smithi122 ceph-mon[25721]: pgmap v257: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:02:47.594 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:02:47 smithi122 ceph-mon[25721]: pgmap v258: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:02:47.595 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Apr 14 14:02:47 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-x[45487]: ::ffff:172.21.15.122 - - [14/Apr/2024:14:02:47] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-04-14T14:02:47.816 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:02:47 smithi069 ceph-mon[24818]: pgmap v258: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:02:47.817 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:02:47 smithi069 ceph-mon[29472]: pgmap v258: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:02:49.817 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:02:49 smithi069 ceph-mon[24818]: pgmap v259: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:02:49.817 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 14 14:02:49 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-y[25054]: ::ffff:172.21.15.122 - - [14/Apr/2024:14:02:49] "GET /metrics HTTP/1.1" 200 207995 "" "Prometheus/2.33.4" 2024-04-14T14:02:49.817 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:02:49 smithi069 ceph-mon[29472]: pgmap v259: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:02:49.844 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:02:49 smithi122 ceph-mon[25721]: pgmap v259: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:02:51.647 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:02:51 smithi122 ceph-mon[25721]: pgmap v260: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:02:51.816 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:02:51 smithi069 ceph-mon[24818]: pgmap v260: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:02:51.817 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:02:51 smithi069 ceph-mon[29472]: pgmap v260: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:02:53.798 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:02:53 smithi069 ceph-mon[24818]: pgmap v261: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:02:53.799 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:02:53 smithi069 ceph-mon[29472]: pgmap v261: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:02:53.799 INFO:journalctl@ceph.alertmanager.a.smithi069.stdout:Apr 14 14:02:53 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-alertmanager-a[49465]: level=error ts=2024-04-14T14:02:53.513Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=6 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://172.21.15.122:8443/api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.122 because it doesn't contain any IP SANs; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://172.21.15.69:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.69 because it doesn't contain any IP SANs" 2024-04-14T14:02:53.799 INFO:journalctl@ceph.alertmanager.a.smithi069.stdout:Apr 14 14:02:53 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-alertmanager-a[49465]: level=warn ts=2024-04-14T14:02:53.515Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.69:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.69 because it doesn't contain any IP SANs" 2024-04-14T14:02:53.799 INFO:journalctl@ceph.alertmanager.a.smithi069.stdout:Apr 14 14:02:53 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-alertmanager-a[49465]: level=warn ts=2024-04-14T14:02:53.515Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.122:8443/api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.122 because it doesn't contain any IP SANs" 2024-04-14T14:02:53.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:02:53 smithi122 ceph-mon[25721]: pgmap v261: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:02:55.817 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:02:55 smithi069 ceph-mon[24818]: from='client.14766 -' entity='client.iscsi.foo.smithi069.yuumtr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T14:02:55.817 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:02:55 smithi069 ceph-mon[24818]: pgmap v262: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:02:55.817 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:02:55 smithi069 ceph-mon[29472]: from='client.14766 -' entity='client.iscsi.foo.smithi069.yuumtr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T14:02:55.817 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:02:55 smithi069 ceph-mon[29472]: pgmap v262: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:02:55.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:02:55 smithi122 ceph-mon[25721]: from='client.14766 -' entity='client.iscsi.foo.smithi069.yuumtr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T14:02:55.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:02:55 smithi122 ceph-mon[25721]: pgmap v262: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:02:57.594 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:02:57 smithi122 ceph-mon[25721]: pgmap v263: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:02:57.595 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Apr 14 14:02:57 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-x[45487]: ::ffff:172.21.15.122 - - [14/Apr/2024:14:02:57] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-04-14T14:02:57.816 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:02:57 smithi069 ceph-mon[24818]: pgmap v263: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:02:57.817 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:02:57 smithi069 ceph-mon[29472]: pgmap v263: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:02:59.817 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:02:59 smithi069 ceph-mon[24818]: pgmap v264: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:02:59.817 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 14 14:02:59 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-y[25054]: ::ffff:172.21.15.122 - - [14/Apr/2024:14:02:59] "GET /metrics HTTP/1.1" 200 208001 "" "Prometheus/2.33.4" 2024-04-14T14:02:59.817 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:02:59 smithi069 ceph-mon[29472]: pgmap v264: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:02:59.844 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:02:59 smithi122 ceph-mon[25721]: pgmap v264: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:03:01.817 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:03:01 smithi069 ceph-mon[24818]: pgmap v265: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:03:01.817 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:03:01 smithi069 ceph-mon[29472]: pgmap v265: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:03:01.844 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:03:01 smithi122 ceph-mon[25721]: pgmap v265: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:03:03.809 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:03:03 smithi069 ceph-mon[24818]: pgmap v266: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:03:03.809 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:03:03 smithi069 ceph-mon[29472]: pgmap v266: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:03:03.810 INFO:journalctl@ceph.alertmanager.a.smithi069.stdout:Apr 14 14:03:03 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-alertmanager-a[49465]: level=error ts=2024-04-14T14:03:03.514Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=6 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://172.21.15.69:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.69 because it doesn't contain any IP SANs; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://172.21.15.122:8443/api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.122 because it doesn't contain any IP SANs" 2024-04-14T14:03:03.810 INFO:journalctl@ceph.alertmanager.a.smithi069.stdout:Apr 14 14:03:03 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-alertmanager-a[49465]: level=warn ts=2024-04-14T14:03:03.516Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.69:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.69 because it doesn't contain any IP SANs" 2024-04-14T14:03:03.810 INFO:journalctl@ceph.alertmanager.a.smithi069.stdout:Apr 14 14:03:03 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-alertmanager-a[49465]: level=warn ts=2024-04-14T14:03:03.516Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.122:8443/api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.122 because it doesn't contain any IP SANs" 2024-04-14T14:03:03.844 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:03:03 smithi122 ceph-mon[25721]: pgmap v266: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:03:05.817 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:03:05 smithi069 ceph-mon[24818]: from='client.14766 -' entity='client.iscsi.foo.smithi069.yuumtr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T14:03:05.817 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:03:05 smithi069 ceph-mon[24818]: pgmap v267: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:03:05.817 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:03:05 smithi069 ceph-mon[29472]: from='client.14766 -' entity='client.iscsi.foo.smithi069.yuumtr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T14:03:05.817 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:03:05 smithi069 ceph-mon[29472]: pgmap v267: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:03:05.844 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:03:05 smithi122 ceph-mon[25721]: from='client.14766 -' entity='client.iscsi.foo.smithi069.yuumtr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T14:03:05.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:03:05 smithi122 ceph-mon[25721]: pgmap v267: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:03:07.595 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:03:07 smithi122 ceph-mon[25721]: pgmap v268: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:03:07.595 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Apr 14 14:03:07 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-x[45487]: ::ffff:172.21.15.122 - - [14/Apr/2024:14:03:07] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-04-14T14:03:07.817 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:03:07 smithi069 ceph-mon[24818]: pgmap v268: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:03:07.817 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:03:07 smithi069 ceph-mon[29472]: pgmap v268: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:03:09.817 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:03:09 smithi069 ceph-mon[24818]: pgmap v269: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:03:09.817 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 14 14:03:09 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-y[25054]: ::ffff:172.21.15.122 - - [14/Apr/2024:14:03:09] "GET /metrics HTTP/1.1" 200 207987 "" "Prometheus/2.33.4" 2024-04-14T14:03:09.817 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:03:09 smithi069 ceph-mon[29472]: pgmap v269: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:03:09.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:03:09 smithi122 ceph-mon[25721]: pgmap v269: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:03:11.816 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:03:11 smithi069 ceph-mon[24818]: pgmap v270: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:03:11.817 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:03:11 smithi069 ceph-mon[29472]: pgmap v270: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:03:11.844 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:03:11 smithi122 ceph-mon[25721]: pgmap v270: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:03:13.817 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:03:13 smithi069 ceph-mon[24818]: pgmap v271: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:03:13.817 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:03:13 smithi069 ceph-mon[29472]: pgmap v271: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:03:13.818 INFO:journalctl@ceph.alertmanager.a.smithi069.stdout:Apr 14 14:03:13 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-alertmanager-a[49465]: level=error ts=2024-04-14T14:03:13.515Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=6 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://172.21.15.122:8443/api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.122 because it doesn't contain any IP SANs; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://172.21.15.69:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.69 because it doesn't contain any IP SANs" 2024-04-14T14:03:13.818 INFO:journalctl@ceph.alertmanager.a.smithi069.stdout:Apr 14 14:03:13 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-alertmanager-a[49465]: level=warn ts=2024-04-14T14:03:13.518Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.69:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.69 because it doesn't contain any IP SANs" 2024-04-14T14:03:13.818 INFO:journalctl@ceph.alertmanager.a.smithi069.stdout:Apr 14 14:03:13 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-alertmanager-a[49465]: level=warn ts=2024-04-14T14:03:13.518Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.122:8443/api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.122 because it doesn't contain any IP SANs" 2024-04-14T14:03:13.844 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:03:13 smithi122 ceph-mon[25721]: pgmap v271: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:03:15.817 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:03:15 smithi069 ceph-mon[29472]: from='client.14766 -' entity='client.iscsi.foo.smithi069.yuumtr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T14:03:15.817 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:03:15 smithi069 ceph-mon[29472]: pgmap v272: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:03:15.817 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:03:15 smithi069 ceph-mon[24818]: from='client.14766 -' entity='client.iscsi.foo.smithi069.yuumtr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T14:03:15.817 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:03:15 smithi069 ceph-mon[24818]: pgmap v272: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:03:15.844 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:03:15 smithi122 ceph-mon[25721]: from='client.14766 -' entity='client.iscsi.foo.smithi069.yuumtr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T14:03:15.844 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:03:15 smithi122 ceph-mon[25721]: pgmap v272: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:03:16.817 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:03:16 smithi069 ceph-mon[24818]: from='mgr.14448 172.21.15.69:0/453473650' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T14:03:16.817 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:03:16 smithi069 ceph-mon[24818]: from='mgr.14448 172.21.15.69:0/453473650' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-14T14:03:16.817 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:03:16 smithi069 ceph-mon[24818]: from='mgr.14448 172.21.15.69:0/453473650' entity='mgr.y' 2024-04-14T14:03:16.817 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:03:16 smithi069 ceph-mon[29472]: from='mgr.14448 172.21.15.69:0/453473650' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T14:03:16.817 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:03:16 smithi069 ceph-mon[29472]: from='mgr.14448 172.21.15.69:0/453473650' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-14T14:03:16.817 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:03:16 smithi069 ceph-mon[29472]: from='mgr.14448 172.21.15.69:0/453473650' entity='mgr.y' 2024-04-14T14:03:16.844 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:03:16 smithi122 ceph-mon[25721]: from='mgr.14448 172.21.15.69:0/453473650' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T14:03:16.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:03:16 smithi122 ceph-mon[25721]: from='mgr.14448 172.21.15.69:0/453473650' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-14T14:03:16.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:03:16 smithi122 ceph-mon[25721]: from='mgr.14448 172.21.15.69:0/453473650' entity='mgr.y' 2024-04-14T14:03:17.594 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:03:17 smithi122 ceph-mon[25721]: pgmap v273: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:03:17.595 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Apr 14 14:03:17 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-x[45487]: ::ffff:172.21.15.122 - - [14/Apr/2024:14:03:17] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-04-14T14:03:17.816 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:03:17 smithi069 ceph-mon[24818]: pgmap v273: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:03:17.817 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:03:17 smithi069 ceph-mon[29472]: pgmap v273: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:03:19.816 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:03:19 smithi069 ceph-mon[24818]: pgmap v274: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:03:19.817 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:03:19 smithi069 ceph-mon[29472]: pgmap v274: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:03:19.817 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 14 14:03:19 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-y[25054]: ::ffff:172.21.15.122 - - [14/Apr/2024:14:03:19] "GET /metrics HTTP/1.1" 200 207987 "" "Prometheus/2.33.4" 2024-04-14T14:03:19.844 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:03:19 smithi122 ceph-mon[25721]: pgmap v274: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:03:21.647 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:03:21 smithi122 ceph-mon[25721]: pgmap v275: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:03:21.816 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:03:21 smithi069 ceph-mon[24818]: pgmap v275: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:03:21.817 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:03:21 smithi069 ceph-mon[29472]: pgmap v275: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:03:23.817 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:03:23 smithi069 ceph-mon[24818]: pgmap v276: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:03:23.817 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:03:23 smithi069 ceph-mon[29472]: pgmap v276: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:03:23.817 INFO:journalctl@ceph.alertmanager.a.smithi069.stdout:Apr 14 14:03:23 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-alertmanager-a[49465]: level=error ts=2024-04-14T14:03:23.515Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=6 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://172.21.15.69:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.69 because it doesn't contain any IP SANs; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://172.21.15.122:8443/api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.122 because it doesn't contain any IP SANs" 2024-04-14T14:03:23.817 INFO:journalctl@ceph.alertmanager.a.smithi069.stdout:Apr 14 14:03:23 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-alertmanager-a[49465]: level=warn ts=2024-04-14T14:03:23.517Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.69:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.69 because it doesn't contain any IP SANs" 2024-04-14T14:03:23.817 INFO:journalctl@ceph.alertmanager.a.smithi069.stdout:Apr 14 14:03:23 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-alertmanager-a[49465]: level=warn ts=2024-04-14T14:03:23.518Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.122:8443/api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.122 because it doesn't contain any IP SANs" 2024-04-14T14:03:23.844 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:03:23 smithi122 ceph-mon[25721]: pgmap v276: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:03:24.817 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:03:24 smithi069 ceph-mon[24818]: from='mgr.14448 172.21.15.69:0/453473650' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-04-14T14:03:24.817 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:03:24 smithi069 ceph-mon[24818]: from='mgr.14448 172.21.15.69:0/453473650' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-04-14T14:03:24.817 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:03:24 smithi069 ceph-mon[29472]: from='mgr.14448 172.21.15.69:0/453473650' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-04-14T14:03:24.817 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:03:24 smithi069 ceph-mon[29472]: from='mgr.14448 172.21.15.69:0/453473650' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-04-14T14:03:24.844 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:03:24 smithi122 ceph-mon[25721]: from='mgr.14448 172.21.15.69:0/453473650' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-04-14T14:03:24.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:03:24 smithi122 ceph-mon[25721]: from='mgr.14448 172.21.15.69:0/453473650' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-04-14T14:03:25.817 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:03:25 smithi069 ceph-mon[24818]: from='client.14766 -' entity='client.iscsi.foo.smithi069.yuumtr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T14:03:25.817 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:03:25 smithi069 ceph-mon[24818]: pgmap v277: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:03:25.817 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:03:25 smithi069 ceph-mon[29472]: from='client.14766 -' entity='client.iscsi.foo.smithi069.yuumtr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T14:03:25.817 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:03:25 smithi069 ceph-mon[29472]: pgmap v277: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:03:25.844 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:03:25 smithi122 ceph-mon[25721]: from='client.14766 -' entity='client.iscsi.foo.smithi069.yuumtr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T14:03:25.844 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:03:25 smithi122 ceph-mon[25721]: pgmap v277: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:03:27.517 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Apr 14 14:03:27 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-x[45487]: ::ffff:172.21.15.122 - - [14/Apr/2024:14:03:27] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-04-14T14:03:27.816 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:03:27 smithi069 ceph-mon[24818]: pgmap v278: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:03:27.817 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:03:27 smithi069 ceph-mon[29472]: pgmap v278: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:03:27.844 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:03:27 smithi122 ceph-mon[25721]: pgmap v278: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:03:29.817 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:03:29 smithi069 ceph-mon[24818]: pgmap v279: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:03:29.817 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 14 14:03:29 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-y[25054]: ::ffff:172.21.15.122 - - [14/Apr/2024:14:03:29] "GET /metrics HTTP/1.1" 200 208000 "" "Prometheus/2.33.4" 2024-04-14T14:03:29.817 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:03:29 smithi069 ceph-mon[29472]: pgmap v279: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:03:29.844 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:03:29 smithi122 ceph-mon[25721]: pgmap v279: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:03:31.817 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:03:31 smithi069 ceph-mon[24818]: pgmap v280: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:03:31.817 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:03:31 smithi069 ceph-mon[29472]: pgmap v280: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:03:31.844 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:03:31 smithi122 ceph-mon[25721]: pgmap v280: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:03:33.816 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:03:33 smithi069 ceph-mon[24818]: pgmap v281: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:03:33.817 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:03:33 smithi069 ceph-mon[29472]: pgmap v281: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:03:33.817 INFO:journalctl@ceph.alertmanager.a.smithi069.stdout:Apr 14 14:03:33 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-alertmanager-a[49465]: level=error ts=2024-04-14T14:03:33.517Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=6 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://172.21.15.69:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.69 because it doesn't contain any IP SANs; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://172.21.15.122:8443/api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.122 because it doesn't contain any IP SANs" 2024-04-14T14:03:33.817 INFO:journalctl@ceph.alertmanager.a.smithi069.stdout:Apr 14 14:03:33 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-alertmanager-a[49465]: level=warn ts=2024-04-14T14:03:33.519Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.69:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.69 because it doesn't contain any IP SANs" 2024-04-14T14:03:33.817 INFO:journalctl@ceph.alertmanager.a.smithi069.stdout:Apr 14 14:03:33 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-alertmanager-a[49465]: level=warn ts=2024-04-14T14:03:33.519Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.122:8443/api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.122 because it doesn't contain any IP SANs" 2024-04-14T14:03:33.844 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:03:33 smithi122 ceph-mon[25721]: pgmap v281: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:03:35.816 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:03:35 smithi069 ceph-mon[24818]: from='client.14766 -' entity='client.iscsi.foo.smithi069.yuumtr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T14:03:35.817 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:03:35 smithi069 ceph-mon[24818]: pgmap v282: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:03:35.817 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:03:35 smithi069 ceph-mon[29472]: from='client.14766 -' entity='client.iscsi.foo.smithi069.yuumtr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T14:03:35.817 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:03:35 smithi069 ceph-mon[29472]: pgmap v282: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:03:35.844 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:03:35 smithi122 ceph-mon[25721]: from='client.14766 -' entity='client.iscsi.foo.smithi069.yuumtr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T14:03:35.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:03:35 smithi122 ceph-mon[25721]: pgmap v282: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:03:37.516 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Apr 14 14:03:37 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-x[45487]: ::ffff:172.21.15.122 - - [14/Apr/2024:14:03:37] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-04-14T14:03:37.816 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:03:37 smithi069 ceph-mon[24818]: pgmap v283: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:03:37.817 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:03:37 smithi069 ceph-mon[29472]: pgmap v283: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:03:37.844 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:03:37 smithi122 ceph-mon[25721]: pgmap v283: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:03:39.817 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:03:39 smithi069 ceph-mon[24818]: pgmap v284: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:03:39.817 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 14 14:03:39 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-y[25054]: ::ffff:172.21.15.122 - - [14/Apr/2024:14:03:39] "GET /metrics HTTP/1.1" 200 208000 "" "Prometheus/2.33.4" 2024-04-14T14:03:39.817 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:03:39 smithi069 ceph-mon[29472]: pgmap v284: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:03:39.844 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:03:39 smithi122 ceph-mon[25721]: pgmap v284: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:03:41.816 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:03:41 smithi069 ceph-mon[24818]: pgmap v285: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:03:41.817 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:03:41 smithi069 ceph-mon[29472]: pgmap v285: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:03:41.844 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:03:41 smithi122 ceph-mon[25721]: pgmap v285: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:03:41.936 DEBUG:teuthology.orchestra.run.smithi069:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid dc326fc2-fa65-11ee-bc8e-c7b262605968 -e sha1=a971fd5f5719038c70b817bafacd6d5400b1d888 -- bash -c 'ceph orch ps' 2024-04-14T14:03:43.817 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:03:43 smithi069 ceph-mon[24818]: pgmap v286: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:03:43.817 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:03:43 smithi069 ceph-mon[29472]: pgmap v286: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:03:43.817 INFO:journalctl@ceph.alertmanager.a.smithi069.stdout:Apr 14 14:03:43 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-alertmanager-a[49465]: level=error ts=2024-04-14T14:03:43.518Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=6 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://172.21.15.69:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.69 because it doesn't contain any IP SANs; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://172.21.15.122:8443/api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.122 because it doesn't contain any IP SANs" 2024-04-14T14:03:43.818 INFO:journalctl@ceph.alertmanager.a.smithi069.stdout:Apr 14 14:03:43 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-alertmanager-a[49465]: level=warn ts=2024-04-14T14:03:43.520Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.69:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.69 because it doesn't contain any IP SANs" 2024-04-14T14:03:43.818 INFO:journalctl@ceph.alertmanager.a.smithi069.stdout:Apr 14 14:03:43 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-alertmanager-a[49465]: level=warn ts=2024-04-14T14:03:43.520Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.122:8443/api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.122 because it doesn't contain any IP SANs" 2024-04-14T14:03:43.842 INFO:teuthology.orchestra.run.smithi069.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-14T14:03:43.842 INFO:teuthology.orchestra.run.smithi069.stdout:alertmanager.a smithi069 *:9093,9094 running (6m) 2m ago 7m 24.4M - ba2b418f427c 148fbcb55232 2024-04-14T14:03:43.842 INFO:teuthology.orchestra.run.smithi069.stdout:grafana.a smithi122 *:3000 running (6m) 88s ago 6m 54.7M - 8.3.5 dad864ee21e9 6c0d806d20ea 2024-04-14T14:03:43.842 INFO:teuthology.orchestra.run.smithi069.stdout:iscsi.foo.smithi069.yuumtr smithi069 running (6m) 2m ago 6m 57.1M - 3.5 e1d6a67b021e 61bdcc2a5081 2024-04-14T14:03:43.842 INFO:teuthology.orchestra.run.smithi069.stdout:mgr.x smithi122 *:8443,9283 running (2m) 88s ago 11m 452M - 19.0.0-2778-ga971fd5f b96c509eef84 813744c2b615 2024-04-14T14:03:43.842 INFO:teuthology.orchestra.run.smithi069.stdout:mgr.y smithi069 *:9283 running (13m) 2m ago 13m 460M - 17.2.0 e1d6a67b021e 52e1363a8f98 2024-04-14T14:03:43.842 INFO:teuthology.orchestra.run.smithi069.stdout:mon.a smithi069 running (13m) 2m ago 13m 50.1M 2048M 17.2.0 e1d6a67b021e d3c553f91913 2024-04-14T14:03:43.842 INFO:teuthology.orchestra.run.smithi069.stdout:mon.b smithi122 running (11m) 88s ago 11m 42.3M 2048M 17.2.0 e1d6a67b021e 3b766bfce4be 2024-04-14T14:03:43.842 INFO:teuthology.orchestra.run.smithi069.stdout:mon.c smithi069 running (12m) 2m ago 12m 36.8M 2048M 17.2.0 e1d6a67b021e 23c078b9d1e5 2024-04-14T14:03:43.842 INFO:teuthology.orchestra.run.smithi069.stdout:node-exporter.a smithi069 *:9100 running (7m) 2m ago 7m 17.8M - 1dbe0e931976 73e900b404df 2024-04-14T14:03:43.843 INFO:teuthology.orchestra.run.smithi069.stdout:node-exporter.b smithi122 *:9100 running (7m) 88s ago 7m 17.1M - 1dbe0e931976 e57f3d5931fb 2024-04-14T14:03:43.843 INFO:teuthology.orchestra.run.smithi069.stdout:osd.0 smithi069 running (11m) 2m ago 11m 52.4M 4016M 17.2.0 e1d6a67b021e 46654654b6b6 2024-04-14T14:03:43.843 INFO:teuthology.orchestra.run.smithi069.stdout:osd.1 smithi069 running (10m) 2m ago 10m 49.6M 4016M 17.2.0 e1d6a67b021e cdba54a59609 2024-04-14T14:03:43.843 INFO:teuthology.orchestra.run.smithi069.stdout:osd.2 smithi069 running (10m) 2m ago 10m 47.0M 4016M 17.2.0 e1d6a67b021e bc37a1a9cba0 2024-04-14T14:03:43.843 INFO:teuthology.orchestra.run.smithi069.stdout:osd.3 smithi069 running (10m) 2m ago 10m 50.9M 4016M 17.2.0 e1d6a67b021e 0925ba3da801 2024-04-14T14:03:43.843 INFO:teuthology.orchestra.run.smithi069.stdout:osd.4 smithi122 running (9m) 88s ago 9m 49.8M 4272M 17.2.0 e1d6a67b021e c00975caf771 2024-04-14T14:03:43.843 INFO:teuthology.orchestra.run.smithi069.stdout:osd.5 smithi122 running (9m) 88s ago 9m 50.2M 4272M 17.2.0 e1d6a67b021e 5f52d217ceb8 2024-04-14T14:03:43.843 INFO:teuthology.orchestra.run.smithi069.stdout:osd.6 smithi122 running (9m) 88s ago 9m 45.4M 4272M 17.2.0 e1d6a67b021e a2acfec61475 2024-04-14T14:03:43.843 INFO:teuthology.orchestra.run.smithi069.stdout:osd.7 smithi122 running (8m) 88s ago 8m 51.4M 4272M 17.2.0 e1d6a67b021e 62411cd06801 2024-04-14T14:03:43.843 INFO:teuthology.orchestra.run.smithi069.stdout:prometheus.a smithi122 *:9095 running (6m) 88s ago 8m 64.2M - 514e6a882f6e 59e61af03fdf 2024-04-14T14:03:43.843 INFO:teuthology.orchestra.run.smithi069.stdout:rgw.foo.smithi069.wesedz smithi069 *:8000 running (6m) 2m ago 6m 72.6M - 17.2.0 e1d6a67b021e 16de491440f3 2024-04-14T14:03:43.843 INFO:teuthology.orchestra.run.smithi069.stdout:rgw.foo.smithi122.jiobyv smithi122 *:8000 running (6m) 88s ago 6m 69.9M - 17.2.0 e1d6a67b021e 6d16de445dba 2024-04-14T14:03:43.844 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:03:43 smithi122 ceph-mon[25721]: pgmap v286: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:03:44.470 DEBUG:teuthology.orchestra.run.smithi069:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid dc326fc2-fa65-11ee-bc8e-c7b262605968 -e sha1=a971fd5f5719038c70b817bafacd6d5400b1d888 -- bash -c 'ceph versions' 2024-04-14T14:03:45.772 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:03:45 smithi069 ceph-mon[24818]: from='client.15012 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T14:03:45.772 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:03:45 smithi069 ceph-mon[24818]: from='client.14766 -' entity='client.iscsi.foo.smithi069.yuumtr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T14:03:45.773 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:03:45 smithi069 ceph-mon[24818]: pgmap v287: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:03:45.773 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:03:45 smithi069 ceph-mon[29472]: from='client.15012 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T14:03:45.773 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:03:45 smithi069 ceph-mon[29472]: from='client.14766 -' entity='client.iscsi.foo.smithi069.yuumtr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T14:03:45.773 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:03:45 smithi069 ceph-mon[29472]: pgmap v287: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:03:45.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:03:45 smithi122 ceph-mon[25721]: from='client.15012 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T14:03:45.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:03:45 smithi122 ceph-mon[25721]: from='client.14766 -' entity='client.iscsi.foo.smithi069.yuumtr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T14:03:45.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:03:45 smithi122 ceph-mon[25721]: pgmap v287: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:03:46.396 INFO:teuthology.orchestra.run.smithi069.stdout:{ 2024-04-14T14:03:46.396 INFO:teuthology.orchestra.run.smithi069.stdout: "mon": { 2024-04-14T14:03:46.396 INFO:teuthology.orchestra.run.smithi069.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 3 2024-04-14T14:03:46.396 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-14T14:03:46.396 INFO:teuthology.orchestra.run.smithi069.stdout: "mgr": { 2024-04-14T14:03:46.397 INFO:teuthology.orchestra.run.smithi069.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 1, 2024-04-14T14:03:46.397 INFO:teuthology.orchestra.run.smithi069.stdout: "ceph version 19.0.0-2778-ga971fd5f (a971fd5f5719038c70b817bafacd6d5400b1d888) squid (dev)": 1 2024-04-14T14:03:46.397 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-14T14:03:46.397 INFO:teuthology.orchestra.run.smithi069.stdout: "osd": { 2024-04-14T14:03:46.397 INFO:teuthology.orchestra.run.smithi069.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 8 2024-04-14T14:03:46.397 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-14T14:03:46.397 INFO:teuthology.orchestra.run.smithi069.stdout: "mds": {}, 2024-04-14T14:03:46.397 INFO:teuthology.orchestra.run.smithi069.stdout: "rgw": { 2024-04-14T14:03:46.397 INFO:teuthology.orchestra.run.smithi069.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-04-14T14:03:46.397 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-14T14:03:46.397 INFO:teuthology.orchestra.run.smithi069.stdout: "overall": { 2024-04-14T14:03:46.397 INFO:teuthology.orchestra.run.smithi069.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 14, 2024-04-14T14:03:46.397 INFO:teuthology.orchestra.run.smithi069.stdout: "ceph version 19.0.0-2778-ga971fd5f (a971fd5f5719038c70b817bafacd6d5400b1d888) squid (dev)": 1 2024-04-14T14:03:46.397 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-14T14:03:46.397 INFO:teuthology.orchestra.run.smithi069.stdout:} 2024-04-14T14:03:46.815 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:03:46 smithi069 ceph-mon[29472]: from='client.? 172.21.15.69:0/3317582490' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T14:03:46.816 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:03:46 smithi069 ceph-mon[24818]: from='client.? 172.21.15.69:0/3317582490' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T14:03:46.844 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:03:46 smithi122 ceph-mon[25721]: from='client.? 172.21.15.69:0/3317582490' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T14:03:46.990 DEBUG:teuthology.orchestra.run.smithi069:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid dc326fc2-fa65-11ee-bc8e-c7b262605968 -e sha1=a971fd5f5719038c70b817bafacd6d5400b1d888 -- bash -c 'ceph -s' 2024-04-14T14:03:47.523 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Apr 14 14:03:47 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-x[45487]: ::ffff:172.21.15.122 - - [14/Apr/2024:14:03:47] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-04-14T14:03:47.550 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:03:47 smithi069 ceph-mon[24818]: pgmap v288: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:03:47.550 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:03:47 smithi069 ceph-mon[29472]: pgmap v288: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:03:47.844 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:03:47 smithi122 ceph-mon[25721]: pgmap v288: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:03:48.848 INFO:teuthology.orchestra.run.smithi069.stdout: cluster: 2024-04-14T14:03:48.848 INFO:teuthology.orchestra.run.smithi069.stdout: id: dc326fc2-fa65-11ee-bc8e-c7b262605968 2024-04-14T14:03:48.848 INFO:teuthology.orchestra.run.smithi069.stdout: health: HEALTH_OK 2024-04-14T14:03:48.848 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-14T14:03:48.848 INFO:teuthology.orchestra.run.smithi069.stdout: services: 2024-04-14T14:03:48.848 INFO:teuthology.orchestra.run.smithi069.stdout: mon: 3 daemons, quorum a,c,b (age 11m) 2024-04-14T14:03:48.849 INFO:teuthology.orchestra.run.smithi069.stdout: mgr: y(active, since 8m), standbys: x 2024-04-14T14:03:48.849 INFO:teuthology.orchestra.run.smithi069.stdout: osd: 8 osds: 8 up (since 8m), 8 in (since 8m) 2024-04-14T14:03:48.849 INFO:teuthology.orchestra.run.smithi069.stdout: rgw: 2 daemons active (2 hosts, 1 zones) 2024-04-14T14:03:48.849 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-14T14:03:48.849 INFO:teuthology.orchestra.run.smithi069.stdout: data: 2024-04-14T14:03:48.849 INFO:teuthology.orchestra.run.smithi069.stdout: pools: 6 pools, 161 pgs 2024-04-14T14:03:48.849 INFO:teuthology.orchestra.run.smithi069.stdout: objects: 209 objects, 585 KiB 2024-04-14T14:03:48.849 INFO:teuthology.orchestra.run.smithi069.stdout: usage: 63 MiB used, 715 GiB / 715 GiB avail 2024-04-14T14:03:48.849 INFO:teuthology.orchestra.run.smithi069.stdout: pgs: 161 active+clean 2024-04-14T14:03:48.849 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-14T14:03:48.849 INFO:teuthology.orchestra.run.smithi069.stdout: io: 2024-04-14T14:03:48.849 INFO:teuthology.orchestra.run.smithi069.stdout: client: 853 B/s rd, 0 op/s rd, 0 op/s wr 2024-04-14T14:03:48.849 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-14T14:03:49.446 DEBUG:teuthology.orchestra.run.smithi069:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid dc326fc2-fa65-11ee-bc8e-c7b262605968 -e sha1=a971fd5f5719038c70b817bafacd6d5400b1d888 -- bash -c 'ceph health detail' 2024-04-14T14:03:49.751 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 14 14:03:49 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-y[25054]: ::ffff:172.21.15.122 - - [14/Apr/2024:14:03:49] "GET /metrics HTTP/1.1" 200 208000 "" "Prometheus/2.33.4" 2024-04-14T14:03:49.751 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:03:49 smithi069 ceph-mon[24818]: pgmap v289: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:03:49.751 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:03:49 smithi069 ceph-mon[24818]: from='client.? 172.21.15.69:0/4133660984' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch 2024-04-14T14:03:49.752 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:03:49 smithi069 ceph-mon[29472]: pgmap v289: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:03:49.752 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:03:49 smithi069 ceph-mon[29472]: from='client.? 172.21.15.69:0/4133660984' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch 2024-04-14T14:03:49.844 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:03:49 smithi122 ceph-mon[25721]: pgmap v289: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:03:49.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:03:49 smithi122 ceph-mon[25721]: from='client.? 172.21.15.69:0/4133660984' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch 2024-04-14T14:03:51.436 INFO:teuthology.orchestra.run.smithi069.stdout:HEALTH_OK 2024-04-14T14:03:51.647 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:03:51 smithi122 ceph-mon[25721]: pgmap v290: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:03:51.817 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:03:51 smithi069 ceph-mon[24818]: pgmap v290: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:03:51.817 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:03:51 smithi069 ceph-mon[29472]: pgmap v290: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:03:52.090 DEBUG:teuthology.orchestra.run.smithi069:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid dc326fc2-fa65-11ee-bc8e-c7b262605968 -e sha1=a971fd5f5719038c70b817bafacd6d5400b1d888 -- bash -c 'ceph versions | jq -e '"'"'.mgr | length == 2'"'"'' 2024-04-14T14:03:52.703 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:03:52 smithi069 ceph-mon[24818]: from='client.? 172.21.15.69:0/2575095740' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-14T14:03:52.704 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:03:52 smithi069 ceph-mon[29472]: from='client.? 172.21.15.69:0/2575095740' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-14T14:03:52.844 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:03:52 smithi122 ceph-mon[25721]: from='client.? 172.21.15.69:0/2575095740' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-14T14:03:53.710 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:03:53 smithi069 ceph-mon[24818]: pgmap v291: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:03:53.710 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:03:53 smithi069 ceph-mon[29472]: pgmap v291: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:03:53.710 INFO:journalctl@ceph.alertmanager.a.smithi069.stdout:Apr 14 14:03:53 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-alertmanager-a[49465]: level=error ts=2024-04-14T14:03:53.519Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=6 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://172.21.15.122:8443/api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.122 because it doesn't contain any IP SANs; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://172.21.15.69:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.69 because it doesn't contain any IP SANs" 2024-04-14T14:03:53.710 INFO:journalctl@ceph.alertmanager.a.smithi069.stdout:Apr 14 14:03:53 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-alertmanager-a[49465]: level=warn ts=2024-04-14T14:03:53.521Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.69:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.69 because it doesn't contain any IP SANs" 2024-04-14T14:03:53.710 INFO:journalctl@ceph.alertmanager.a.smithi069.stdout:Apr 14 14:03:53 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-alertmanager-a[49465]: level=warn ts=2024-04-14T14:03:53.522Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.122:8443/api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.122 because it doesn't contain any IP SANs" 2024-04-14T14:03:53.844 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:03:53 smithi122 ceph-mon[25721]: pgmap v291: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:03:54.068 INFO:teuthology.orchestra.run.smithi069.stdout:true 2024-04-14T14:03:54.671 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:03:54 smithi069 ceph-mon[24818]: from='client.? 172.21.15.69:0/972194777' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T14:03:54.672 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:03:54 smithi069 ceph-mon[29472]: from='client.? 172.21.15.69:0/972194777' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T14:03:54.694 DEBUG:teuthology.orchestra.run.smithi069:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid dc326fc2-fa65-11ee-bc8e-c7b262605968 -e sha1=a971fd5f5719038c70b817bafacd6d5400b1d888 -- bash -c 'ceph mgr fail' 2024-04-14T14:03:54.846 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:03:54 smithi122 ceph-mon[25721]: from='client.? 172.21.15.69:0/972194777' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T14:03:55.721 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:03:55 smithi069 ceph-mon[24818]: from='client.14766 -' entity='client.iscsi.foo.smithi069.yuumtr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T14:03:55.721 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:03:55 smithi069 ceph-mon[24818]: pgmap v292: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:03:55.721 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:03:55 smithi069 ceph-mon[29472]: from='client.14766 -' entity='client.iscsi.foo.smithi069.yuumtr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T14:03:55.722 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:03:55 smithi069 ceph-mon[29472]: pgmap v292: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:03:55.844 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:03:55 smithi122 ceph-mon[25721]: from='client.14766 -' entity='client.iscsi.foo.smithi069.yuumtr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T14:03:55.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:03:55 smithi122 ceph-mon[25721]: pgmap v292: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:03:57.524 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Apr 14 14:03:57 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-x[45487]: ::ffff:172.21.15.122 - - [14/Apr/2024:14:03:57] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-04-14T14:03:57.816 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:03:57 smithi069 ceph-mon[24818]: pgmap v293: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:03:57.817 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:03:57 smithi069 ceph-mon[24818]: from='client.? 172.21.15.69:0/887386008' entity='client.admin' cmd=[{"prefix": "mgr fail"}]: dispatch 2024-04-14T14:03:57.817 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:03:57 smithi069 ceph-mon[24818]: osdmap e94: 8 total, 8 up, 8 in 2024-04-14T14:03:57.817 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 14 14:03:57 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-y[25054]: 2024-04-14T14:03:57.519+0000 7fa8d1f97700 -1 mgr handle_mgr_map I was active but no longer am 2024-04-14T14:03:57.817 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 14 14:03:57 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-y[25054]: ignoring --setuser ceph since I am not root 2024-04-14T14:03:57.817 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 14 14:03:57 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-y[25054]: ignoring --setgroup ceph since I am not root 2024-04-14T14:03:57.817 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 14 14:03:57 smithi069 ceph-mgr[25058]: -- 172.21.15.69:0/4161795014 <== mon.0 v2:172.21.15.69:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55ba049e4340 con 0x55ba049bd000 2024-04-14T14:03:57.817 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:03:57 smithi069 ceph-mon[29472]: pgmap v293: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:03:57.818 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:03:57 smithi069 ceph-mon[29472]: from='client.? 172.21.15.69:0/887386008' entity='client.admin' cmd=[{"prefix": "mgr fail"}]: dispatch 2024-04-14T14:03:57.818 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:03:57 smithi069 ceph-mon[29472]: osdmap e94: 8 total, 8 up, 8 in 2024-04-14T14:03:57.844 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:03:57 smithi122 ceph-mon[25721]: pgmap v293: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:03:57.844 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:03:57 smithi122 ceph-mon[25721]: from='client.? 172.21.15.69:0/887386008' entity='client.admin' cmd=[{"prefix": "mgr fail"}]: dispatch 2024-04-14T14:03:57.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:03:57 smithi122 ceph-mon[25721]: osdmap e94: 8 total, 8 up, 8 in 2024-04-14T14:03:57.845 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Apr 14 14:03:57 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-x[45487]: [14/Apr/2024:14:03:57] ENGINE Bus STOPPING 2024-04-14T14:03:57.845 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Apr 14 14:03:57 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-x[45487]: [14/Apr/2024:14:03:57] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-04-14T14:03:57.845 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Apr 14 14:03:57 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-x[45487]: [14/Apr/2024:14:03:57] ENGINE Bus STOPPED 2024-04-14T14:03:57.845 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Apr 14 14:03:57 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-x[45487]: [14/Apr/2024:14:03:57] ENGINE Bus STARTING 2024-04-14T14:03:57.845 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Apr 14 14:03:57 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-x[45487]: [14/Apr/2024:14:03:57] ENGINE Serving on http://:::9283 2024-04-14T14:03:57.845 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Apr 14 14:03:57 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-x[45487]: [14/Apr/2024:14:03:57] ENGINE Bus STARTED 2024-04-14T14:03:58.244 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 14 14:03:58 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-y[25054]: 2024-04-14T14:03:58.058+0000 7f7470793000 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-14T14:03:58.245 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 14 14:03:58 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-y[25054]: 2024-04-14T14:03:58.155+0000 7f7470793000 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-14T14:03:58.269 DEBUG:teuthology.orchestra.run.smithi069:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid dc326fc2-fa65-11ee-bc8e-c7b262605968 -e sha1=a971fd5f5719038c70b817bafacd6d5400b1d888 -- bash -c 'sleep 180' 2024-04-14T14:03:58.815 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 14 14:03:58 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-y[25054]: 2024-04-14T14:03:58.773+0000 7f7470793000 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-14T14:03:58.815 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:03:58 smithi069 ceph-mon[24818]: from='client.? 172.21.15.69:0/887386008' entity='client.admin' cmd='[{"prefix": "mgr fail"}]': finished 2024-04-14T14:03:58.815 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:03:58 smithi069 ceph-mon[24818]: mgrmap e22: x(active, starting, since 0.830937s) 2024-04-14T14:03:58.815 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:03:58 smithi069 ceph-mon[24818]: from='mgr.24749 172.21.15.122:0/1892077634' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-14T14:03:58.815 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:03:58 smithi069 ceph-mon[24818]: from='mgr.24749 172.21.15.122:0/1892077634' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-14T14:03:58.816 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:03:58 smithi069 ceph-mon[24818]: from='mgr.24749 172.21.15.122:0/1892077634' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-14T14:03:58.816 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:03:58 smithi069 ceph-mon[24818]: from='mgr.24749 172.21.15.122:0/1892077634' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-04-14T14:03:58.816 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:03:58 smithi069 ceph-mon[24818]: from='mgr.24749 172.21.15.122:0/1892077634' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-14T14:03:58.816 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:03:58 smithi069 ceph-mon[24818]: from='mgr.24749 172.21.15.122:0/1892077634' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-14T14:03:58.816 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:03:58 smithi069 ceph-mon[24818]: from='mgr.24749 172.21.15.122:0/1892077634' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-14T14:03:58.816 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:03:58 smithi069 ceph-mon[24818]: from='mgr.24749 172.21.15.122:0/1892077634' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-14T14:03:58.816 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:03:58 smithi069 ceph-mon[24818]: from='mgr.24749 172.21.15.122:0/1892077634' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-14T14:03:58.816 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:03:58 smithi069 ceph-mon[24818]: from='mgr.24749 172.21.15.122:0/1892077634' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-14T14:03:58.816 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:03:58 smithi069 ceph-mon[24818]: from='mgr.24749 172.21.15.122:0/1892077634' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-14T14:03:58.816 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:03:58 smithi069 ceph-mon[24818]: from='mgr.24749 172.21.15.122:0/1892077634' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-14T14:03:58.816 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:03:58 smithi069 ceph-mon[24818]: from='mgr.24749 172.21.15.122:0/1892077634' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-04-14T14:03:58.816 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:03:58 smithi069 ceph-mon[24818]: from='mgr.24749 172.21.15.122:0/1892077634' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-04-14T14:03:58.816 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:03:58 smithi069 ceph-mon[24818]: from='mgr.24749 172.21.15.122:0/1892077634' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-04-14T14:03:58.816 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:03:58 smithi069 ceph-mon[24818]: Manager daemon x is now available 2024-04-14T14:03:58.817 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:03:58 smithi069 ceph-mon[24818]: from='mgr.24749 ' entity='mgr.x' 2024-04-14T14:03:58.817 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:03:58 smithi069 ceph-mon[24818]: Queued rgw.foo for migration 2024-04-14T14:03:58.817 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:03:58 smithi069 ceph-mon[24818]: No Migration is needed for rgw spec: {'placement': {'count': 2}, 'service_id': 'foo', 'service_name': 'rgw.foo', 'service_type': 'rgw', 'spec': {'rgw_frontend_port': 8000, 'rgw_realm': 'r', 'rgw_zone': 'z'}} 2024-04-14T14:03:58.817 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:03:58 smithi069 ceph-mon[24818]: from='mgr.24749 ' entity='mgr.x' 2024-04-14T14:03:58.817 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:03:58 smithi069 ceph-mon[24818]: from='mgr.24749 172.21.15.122:0/1892077634' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T14:03:58.817 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:03:58 smithi069 ceph-mon[24818]: from='mgr.24749 172.21.15.122:0/1892077634' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-04-14T14:03:58.817 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:03:58 smithi069 ceph-mon[24818]: from='mgr.24749 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-04-14T14:03:58.817 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:03:58 smithi069 ceph-mon[24818]: from='mgr.24749 172.21.15.122:0/1892077634' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-04-14T14:03:58.817 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:03:58 smithi069 ceph-mon[24818]: from='mgr.24749 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-04-14T14:03:58.817 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:03:58 smithi069 ceph-mon[24818]: from='mgr.24749 ' entity='mgr.x' 2024-04-14T14:03:58.817 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:03:58 smithi069 ceph-mon[24818]: from='mgr.24749 ' entity='mgr.x' 2024-04-14T14:03:58.818 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:03:58 smithi069 ceph-mon[29472]: from='client.? 172.21.15.69:0/887386008' entity='client.admin' cmd='[{"prefix": "mgr fail"}]': finished 2024-04-14T14:03:58.818 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:03:58 smithi069 ceph-mon[29472]: mgrmap e22: x(active, starting, since 0.830937s) 2024-04-14T14:03:58.818 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:03:58 smithi069 ceph-mon[29472]: from='mgr.24749 172.21.15.122:0/1892077634' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-14T14:03:58.818 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:03:58 smithi069 ceph-mon[29472]: from='mgr.24749 172.21.15.122:0/1892077634' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-14T14:03:58.818 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:03:58 smithi069 ceph-mon[29472]: from='mgr.24749 172.21.15.122:0/1892077634' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-14T14:03:58.818 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:03:58 smithi069 ceph-mon[29472]: from='mgr.24749 172.21.15.122:0/1892077634' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-04-14T14:03:58.818 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:03:58 smithi069 ceph-mon[29472]: from='mgr.24749 172.21.15.122:0/1892077634' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-14T14:03:58.818 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:03:58 smithi069 ceph-mon[29472]: from='mgr.24749 172.21.15.122:0/1892077634' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-14T14:03:58.818 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:03:58 smithi069 ceph-mon[29472]: from='mgr.24749 172.21.15.122:0/1892077634' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-14T14:03:58.818 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:03:58 smithi069 ceph-mon[29472]: from='mgr.24749 172.21.15.122:0/1892077634' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-14T14:03:58.818 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:03:58 smithi069 ceph-mon[29472]: from='mgr.24749 172.21.15.122:0/1892077634' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-14T14:03:58.818 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:03:58 smithi069 ceph-mon[29472]: from='mgr.24749 172.21.15.122:0/1892077634' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-14T14:03:58.818 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:03:58 smithi069 ceph-mon[29472]: from='mgr.24749 172.21.15.122:0/1892077634' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-14T14:03:58.819 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:03:58 smithi069 ceph-mon[29472]: from='mgr.24749 172.21.15.122:0/1892077634' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-14T14:03:58.819 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:03:58 smithi069 ceph-mon[29472]: from='mgr.24749 172.21.15.122:0/1892077634' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-04-14T14:03:58.819 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:03:58 smithi069 ceph-mon[29472]: from='mgr.24749 172.21.15.122:0/1892077634' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-04-14T14:03:58.819 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:03:58 smithi069 ceph-mon[29472]: from='mgr.24749 172.21.15.122:0/1892077634' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-04-14T14:03:58.819 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:03:58 smithi069 ceph-mon[29472]: Manager daemon x is now available 2024-04-14T14:03:58.819 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:03:58 smithi069 ceph-mon[29472]: from='mgr.24749 ' entity='mgr.x' 2024-04-14T14:03:58.819 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:03:58 smithi069 ceph-mon[29472]: Queued rgw.foo for migration 2024-04-14T14:03:58.819 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:03:58 smithi069 ceph-mon[29472]: No Migration is needed for rgw spec: {'placement': {'count': 2}, 'service_id': 'foo', 'service_name': 'rgw.foo', 'service_type': 'rgw', 'spec': {'rgw_frontend_port': 8000, 'rgw_realm': 'r', 'rgw_zone': 'z'}} 2024-04-14T14:03:58.819 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:03:58 smithi069 ceph-mon[29472]: from='mgr.24749 ' entity='mgr.x' 2024-04-14T14:03:58.819 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:03:58 smithi069 ceph-mon[29472]: from='mgr.24749 172.21.15.122:0/1892077634' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T14:03:58.819 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:03:58 smithi069 ceph-mon[29472]: from='mgr.24749 172.21.15.122:0/1892077634' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-04-14T14:03:58.819 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:03:58 smithi069 ceph-mon[29472]: from='mgr.24749 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-04-14T14:03:58.819 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:03:58 smithi069 ceph-mon[29472]: from='mgr.24749 172.21.15.122:0/1892077634' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-04-14T14:03:58.819 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:03:58 smithi069 ceph-mon[29472]: from='mgr.24749 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-04-14T14:03:58.819 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:03:58 smithi069 ceph-mon[29472]: from='mgr.24749 ' entity='mgr.x' 2024-04-14T14:03:58.819 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:03:58 smithi069 ceph-mon[29472]: from='mgr.24749 ' entity='mgr.x' 2024-04-14T14:03:58.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:03:58 smithi122 ceph-mon[25721]: from='client.? 172.21.15.69:0/887386008' entity='client.admin' cmd='[{"prefix": "mgr fail"}]': finished 2024-04-14T14:03:58.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:03:58 smithi122 ceph-mon[25721]: mgrmap e22: x(active, starting, since 0.830937s) 2024-04-14T14:03:58.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:03:58 smithi122 ceph-mon[25721]: from='mgr.24749 172.21.15.122:0/1892077634' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-14T14:03:58.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:03:58 smithi122 ceph-mon[25721]: from='mgr.24749 172.21.15.122:0/1892077634' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-14T14:03:58.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:03:58 smithi122 ceph-mon[25721]: from='mgr.24749 172.21.15.122:0/1892077634' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-14T14:03:58.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:03:58 smithi122 ceph-mon[25721]: from='mgr.24749 172.21.15.122:0/1892077634' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-04-14T14:03:58.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:03:58 smithi122 ceph-mon[25721]: from='mgr.24749 172.21.15.122:0/1892077634' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-14T14:03:58.846 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:03:58 smithi122 ceph-mon[25721]: from='mgr.24749 172.21.15.122:0/1892077634' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-14T14:03:58.846 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:03:58 smithi122 ceph-mon[25721]: from='mgr.24749 172.21.15.122:0/1892077634' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-14T14:03:58.846 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:03:58 smithi122 ceph-mon[25721]: from='mgr.24749 172.21.15.122:0/1892077634' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-14T14:03:58.846 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:03:58 smithi122 ceph-mon[25721]: from='mgr.24749 172.21.15.122:0/1892077634' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-14T14:03:58.846 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:03:58 smithi122 ceph-mon[25721]: from='mgr.24749 172.21.15.122:0/1892077634' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-14T14:03:58.846 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:03:58 smithi122 ceph-mon[25721]: from='mgr.24749 172.21.15.122:0/1892077634' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-14T14:03:58.846 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:03:58 smithi122 ceph-mon[25721]: from='mgr.24749 172.21.15.122:0/1892077634' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-14T14:03:58.846 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:03:58 smithi122 ceph-mon[25721]: from='mgr.24749 172.21.15.122:0/1892077634' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-04-14T14:03:58.846 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:03:58 smithi122 ceph-mon[25721]: from='mgr.24749 172.21.15.122:0/1892077634' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-04-14T14:03:58.846 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:03:58 smithi122 ceph-mon[25721]: from='mgr.24749 172.21.15.122:0/1892077634' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-04-14T14:03:58.846 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:03:58 smithi122 ceph-mon[25721]: Manager daemon x is now available 2024-04-14T14:03:58.846 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:03:58 smithi122 ceph-mon[25721]: from='mgr.24749 ' entity='mgr.x' 2024-04-14T14:03:58.846 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:03:58 smithi122 ceph-mon[25721]: Queued rgw.foo for migration 2024-04-14T14:03:58.846 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:03:58 smithi122 ceph-mon[25721]: No Migration is needed for rgw spec: {'placement': {'count': 2}, 'service_id': 'foo', 'service_name': 'rgw.foo', 'service_type': 'rgw', 'spec': {'rgw_frontend_port': 8000, 'rgw_realm': 'r', 'rgw_zone': 'z'}} 2024-04-14T14:03:58.847 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:03:58 smithi122 ceph-mon[25721]: from='mgr.24749 ' entity='mgr.x' 2024-04-14T14:03:58.847 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:03:58 smithi122 ceph-mon[25721]: from='mgr.24749 172.21.15.122:0/1892077634' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T14:03:58.847 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:03:58 smithi122 ceph-mon[25721]: from='mgr.24749 172.21.15.122:0/1892077634' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-04-14T14:03:58.847 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:03:58 smithi122 ceph-mon[25721]: from='mgr.24749 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-04-14T14:03:58.847 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:03:58 smithi122 ceph-mon[25721]: from='mgr.24749 172.21.15.122:0/1892077634' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-04-14T14:03:58.847 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:03:58 smithi122 ceph-mon[25721]: from='mgr.24749 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-04-14T14:03:58.847 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:03:58 smithi122 ceph-mon[25721]: from='mgr.24749 ' entity='mgr.x' 2024-04-14T14:03:58.847 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:03:58 smithi122 ceph-mon[25721]: from='mgr.24749 ' entity='mgr.x' 2024-04-14T14:03:59.066 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 14 14:03:58 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-y[25054]: 2024-04-14T14:03:58.869+0000 7f7470793000 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-14T14:03:59.066 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 14 14:03:58 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-y[25054]: 2024-04-14T14:03:58.953+0000 7f7470793000 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-14T14:03:59.362 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 14 14:03:59 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-y[25054]: 2024-04-14T14:03:59.106+0000 7f7470793000 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-14T14:03:59.362 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 14 14:03:59 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-y[25054]: 2024-04-14T14:03:59.187+0000 7f7470793000 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-14T14:03:59.362 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 14 14:03:59 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-y[25054]: 2024-04-14T14:03:59.296+0000 7f7470793000 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-14T14:03:59.635 INFO:journalctl@ceph.alertmanager.a.smithi069.stdout:Apr 14 14:03:59 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-alertmanager-a[49465]: level=warn ts=2024-04-14T14:03:59.361Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=6 err="Post \"https://172.21.15.69:8443//api/prometheus_receiver\": dial tcp 172.21.15.69:8443: connect: connection refused" 2024-04-14T14:03:59.635 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:03:59 smithi069 ceph-mon[24818]: Deploying cephadm binary to smithi122 2024-04-14T14:03:59.635 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:03:59 smithi069 ceph-mon[24818]: Deploying cephadm binary to smithi069 2024-04-14T14:03:59.635 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:03:59 smithi069 ceph-mon[24818]: mgrmap e23: x(active, since 1.84727s) 2024-04-14T14:03:59.635 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:03:59 smithi069 ceph-mon[24818]: pgmap v3: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail 2024-04-14T14:03:59.636 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 14 14:03:59 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-y[25054]: 2024-04-14T14:03:59.380+0000 7f7470793000 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-14T14:03:59.636 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 14 14:03:59 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-y[25054]: 2024-04-14T14:03:59.553+0000 7f7470793000 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-14T14:03:59.636 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:03:59 smithi069 ceph-mon[29472]: Deploying cephadm binary to smithi122 2024-04-14T14:03:59.636 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:03:59 smithi069 ceph-mon[29472]: Deploying cephadm binary to smithi069 2024-04-14T14:03:59.636 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:03:59 smithi069 ceph-mon[29472]: mgrmap e23: x(active, since 1.84727s) 2024-04-14T14:03:59.636 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:03:59 smithi069 ceph-mon[29472]: pgmap v3: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail 2024-04-14T14:03:59.844 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:03:59 smithi122 ceph-mon[25721]: Deploying cephadm binary to smithi122 2024-04-14T14:03:59.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:03:59 smithi122 ceph-mon[25721]: Deploying cephadm binary to smithi069 2024-04-14T14:03:59.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:03:59 smithi122 ceph-mon[25721]: mgrmap e23: x(active, since 1.84727s) 2024-04-14T14:03:59.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:03:59 smithi122 ceph-mon[25721]: pgmap v3: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail 2024-04-14T14:03:59.904 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 14 14:03:59 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-y[25054]: 2024-04-14T14:03:59.641+0000 7f7470793000 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-14T14:04:00.812 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:04:00 smithi069 ceph-mon[24818]: pgmap v4: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail 2024-04-14T14:04:00.812 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:04:00 smithi069 ceph-mon[24818]: [14/Apr/2024:14:03:59] ENGINE Bus STARTING 2024-04-14T14:04:00.812 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 14 14:04:00 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-y[25054]: 2024-04-14T14:04:00.680+0000 7f7470793000 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-14T14:04:00.812 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:04:00 smithi069 ceph-mon[29472]: pgmap v4: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail 2024-04-14T14:04:00.812 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:04:00 smithi069 ceph-mon[29472]: [14/Apr/2024:14:03:59] ENGINE Bus STARTING 2024-04-14T14:04:00.844 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:04:00 smithi122 ceph-mon[25721]: pgmap v4: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail 2024-04-14T14:04:00.844 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:04:00 smithi122 ceph-mon[25721]: [14/Apr/2024:14:03:59] ENGINE Bus STARTING 2024-04-14T14:04:01.066 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 14 14:04:00 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-y[25054]: 2024-04-14T14:04:00.810+0000 7f7470793000 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-14T14:04:01.066 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 14 14:04:01 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-y[25054]: 2024-04-14T14:04:01.001+0000 7f7470793000 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-14T14:04:01.551 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 14 14:04:01 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-y[25054]: 2024-04-14T14:04:01.089+0000 7f7470793000 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-14T14:04:01.552 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 14 14:04:01 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-y[25054]: 2024-04-14T14:04:01.227+0000 7f7470793000 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-14T14:04:01.816 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:04:01 smithi069 ceph-mon[24818]: [14/Apr/2024:14:03:59] ENGINE Serving on https://172.21.15.122:7150 2024-04-14T14:04:01.817 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:04:01 smithi069 ceph-mon[24818]: [14/Apr/2024:14:04:00] ENGINE Serving on http://172.21.15.122:8765 2024-04-14T14:04:01.817 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:04:01 smithi069 ceph-mon[24818]: [14/Apr/2024:14:04:00] ENGINE Bus STARTED 2024-04-14T14:04:01.817 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:04:01 smithi069 ceph-mon[24818]: mgrmap e24: x(active, since 3s) 2024-04-14T14:04:01.817 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 14 14:04:01 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-y[25054]: 2024-04-14T14:04:01.550+0000 7f7470793000 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-14T14:04:01.817 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 14 14:04:01 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-y[25054]: 2024-04-14T14:04:01.774+0000 7f7470793000 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-14T14:04:01.817 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:04:01 smithi069 ceph-mon[29472]: [14/Apr/2024:14:03:59] ENGINE Serving on https://172.21.15.122:7150 2024-04-14T14:04:01.817 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:04:01 smithi069 ceph-mon[29472]: [14/Apr/2024:14:04:00] ENGINE Serving on http://172.21.15.122:8765 2024-04-14T14:04:01.817 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:04:01 smithi069 ceph-mon[29472]: [14/Apr/2024:14:04:00] ENGINE Bus STARTED 2024-04-14T14:04:01.818 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:04:01 smithi069 ceph-mon[29472]: mgrmap e24: x(active, since 3s) 2024-04-14T14:04:01.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:04:01 smithi122 ceph-mon[25721]: [14/Apr/2024:14:03:59] ENGINE Serving on https://172.21.15.122:7150 2024-04-14T14:04:01.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:04:01 smithi122 ceph-mon[25721]: [14/Apr/2024:14:04:00] ENGINE Serving on http://172.21.15.122:8765 2024-04-14T14:04:01.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:04:01 smithi122 ceph-mon[25721]: [14/Apr/2024:14:04:00] ENGINE Bus STARTED 2024-04-14T14:04:01.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:04:01 smithi122 ceph-mon[25721]: mgrmap e24: x(active, since 3s) 2024-04-14T14:04:02.317 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 14 14:04:01 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-y[25054]: 2024-04-14T14:04:01.861+0000 7f7470793000 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-14T14:04:02.317 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 14 14:04:02 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-y[25054]: 2024-04-14T14:04:02.033+0000 7f7470793000 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-14T14:04:02.816 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:04:02 smithi069 ceph-mon[24818]: pgmap v5: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail 2024-04-14T14:04:02.817 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:04:02 smithi069 ceph-mon[29472]: pgmap v5: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail 2024-04-14T14:04:02.844 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:04:02 smithi122 ceph-mon[25721]: pgmap v5: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail 2024-04-14T14:04:03.290 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 14 14:04:02 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-y[25054]: 2024-04-14T14:04:02.969+0000 7f7470793000 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-14T14:04:03.290 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 14 14:04:03 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-y[25054]: 2024-04-14T14:04:03.288+0000 7f7470793000 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-14T14:04:03.553 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 14 14:04:03 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-y[25054]: 2024-04-14T14:04:03.394+0000 7f7470793000 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-14T14:04:03.553 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 14 14:04:03 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-y[25054]: 2024-04-14T14:04:03.551+0000 7f7470793000 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-14T14:04:03.553 INFO:journalctl@ceph.alertmanager.a.smithi069.stdout:Apr 14 14:04:03 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-alertmanager-a[49465]: level=error ts=2024-04-14T14:04:03.521Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=6 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://172.21.15.122:8443/api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.122 because it doesn't contain any IP SANs; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://172.21.15.69:8443//api/prometheus_receiver\": dial tcp 172.21.15.69:8443: connect: connection refused" 2024-04-14T14:04:03.553 INFO:journalctl@ceph.alertmanager.a.smithi069.stdout:Apr 14 14:04:03 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-alertmanager-a[49465]: level=warn ts=2024-04-14T14:04:03.521Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.69:8443//api/prometheus_receiver\": dial tcp 172.21.15.69:8443: connect: connection refused" 2024-04-14T14:04:03.553 INFO:journalctl@ceph.alertmanager.a.smithi069.stdout:Apr 14 14:04:03 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-alertmanager-a[49465]: level=warn ts=2024-04-14T14:04:03.523Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.122:8443/api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.122 because it doesn't contain any IP SANs" 2024-04-14T14:04:03.797 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:04:03 smithi122 ceph-mon[25721]: Standby manager daemon y started 2024-04-14T14:04:03.816 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:04:03 smithi069 ceph-mon[24818]: Standby manager daemon y started 2024-04-14T14:04:03.817 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:04:03 smithi069 ceph-mon[29472]: Standby manager daemon y started 2024-04-14T14:04:03.817 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 14 14:04:03 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-y[25054]: [14/Apr/2024:14:04:03] ENGINE Bus STARTING 2024-04-14T14:04:03.817 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 14 14:04:03 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-y[25054]: CherryPy Checker: 2024-04-14T14:04:03.817 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 14 14:04:03 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-y[25054]: The Application mounted at '' has an empty config. 2024-04-14T14:04:03.817 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 14 14:04:03 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-y[25054]: 2024-04-14T14:04:03.817 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 14 14:04:03 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-y[25054]: [14/Apr/2024:14:04:03] ENGINE Serving on http://:::9283 2024-04-14T14:04:03.817 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 14 14:04:03 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-y[25054]: [14/Apr/2024:14:04:03] ENGINE Bus STARTED 2024-04-14T14:04:04.475 INFO:journalctl@ceph.alertmanager.a.smithi069.stdout:Apr 14 14:04:04 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-alertmanager-a[49465]: level=warn ts=2024-04-14T14:04:04.203Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=2 err="Post \"https://172.21.15.69:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.69 because it doesn't contain any IP SANs" 2024-04-14T14:04:04.595 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:04:04 smithi122 ceph-mon[25721]: pgmap v6: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail 2024-04-14T14:04:04.595 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:04:04 smithi122 ceph-mon[25721]: from='mgr.? 172.21.15.69:0/3045798068' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-04-14T14:04:04.595 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:04:04 smithi122 ceph-mon[25721]: from='mgr.? 172.21.15.69:0/3045798068' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-14T14:04:04.595 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:04:04 smithi122 ceph-mon[25721]: mgrmap e25: x(active, since 6s), standbys: y 2024-04-14T14:04:04.595 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:04:04 smithi122 ceph-mon[25721]: from='mgr.? 172.21.15.69:0/3045798068' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-04-14T14:04:04.595 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:04:04 smithi122 ceph-mon[25721]: from='mgr.? 172.21.15.69:0/3045798068' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-14T14:04:04.595 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:04:04 smithi122 ceph-mon[25721]: from='mgr.24749 172.21.15.122:0/1892077634' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-04-14T14:04:04.595 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:04:04 smithi122 ceph-mon[25721]: from='client.14766 -' entity='client.iscsi.foo.smithi069.yuumtr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T14:04:04.817 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:04:04 smithi069 ceph-mon[24818]: pgmap v6: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail 2024-04-14T14:04:04.817 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:04:04 smithi069 ceph-mon[24818]: from='mgr.? 172.21.15.69:0/3045798068' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-04-14T14:04:04.817 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:04:04 smithi069 ceph-mon[24818]: from='mgr.? 172.21.15.69:0/3045798068' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-14T14:04:04.817 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:04:04 smithi069 ceph-mon[24818]: mgrmap e25: x(active, since 6s), standbys: y 2024-04-14T14:04:04.817 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:04:04 smithi069 ceph-mon[24818]: from='mgr.? 172.21.15.69:0/3045798068' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-04-14T14:04:04.817 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:04:04 smithi069 ceph-mon[24818]: from='mgr.? 172.21.15.69:0/3045798068' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-14T14:04:04.817 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:04:04 smithi069 ceph-mon[24818]: from='mgr.24749 172.21.15.122:0/1892077634' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-04-14T14:04:04.817 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:04:04 smithi069 ceph-mon[24818]: from='client.14766 -' entity='client.iscsi.foo.smithi069.yuumtr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T14:04:04.817 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:04:04 smithi069 ceph-mon[29472]: pgmap v6: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail 2024-04-14T14:04:04.817 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:04:04 smithi069 ceph-mon[29472]: from='mgr.? 172.21.15.69:0/3045798068' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-04-14T14:04:04.817 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:04:04 smithi069 ceph-mon[29472]: from='mgr.? 172.21.15.69:0/3045798068' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-14T14:04:04.818 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:04:04 smithi069 ceph-mon[29472]: mgrmap e25: x(active, since 6s), standbys: y 2024-04-14T14:04:04.818 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:04:04 smithi069 ceph-mon[29472]: from='mgr.? 172.21.15.69:0/3045798068' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-04-14T14:04:04.818 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:04:04 smithi069 ceph-mon[29472]: from='mgr.? 172.21.15.69:0/3045798068' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-14T14:04:04.818 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:04:04 smithi069 ceph-mon[29472]: from='mgr.24749 172.21.15.122:0/1892077634' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-04-14T14:04:04.818 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:04:04 smithi069 ceph-mon[29472]: from='client.14766 -' entity='client.iscsi.foo.smithi069.yuumtr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T14:04:06.703 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:04:06 smithi122 ceph-mon[25721]: pgmap v7: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 30 KiB/s rd, 0 B/s wr, 17 op/s 2024-04-14T14:04:06.703 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:04:06 smithi122 ceph-mon[25721]: from='mgr.24749 ' entity='mgr.x' 2024-04-14T14:04:06.703 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:04:06 smithi122 ceph-mon[25721]: from='mgr.24749 ' entity='mgr.x' 2024-04-14T14:04:06.816 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:04:06 smithi069 ceph-mon[24818]: pgmap v7: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 30 KiB/s rd, 0 B/s wr, 17 op/s 2024-04-14T14:04:06.817 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:04:06 smithi069 ceph-mon[24818]: from='mgr.24749 ' entity='mgr.x' 2024-04-14T14:04:06.817 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:04:06 smithi069 ceph-mon[24818]: from='mgr.24749 ' entity='mgr.x' 2024-04-14T14:04:06.817 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:04:06 smithi069 ceph-mon[29472]: pgmap v7: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 30 KiB/s rd, 0 B/s wr, 17 op/s 2024-04-14T14:04:06.817 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:04:06 smithi069 ceph-mon[29472]: from='mgr.24749 ' entity='mgr.x' 2024-04-14T14:04:06.817 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:04:06 smithi069 ceph-mon[29472]: from='mgr.24749 ' entity='mgr.x' 2024-04-14T14:04:07.594 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Apr 14 14:04:07 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-x[45487]: ::ffff:172.21.15.122 - - [14/Apr/2024:14:04:07] "GET /metrics HTTP/1.1" 200 34899 "" "Prometheus/2.33.4" 2024-04-14T14:04:08.317 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:04:07 smithi069 ceph-mon[24818]: from='mgr.24749 ' entity='mgr.x' 2024-04-14T14:04:08.317 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:04:07 smithi069 ceph-mon[24818]: from='mgr.24749 ' entity='mgr.x' 2024-04-14T14:04:08.317 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:04:07 smithi069 ceph-mon[24818]: from='mgr.24749 ' entity='mgr.x' 2024-04-14T14:04:08.318 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:04:07 smithi069 ceph-mon[24818]: from='mgr.24749 ' entity='mgr.x' 2024-04-14T14:04:08.318 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:04:07 smithi069 ceph-mon[24818]: from='mgr.24749 172.21.15.122:0/1892077634' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-14T14:04:08.318 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:04:07 smithi069 ceph-mon[24818]: from='mgr.24749 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-14T14:04:08.318 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:04:07 smithi069 ceph-mon[24818]: from='mgr.24749 172.21.15.122:0/1892077634' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-14T14:04:08.318 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:04:07 smithi069 ceph-mon[24818]: from='mgr.24749 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-14T14:04:08.318 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:04:07 smithi069 ceph-mon[24818]: from='mgr.24749 172.21.15.122:0/1892077634' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-14T14:04:08.318 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:04:07 smithi069 ceph-mon[24818]: from='mgr.24749 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-14T14:04:08.318 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:04:07 smithi069 ceph-mon[24818]: from='mgr.24749 172.21.15.122:0/1892077634' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-14T14:04:08.318 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:04:07 smithi069 ceph-mon[24818]: from='mgr.24749 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-14T14:04:08.318 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:04:07 smithi069 ceph-mon[24818]: Adjusting osd_memory_target on smithi069 to 2992M 2024-04-14T14:04:08.318 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:04:07 smithi069 ceph-mon[24818]: from='mgr.24749 ' entity='mgr.x' 2024-04-14T14:04:08.318 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:04:07 smithi069 ceph-mon[24818]: pgmap v8: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 24 KiB/s rd, 0 B/s wr, 13 op/s 2024-04-14T14:04:08.319 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:04:07 smithi069 ceph-mon[29472]: from='mgr.24749 ' entity='mgr.x' 2024-04-14T14:04:08.319 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:04:07 smithi069 ceph-mon[29472]: from='mgr.24749 ' entity='mgr.x' 2024-04-14T14:04:08.319 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:04:07 smithi069 ceph-mon[29472]: from='mgr.24749 ' entity='mgr.x' 2024-04-14T14:04:08.319 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:04:07 smithi069 ceph-mon[29472]: from='mgr.24749 ' entity='mgr.x' 2024-04-14T14:04:08.319 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:04:07 smithi069 ceph-mon[29472]: from='mgr.24749 172.21.15.122:0/1892077634' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-14T14:04:08.319 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:04:07 smithi069 ceph-mon[29472]: from='mgr.24749 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-14T14:04:08.319 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:04:07 smithi069 ceph-mon[29472]: from='mgr.24749 172.21.15.122:0/1892077634' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-14T14:04:08.319 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:04:07 smithi069 ceph-mon[29472]: from='mgr.24749 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-14T14:04:08.319 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:04:07 smithi069 ceph-mon[29472]: from='mgr.24749 172.21.15.122:0/1892077634' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-14T14:04:08.319 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:04:07 smithi069 ceph-mon[29472]: from='mgr.24749 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-14T14:04:08.319 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:04:07 smithi069 ceph-mon[29472]: from='mgr.24749 172.21.15.122:0/1892077634' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-14T14:04:08.319 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:04:07 smithi069 ceph-mon[29472]: from='mgr.24749 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-14T14:04:08.319 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:04:07 smithi069 ceph-mon[29472]: Adjusting osd_memory_target on smithi069 to 2992M 2024-04-14T14:04:08.319 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:04:07 smithi069 ceph-mon[29472]: from='mgr.24749 ' entity='mgr.x' 2024-04-14T14:04:08.320 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:04:07 smithi069 ceph-mon[29472]: pgmap v8: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 24 KiB/s rd, 0 B/s wr, 13 op/s 2024-04-14T14:04:08.344 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:04:07 smithi122 ceph-mon[25721]: from='mgr.24749 ' entity='mgr.x' 2024-04-14T14:04:08.345 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:04:07 smithi122 ceph-mon[25721]: from='mgr.24749 ' entity='mgr.x' 2024-04-14T14:04:08.345 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:04:07 smithi122 ceph-mon[25721]: from='mgr.24749 ' entity='mgr.x' 2024-04-14T14:04:08.345 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:04:07 smithi122 ceph-mon[25721]: from='mgr.24749 ' entity='mgr.x' 2024-04-14T14:04:08.345 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:04:07 smithi122 ceph-mon[25721]: from='mgr.24749 172.21.15.122:0/1892077634' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-14T14:04:08.345 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:04:07 smithi122 ceph-mon[25721]: from='mgr.24749 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-14T14:04:08.345 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:04:07 smithi122 ceph-mon[25721]: from='mgr.24749 172.21.15.122:0/1892077634' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-14T14:04:08.345 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:04:07 smithi122 ceph-mon[25721]: from='mgr.24749 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-14T14:04:08.345 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:04:07 smithi122 ceph-mon[25721]: from='mgr.24749 172.21.15.122:0/1892077634' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-14T14:04:08.345 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:04:07 smithi122 ceph-mon[25721]: from='mgr.24749 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-14T14:04:08.345 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:04:07 smithi122 ceph-mon[25721]: from='mgr.24749 172.21.15.122:0/1892077634' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-14T14:04:08.345 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:04:07 smithi122 ceph-mon[25721]: from='mgr.24749 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-14T14:04:08.345 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:04:07 smithi122 ceph-mon[25721]: Adjusting osd_memory_target on smithi069 to 2992M 2024-04-14T14:04:08.345 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:04:07 smithi122 ceph-mon[25721]: from='mgr.24749 ' entity='mgr.x' 2024-04-14T14:04:08.346 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:04:07 smithi122 ceph-mon[25721]: pgmap v8: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 24 KiB/s rd, 0 B/s wr, 13 op/s 2024-04-14T14:04:09.317 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:04:09 smithi069 ceph-mon[24818]: from='mgr.24749 ' entity='mgr.x' 2024-04-14T14:04:09.317 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:04:09 smithi069 ceph-mon[24818]: from='mgr.24749 ' entity='mgr.x' 2024-04-14T14:04:09.317 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:04:09 smithi069 ceph-mon[24818]: from='mgr.24749 172.21.15.122:0/1892077634' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-14T14:04:09.317 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:04:09 smithi069 ceph-mon[24818]: from='mgr.24749 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-14T14:04:09.317 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:04:09 smithi069 ceph-mon[24818]: from='mgr.24749 172.21.15.122:0/1892077634' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-14T14:04:09.317 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:04:09 smithi069 ceph-mon[24818]: from='mgr.24749 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-14T14:04:09.317 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:04:09 smithi069 ceph-mon[24818]: from='mgr.24749 172.21.15.122:0/1892077634' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-14T14:04:09.317 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:04:09 smithi069 ceph-mon[24818]: from='mgr.24749 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-14T14:04:09.317 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:04:09 smithi069 ceph-mon[24818]: from='mgr.24749 172.21.15.122:0/1892077634' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-14T14:04:09.317 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:04:09 smithi069 ceph-mon[24818]: from='mgr.24749 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-14T14:04:09.318 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:04:09 smithi069 ceph-mon[24818]: Adjusting osd_memory_target on smithi122 to 3248M 2024-04-14T14:04:09.318 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:04:09 smithi069 ceph-mon[24818]: from='mgr.24749 ' entity='mgr.x' 2024-04-14T14:04:09.318 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:04:09 smithi069 ceph-mon[24818]: from='mgr.24749 172.21.15.122:0/1892077634' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T14:04:09.318 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:04:09 smithi069 ceph-mon[24818]: from='mgr.24749 172.21.15.122:0/1892077634' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-14T14:04:09.318 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:04:09 smithi069 ceph-mon[24818]: Updating smithi069:/etc/ceph/ceph.conf 2024-04-14T14:04:09.318 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:04:09 smithi069 ceph-mon[24818]: Updating smithi122:/etc/ceph/ceph.conf 2024-04-14T14:04:09.318 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:04:09 smithi069 ceph-mon[24818]: Updating smithi122:/var/lib/ceph/dc326fc2-fa65-11ee-bc8e-c7b262605968/config/ceph.conf 2024-04-14T14:04:09.318 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:04:09 smithi069 ceph-mon[24818]: Updating smithi069:/var/lib/ceph/dc326fc2-fa65-11ee-bc8e-c7b262605968/config/ceph.conf 2024-04-14T14:04:09.318 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:04:09 smithi069 ceph-mon[24818]: Updating smithi069:/etc/ceph/ceph.client.admin.keyring 2024-04-14T14:04:09.318 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:04:09 smithi069 ceph-mon[24818]: Updating smithi122:/etc/ceph/ceph.client.admin.keyring 2024-04-14T14:04:09.318 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:04:09 smithi069 ceph-mon[24818]: Updating smithi069:/var/lib/ceph/dc326fc2-fa65-11ee-bc8e-c7b262605968/config/ceph.client.admin.keyring 2024-04-14T14:04:09.318 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:04:09 smithi069 ceph-mon[24818]: Updating smithi122:/var/lib/ceph/dc326fc2-fa65-11ee-bc8e-c7b262605968/config/ceph.client.admin.keyring 2024-04-14T14:04:09.319 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:04:09 smithi069 ceph-mon[29472]: from='mgr.24749 ' entity='mgr.x' 2024-04-14T14:04:09.319 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:04:09 smithi069 ceph-mon[29472]: from='mgr.24749 ' entity='mgr.x' 2024-04-14T14:04:09.319 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:04:09 smithi069 ceph-mon[29472]: from='mgr.24749 172.21.15.122:0/1892077634' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-14T14:04:09.319 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:04:09 smithi069 ceph-mon[29472]: from='mgr.24749 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-14T14:04:09.319 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:04:09 smithi069 ceph-mon[29472]: from='mgr.24749 172.21.15.122:0/1892077634' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-14T14:04:09.319 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:04:09 smithi069 ceph-mon[29472]: from='mgr.24749 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-14T14:04:09.319 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:04:09 smithi069 ceph-mon[29472]: from='mgr.24749 172.21.15.122:0/1892077634' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-14T14:04:09.319 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:04:09 smithi069 ceph-mon[29472]: from='mgr.24749 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-14T14:04:09.319 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:04:09 smithi069 ceph-mon[29472]: from='mgr.24749 172.21.15.122:0/1892077634' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-14T14:04:09.319 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:04:09 smithi069 ceph-mon[29472]: from='mgr.24749 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-14T14:04:09.319 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:04:09 smithi069 ceph-mon[29472]: Adjusting osd_memory_target on smithi122 to 3248M 2024-04-14T14:04:09.319 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:04:09 smithi069 ceph-mon[29472]: from='mgr.24749 ' entity='mgr.x' 2024-04-14T14:04:09.319 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:04:09 smithi069 ceph-mon[29472]: from='mgr.24749 172.21.15.122:0/1892077634' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T14:04:09.319 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:04:09 smithi069 ceph-mon[29472]: from='mgr.24749 172.21.15.122:0/1892077634' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-14T14:04:09.320 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:04:09 smithi069 ceph-mon[29472]: Updating smithi069:/etc/ceph/ceph.conf 2024-04-14T14:04:09.320 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:04:09 smithi069 ceph-mon[29472]: Updating smithi122:/etc/ceph/ceph.conf 2024-04-14T14:04:09.320 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:04:09 smithi069 ceph-mon[29472]: Updating smithi122:/var/lib/ceph/dc326fc2-fa65-11ee-bc8e-c7b262605968/config/ceph.conf 2024-04-14T14:04:09.320 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:04:09 smithi069 ceph-mon[29472]: Updating smithi069:/var/lib/ceph/dc326fc2-fa65-11ee-bc8e-c7b262605968/config/ceph.conf 2024-04-14T14:04:09.320 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:04:09 smithi069 ceph-mon[29472]: Updating smithi069:/etc/ceph/ceph.client.admin.keyring 2024-04-14T14:04:09.320 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:04:09 smithi069 ceph-mon[29472]: Updating smithi122:/etc/ceph/ceph.client.admin.keyring 2024-04-14T14:04:09.320 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:04:09 smithi069 ceph-mon[29472]: Updating smithi069:/var/lib/ceph/dc326fc2-fa65-11ee-bc8e-c7b262605968/config/ceph.client.admin.keyring 2024-04-14T14:04:09.320 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:04:09 smithi069 ceph-mon[29472]: Updating smithi122:/var/lib/ceph/dc326fc2-fa65-11ee-bc8e-c7b262605968/config/ceph.client.admin.keyring 2024-04-14T14:04:09.344 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:04:09 smithi122 ceph-mon[25721]: from='mgr.24749 ' entity='mgr.x' 2024-04-14T14:04:09.344 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:04:09 smithi122 ceph-mon[25721]: from='mgr.24749 ' entity='mgr.x' 2024-04-14T14:04:09.345 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:04:09 smithi122 ceph-mon[25721]: from='mgr.24749 172.21.15.122:0/1892077634' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-14T14:04:09.345 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:04:09 smithi122 ceph-mon[25721]: from='mgr.24749 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-14T14:04:09.345 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:04:09 smithi122 ceph-mon[25721]: from='mgr.24749 172.21.15.122:0/1892077634' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-14T14:04:09.345 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:04:09 smithi122 ceph-mon[25721]: from='mgr.24749 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-14T14:04:09.345 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:04:09 smithi122 ceph-mon[25721]: from='mgr.24749 172.21.15.122:0/1892077634' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-14T14:04:09.345 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:04:09 smithi122 ceph-mon[25721]: from='mgr.24749 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-14T14:04:09.345 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:04:09 smithi122 ceph-mon[25721]: from='mgr.24749 172.21.15.122:0/1892077634' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-14T14:04:09.345 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:04:09 smithi122 ceph-mon[25721]: from='mgr.24749 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-14T14:04:09.345 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:04:09 smithi122 ceph-mon[25721]: Adjusting osd_memory_target on smithi122 to 3248M 2024-04-14T14:04:09.345 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:04:09 smithi122 ceph-mon[25721]: from='mgr.24749 ' entity='mgr.x' 2024-04-14T14:04:09.345 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:04:09 smithi122 ceph-mon[25721]: from='mgr.24749 172.21.15.122:0/1892077634' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T14:04:09.345 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:04:09 smithi122 ceph-mon[25721]: from='mgr.24749 172.21.15.122:0/1892077634' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-14T14:04:09.345 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:04:09 smithi122 ceph-mon[25721]: Updating smithi069:/etc/ceph/ceph.conf 2024-04-14T14:04:09.345 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:04:09 smithi122 ceph-mon[25721]: Updating smithi122:/etc/ceph/ceph.conf 2024-04-14T14:04:09.346 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:04:09 smithi122 ceph-mon[25721]: Updating smithi122:/var/lib/ceph/dc326fc2-fa65-11ee-bc8e-c7b262605968/config/ceph.conf 2024-04-14T14:04:09.346 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:04:09 smithi122 ceph-mon[25721]: Updating smithi069:/var/lib/ceph/dc326fc2-fa65-11ee-bc8e-c7b262605968/config/ceph.conf 2024-04-14T14:04:09.346 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:04:09 smithi122 ceph-mon[25721]: Updating smithi069:/etc/ceph/ceph.client.admin.keyring 2024-04-14T14:04:09.346 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:04:09 smithi122 ceph-mon[25721]: Updating smithi122:/etc/ceph/ceph.client.admin.keyring 2024-04-14T14:04:09.346 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:04:09 smithi122 ceph-mon[25721]: Updating smithi069:/var/lib/ceph/dc326fc2-fa65-11ee-bc8e-c7b262605968/config/ceph.client.admin.keyring 2024-04-14T14:04:09.346 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:04:09 smithi122 ceph-mon[25721]: Updating smithi122:/var/lib/ceph/dc326fc2-fa65-11ee-bc8e-c7b262605968/config/ceph.client.admin.keyring 2024-04-14T14:04:09.817 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 14 14:04:09 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-y[25054]: ::ffff:172.21.15.122 - - [14/Apr/2024:14:04:09] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-04-14T14:04:09.818 INFO:journalctl@ceph.node-exporter.a.smithi069.stdout:Apr 14 14:04:09 smithi069 systemd[1]: Stopping Ceph node-exporter.a for dc326fc2-fa65-11ee-bc8e-c7b262605968... 2024-04-14T14:04:09.818 INFO:journalctl@ceph.node-exporter.a.smithi069.stdout:Apr 14 14:04:09 smithi069 bash[58413]: Error: no container with name or ID "ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-node-exporter.a" found: no such container 2024-04-14T14:04:10.117 INFO:journalctl@ceph.node-exporter.a.smithi069.stdout:Apr 14 14:04:09 smithi069 podman[58427]: 2024-04-14 14:04:09.830411024 +0000 UTC m=+0.101744143 container died 73e900b404dfe98aa7d271cd50d2ebbb14c7206183ed360fe056923b67536bee (image=quay.io/prometheus/node-exporter:v1.3.1, name=ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-node-exporter-a, maintainer=The Prometheus Authors ) 2024-04-14T14:04:10.380 INFO:journalctl@ceph.node-exporter.a.smithi069.stdout:Apr 14 14:04:10 smithi069 podman[58427]: 2024-04-14 14:04:10.161056054 +0000 UTC m=+0.432389181 container cleanup 73e900b404dfe98aa7d271cd50d2ebbb14c7206183ed360fe056923b67536bee (image=quay.io/prometheus/node-exporter:v1.3.1, name=ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-node-exporter-a, maintainer=The Prometheus Authors ) 2024-04-14T14:04:10.380 INFO:journalctl@ceph.node-exporter.a.smithi069.stdout:Apr 14 14:04:10 smithi069 bash[58427]: ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-node-exporter-a 2024-04-14T14:04:10.380 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:04:10 smithi069 ceph-mon[29472]: from='mgr.24749 ' entity='mgr.x' 2024-04-14T14:04:10.380 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:04:10 smithi069 ceph-mon[29472]: from='mgr.24749 ' entity='mgr.x' 2024-04-14T14:04:10.380 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:04:10 smithi069 ceph-mon[29472]: from='mgr.24749 ' entity='mgr.x' 2024-04-14T14:04:10.381 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:04:10 smithi069 ceph-mon[29472]: from='mgr.24749 ' entity='mgr.x' 2024-04-14T14:04:10.381 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:04:10 smithi069 ceph-mon[29472]: from='mgr.24749 ' entity='mgr.x' 2024-04-14T14:04:10.381 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:04:10 smithi069 ceph-mon[29472]: from='mgr.24749 ' entity='mgr.x' 2024-04-14T14:04:10.381 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:04:10 smithi069 ceph-mon[29472]: from='mgr.24749 ' entity='mgr.x' 2024-04-14T14:04:10.381 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:04:10 smithi069 ceph-mon[29472]: from='mgr.24749 ' entity='mgr.x' 2024-04-14T14:04:10.381 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:04:10 smithi069 ceph-mon[29472]: from='mgr.24749 ' entity='mgr.x' 2024-04-14T14:04:10.381 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:04:10 smithi069 ceph-mon[29472]: Reconfiguring node-exporter.a (dependencies changed)... 2024-04-14T14:04:10.381 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:04:10 smithi069 ceph-mon[29472]: Reconfiguring daemon node-exporter.a on smithi069 2024-04-14T14:04:10.381 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:04:10 smithi069 ceph-mon[29472]: pgmap v9: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 20 KiB/s rd, 0 B/s wr, 11 op/s 2024-04-14T14:04:10.381 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:04:10 smithi069 ceph-mon[24818]: from='mgr.24749 ' entity='mgr.x' 2024-04-14T14:04:10.381 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:04:10 smithi069 ceph-mon[24818]: from='mgr.24749 ' entity='mgr.x' 2024-04-14T14:04:10.381 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:04:10 smithi069 ceph-mon[24818]: from='mgr.24749 ' entity='mgr.x' 2024-04-14T14:04:10.381 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:04:10 smithi069 ceph-mon[24818]: from='mgr.24749 ' entity='mgr.x' 2024-04-14T14:04:10.382 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:04:10 smithi069 ceph-mon[24818]: from='mgr.24749 ' entity='mgr.x' 2024-04-14T14:04:10.382 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:04:10 smithi069 ceph-mon[24818]: from='mgr.24749 ' entity='mgr.x' 2024-04-14T14:04:10.382 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:04:10 smithi069 ceph-mon[24818]: from='mgr.24749 ' entity='mgr.x' 2024-04-14T14:04:10.382 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:04:10 smithi069 ceph-mon[24818]: from='mgr.24749 ' entity='mgr.x' 2024-04-14T14:04:10.382 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:04:10 smithi069 ceph-mon[24818]: from='mgr.24749 ' entity='mgr.x' 2024-04-14T14:04:10.382 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:04:10 smithi069 ceph-mon[24818]: Reconfiguring node-exporter.a (dependencies changed)... 2024-04-14T14:04:10.382 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:04:10 smithi069 ceph-mon[24818]: Reconfiguring daemon node-exporter.a on smithi069 2024-04-14T14:04:10.382 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:04:10 smithi069 ceph-mon[24818]: pgmap v9: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 20 KiB/s rd, 0 B/s wr, 11 op/s 2024-04-14T14:04:10.594 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:04:10 smithi122 ceph-mon[25721]: from='mgr.24749 ' entity='mgr.x' 2024-04-14T14:04:10.595 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:04:10 smithi122 ceph-mon[25721]: from='mgr.24749 ' entity='mgr.x' 2024-04-14T14:04:10.595 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:04:10 smithi122 ceph-mon[25721]: from='mgr.24749 ' entity='mgr.x' 2024-04-14T14:04:10.595 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:04:10 smithi122 ceph-mon[25721]: from='mgr.24749 ' entity='mgr.x' 2024-04-14T14:04:10.595 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:04:10 smithi122 ceph-mon[25721]: from='mgr.24749 ' entity='mgr.x' 2024-04-14T14:04:10.595 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:04:10 smithi122 ceph-mon[25721]: from='mgr.24749 ' entity='mgr.x' 2024-04-14T14:04:10.595 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:04:10 smithi122 ceph-mon[25721]: from='mgr.24749 ' entity='mgr.x' 2024-04-14T14:04:10.595 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:04:10 smithi122 ceph-mon[25721]: from='mgr.24749 ' entity='mgr.x' 2024-04-14T14:04:10.595 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:04:10 smithi122 ceph-mon[25721]: from='mgr.24749 ' entity='mgr.x' 2024-04-14T14:04:10.595 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:04:10 smithi122 ceph-mon[25721]: Reconfiguring node-exporter.a (dependencies changed)... 2024-04-14T14:04:10.595 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:04:10 smithi122 ceph-mon[25721]: Reconfiguring daemon node-exporter.a on smithi069 2024-04-14T14:04:10.595 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:04:10 smithi122 ceph-mon[25721]: pgmap v9: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 20 KiB/s rd, 0 B/s wr, 11 op/s 2024-04-14T14:04:10.630 INFO:journalctl@ceph.node-exporter.a.smithi069.stdout:Apr 14 14:04:10 smithi069 podman[58441]: 2024-04-14 14:04:10.379239684 +0000 UTC m=+0.552830556 container remove 73e900b404dfe98aa7d271cd50d2ebbb14c7206183ed360fe056923b67536bee (image=quay.io/prometheus/node-exporter:v1.3.1, name=ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-node-exporter-a, maintainer=The Prometheus Authors ) 2024-04-14T14:04:10.631 INFO:journalctl@ceph.node-exporter.a.smithi069.stdout:Apr 14 14:04:10 smithi069 systemd[1]: ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968@node-exporter.a.service: Main process exited, code=exited, status=143/n/a 2024-04-14T14:04:10.631 INFO:journalctl@ceph.node-exporter.a.smithi069.stdout:Apr 14 14:04:10 smithi069 bash[58455]: Error: no container with name or ID "ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-node-exporter.a" found: no such container 2024-04-14T14:04:10.631 INFO:journalctl@ceph.node-exporter.a.smithi069.stdout:Apr 14 14:04:10 smithi069 systemd[1]: ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968@node-exporter.a.service: Failed with result 'exit-code'. 2024-04-14T14:04:10.631 INFO:journalctl@ceph.node-exporter.a.smithi069.stdout:Apr 14 14:04:10 smithi069 systemd[1]: Stopped Ceph node-exporter.a for dc326fc2-fa65-11ee-bc8e-c7b262605968. 2024-04-14T14:04:10.631 INFO:journalctl@ceph.node-exporter.a.smithi069.stdout:Apr 14 14:04:10 smithi069 systemd[1]: ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968@node-exporter.a.service: Consumed 3.798s CPU time. 2024-04-14T14:04:10.631 INFO:journalctl@ceph.node-exporter.a.smithi069.stdout:Apr 14 14:04:10 smithi069 systemd[1]: Starting Ceph node-exporter.a for dc326fc2-fa65-11ee-bc8e-c7b262605968... 2024-04-14T14:04:10.631 INFO:journalctl@ceph.node-exporter.a.smithi069.stdout:Apr 14 14:04:10 smithi069 podman[58523]: 2024-04-14 14:04:10.52995571 +0000 UTC m=+0.017472266 image pull 1dbe0e931976487e20e5cfb272087e08a9779c88fd5e9617ed7042dd9751ec26 quay.io/prometheus/node-exporter:v1.3.1 2024-04-14T14:04:10.631 INFO:journalctl@ceph.node-exporter.a.smithi069.stdout:Apr 14 14:04:10 smithi069 podman[58523]: 2024-04-14 14:04:10.630035964 +0000 UTC m=+0.117552516 container create ca074142ab73b87b0f942e4242375445f3e1852cd0f6dbcba0fbb79b81e1027b (image=quay.io/prometheus/node-exporter:v1.3.1, name=ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-node-exporter-a, maintainer=The Prometheus Authors ) 2024-04-14T14:04:10.942 INFO:journalctl@ceph.node-exporter.a.smithi069.stdout:Apr 14 14:04:10 smithi069 podman[58523]: 2024-04-14 14:04:10.857452902 +0000 UTC m=+0.344969459 container init ca074142ab73b87b0f942e4242375445f3e1852cd0f6dbcba0fbb79b81e1027b (image=quay.io/prometheus/node-exporter:v1.3.1, name=ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-node-exporter-a, maintainer=The Prometheus Authors ) 2024-04-14T14:04:10.942 INFO:journalctl@ceph.node-exporter.a.smithi069.stdout:Apr 14 14:04:10 smithi069 podman[58523]: 2024-04-14 14:04:10.859947642 +0000 UTC m=+0.347464196 container start ca074142ab73b87b0f942e4242375445f3e1852cd0f6dbcba0fbb79b81e1027b (image=quay.io/prometheus/node-exporter:v1.3.1, name=ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-node-exporter-a, maintainer=The Prometheus Authors ) 2024-04-14T14:04:10.942 INFO:journalctl@ceph.node-exporter.a.smithi069.stdout:Apr 14 14:04:10 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-node-exporter-a[58537]: ts=2024-04-14T14:04:10.863Z caller=node_exporter.go:182 level=info msg="Starting node_exporter" version="(version=1.3.1, branch=HEAD, revision=a2321e7b940ddcff26873612bccdf7cd4c42b6b6)" 2024-04-14T14:04:10.942 INFO:journalctl@ceph.node-exporter.a.smithi069.stdout:Apr 14 14:04:10 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-node-exporter-a[58537]: ts=2024-04-14T14:04:10.863Z caller=node_exporter.go:183 level=info msg="Build context" build_context="(go=go1.17.3, user=root@243aafa5525c, date=20211205-11:09:49)" 2024-04-14T14:04:10.942 INFO:journalctl@ceph.node-exporter.a.smithi069.stdout:Apr 14 14:04:10 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-node-exporter-a[58537]: ts=2024-04-14T14:04:10.864Z caller=filesystem_common.go:111 level=info collector=filesystem msg="Parsed flag --collector.filesystem.mount-points-exclude" flag=^/(dev|proc|run/credentials/.+|sys|var/lib/docker/.+)($|/) 2024-04-14T14:04:10.942 INFO:journalctl@ceph.node-exporter.a.smithi069.stdout:Apr 14 14:04:10 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-node-exporter-a[58537]: ts=2024-04-14T14:04:10.864Z caller=filesystem_common.go:113 level=info collector=filesystem msg="Parsed flag --collector.filesystem.fs-types-exclude" flag=^(autofs|binfmt_misc|bpf|cgroup2?|configfs|debugfs|devpts|devtmpfs|fusectl|hugetlbfs|iso9660|mqueue|nsfs|overlay|proc|procfs|pstore|rpc_pipefs|securityfs|selinuxfs|squashfs|sysfs|tracefs)$ 2024-04-14T14:04:10.942 INFO:journalctl@ceph.node-exporter.a.smithi069.stdout:Apr 14 14:04:10 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-node-exporter-a[58537]: ts=2024-04-14T14:04:10.864Z caller=node_exporter.go:108 level=info msg="Enabled collectors" 2024-04-14T14:04:10.942 INFO:journalctl@ceph.node-exporter.a.smithi069.stdout:Apr 14 14:04:10 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-node-exporter-a[58537]: ts=2024-04-14T14:04:10.864Z caller=node_exporter.go:115 level=info collector=arp 2024-04-14T14:04:10.942 INFO:journalctl@ceph.node-exporter.a.smithi069.stdout:Apr 14 14:04:10 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-node-exporter-a[58537]: ts=2024-04-14T14:04:10.864Z caller=node_exporter.go:115 level=info collector=bcache 2024-04-14T14:04:10.942 INFO:journalctl@ceph.node-exporter.a.smithi069.stdout:Apr 14 14:04:10 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-node-exporter-a[58537]: ts=2024-04-14T14:04:10.864Z caller=node_exporter.go:115 level=info collector=bonding 2024-04-14T14:04:10.942 INFO:journalctl@ceph.node-exporter.a.smithi069.stdout:Apr 14 14:04:10 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-node-exporter-a[58537]: ts=2024-04-14T14:04:10.864Z caller=node_exporter.go:115 level=info collector=btrfs 2024-04-14T14:04:10.943 INFO:journalctl@ceph.node-exporter.a.smithi069.stdout:Apr 14 14:04:10 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-node-exporter-a[58537]: ts=2024-04-14T14:04:10.864Z caller=node_exporter.go:115 level=info collector=conntrack 2024-04-14T14:04:10.943 INFO:journalctl@ceph.node-exporter.a.smithi069.stdout:Apr 14 14:04:10 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-node-exporter-a[58537]: ts=2024-04-14T14:04:10.864Z caller=node_exporter.go:115 level=info collector=cpu 2024-04-14T14:04:10.943 INFO:journalctl@ceph.node-exporter.a.smithi069.stdout:Apr 14 14:04:10 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-node-exporter-a[58537]: ts=2024-04-14T14:04:10.864Z caller=node_exporter.go:115 level=info collector=cpufreq 2024-04-14T14:04:10.943 INFO:journalctl@ceph.node-exporter.a.smithi069.stdout:Apr 14 14:04:10 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-node-exporter-a[58537]: ts=2024-04-14T14:04:10.864Z caller=node_exporter.go:115 level=info collector=diskstats 2024-04-14T14:04:10.943 INFO:journalctl@ceph.node-exporter.a.smithi069.stdout:Apr 14 14:04:10 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-node-exporter-a[58537]: ts=2024-04-14T14:04:10.864Z caller=node_exporter.go:115 level=info collector=dmi 2024-04-14T14:04:10.943 INFO:journalctl@ceph.node-exporter.a.smithi069.stdout:Apr 14 14:04:10 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-node-exporter-a[58537]: ts=2024-04-14T14:04:10.864Z caller=node_exporter.go:115 level=info collector=edac 2024-04-14T14:04:10.943 INFO:journalctl@ceph.node-exporter.a.smithi069.stdout:Apr 14 14:04:10 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-node-exporter-a[58537]: ts=2024-04-14T14:04:10.864Z caller=node_exporter.go:115 level=info collector=entropy 2024-04-14T14:04:10.943 INFO:journalctl@ceph.node-exporter.a.smithi069.stdout:Apr 14 14:04:10 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-node-exporter-a[58537]: ts=2024-04-14T14:04:10.864Z caller=node_exporter.go:115 level=info collector=fibrechannel 2024-04-14T14:04:10.944 INFO:journalctl@ceph.node-exporter.a.smithi069.stdout:Apr 14 14:04:10 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-node-exporter-a[58537]: ts=2024-04-14T14:04:10.864Z caller=node_exporter.go:115 level=info collector=filefd 2024-04-14T14:04:10.944 INFO:journalctl@ceph.node-exporter.a.smithi069.stdout:Apr 14 14:04:10 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-node-exporter-a[58537]: ts=2024-04-14T14:04:10.864Z caller=node_exporter.go:115 level=info collector=filesystem 2024-04-14T14:04:10.944 INFO:journalctl@ceph.node-exporter.a.smithi069.stdout:Apr 14 14:04:10 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-node-exporter-a[58537]: ts=2024-04-14T14:04:10.864Z caller=node_exporter.go:115 level=info collector=hwmon 2024-04-14T14:04:10.944 INFO:journalctl@ceph.node-exporter.a.smithi069.stdout:Apr 14 14:04:10 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-node-exporter-a[58537]: ts=2024-04-14T14:04:10.864Z caller=node_exporter.go:115 level=info collector=infiniband 2024-04-14T14:04:10.944 INFO:journalctl@ceph.node-exporter.a.smithi069.stdout:Apr 14 14:04:10 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-node-exporter-a[58537]: ts=2024-04-14T14:04:10.864Z caller=node_exporter.go:115 level=info collector=ipvs 2024-04-14T14:04:10.944 INFO:journalctl@ceph.node-exporter.a.smithi069.stdout:Apr 14 14:04:10 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-node-exporter-a[58537]: ts=2024-04-14T14:04:10.864Z caller=node_exporter.go:115 level=info collector=loadavg 2024-04-14T14:04:10.944 INFO:journalctl@ceph.node-exporter.a.smithi069.stdout:Apr 14 14:04:10 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-node-exporter-a[58537]: ts=2024-04-14T14:04:10.864Z caller=node_exporter.go:115 level=info collector=mdadm 2024-04-14T14:04:10.944 INFO:journalctl@ceph.node-exporter.a.smithi069.stdout:Apr 14 14:04:10 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-node-exporter-a[58537]: ts=2024-04-14T14:04:10.864Z caller=node_exporter.go:115 level=info collector=meminfo 2024-04-14T14:04:10.944 INFO:journalctl@ceph.node-exporter.a.smithi069.stdout:Apr 14 14:04:10 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-node-exporter-a[58537]: ts=2024-04-14T14:04:10.864Z caller=node_exporter.go:115 level=info collector=netclass 2024-04-14T14:04:10.944 INFO:journalctl@ceph.node-exporter.a.smithi069.stdout:Apr 14 14:04:10 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-node-exporter-a[58537]: ts=2024-04-14T14:04:10.864Z caller=node_exporter.go:115 level=info collector=netdev 2024-04-14T14:04:10.944 INFO:journalctl@ceph.node-exporter.a.smithi069.stdout:Apr 14 14:04:10 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-node-exporter-a[58537]: ts=2024-04-14T14:04:10.864Z caller=node_exporter.go:115 level=info collector=netstat 2024-04-14T14:04:10.944 INFO:journalctl@ceph.node-exporter.a.smithi069.stdout:Apr 14 14:04:10 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-node-exporter-a[58537]: ts=2024-04-14T14:04:10.864Z caller=node_exporter.go:115 level=info collector=nfs 2024-04-14T14:04:10.944 INFO:journalctl@ceph.node-exporter.a.smithi069.stdout:Apr 14 14:04:10 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-node-exporter-a[58537]: ts=2024-04-14T14:04:10.864Z caller=node_exporter.go:115 level=info collector=nfsd 2024-04-14T14:04:10.944 INFO:journalctl@ceph.node-exporter.a.smithi069.stdout:Apr 14 14:04:10 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-node-exporter-a[58537]: ts=2024-04-14T14:04:10.864Z caller=node_exporter.go:115 level=info collector=nvme 2024-04-14T14:04:10.945 INFO:journalctl@ceph.node-exporter.a.smithi069.stdout:Apr 14 14:04:10 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-node-exporter-a[58537]: ts=2024-04-14T14:04:10.864Z caller=node_exporter.go:115 level=info collector=os 2024-04-14T14:04:10.945 INFO:journalctl@ceph.node-exporter.a.smithi069.stdout:Apr 14 14:04:10 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-node-exporter-a[58537]: ts=2024-04-14T14:04:10.864Z caller=node_exporter.go:115 level=info collector=powersupplyclass 2024-04-14T14:04:10.945 INFO:journalctl@ceph.node-exporter.a.smithi069.stdout:Apr 14 14:04:10 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-node-exporter-a[58537]: ts=2024-04-14T14:04:10.864Z caller=node_exporter.go:115 level=info collector=pressure 2024-04-14T14:04:10.945 INFO:journalctl@ceph.node-exporter.a.smithi069.stdout:Apr 14 14:04:10 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-node-exporter-a[58537]: ts=2024-04-14T14:04:10.864Z caller=node_exporter.go:115 level=info collector=rapl 2024-04-14T14:04:10.945 INFO:journalctl@ceph.node-exporter.a.smithi069.stdout:Apr 14 14:04:10 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-node-exporter-a[58537]: ts=2024-04-14T14:04:10.864Z caller=node_exporter.go:115 level=info collector=schedstat 2024-04-14T14:04:10.945 INFO:journalctl@ceph.node-exporter.a.smithi069.stdout:Apr 14 14:04:10 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-node-exporter-a[58537]: ts=2024-04-14T14:04:10.864Z caller=node_exporter.go:115 level=info collector=sockstat 2024-04-14T14:04:10.945 INFO:journalctl@ceph.node-exporter.a.smithi069.stdout:Apr 14 14:04:10 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-node-exporter-a[58537]: ts=2024-04-14T14:04:10.864Z caller=node_exporter.go:115 level=info collector=softnet 2024-04-14T14:04:10.945 INFO:journalctl@ceph.node-exporter.a.smithi069.stdout:Apr 14 14:04:10 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-node-exporter-a[58537]: ts=2024-04-14T14:04:10.865Z caller=node_exporter.go:115 level=info collector=stat 2024-04-14T14:04:10.945 INFO:journalctl@ceph.node-exporter.a.smithi069.stdout:Apr 14 14:04:10 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-node-exporter-a[58537]: ts=2024-04-14T14:04:10.865Z caller=node_exporter.go:115 level=info collector=tapestats 2024-04-14T14:04:10.945 INFO:journalctl@ceph.node-exporter.a.smithi069.stdout:Apr 14 14:04:10 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-node-exporter-a[58537]: ts=2024-04-14T14:04:10.865Z caller=node_exporter.go:115 level=info collector=textfile 2024-04-14T14:04:10.945 INFO:journalctl@ceph.node-exporter.a.smithi069.stdout:Apr 14 14:04:10 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-node-exporter-a[58537]: ts=2024-04-14T14:04:10.865Z caller=node_exporter.go:115 level=info collector=thermal_zone 2024-04-14T14:04:10.945 INFO:journalctl@ceph.node-exporter.a.smithi069.stdout:Apr 14 14:04:10 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-node-exporter-a[58537]: ts=2024-04-14T14:04:10.865Z caller=node_exporter.go:115 level=info collector=time 2024-04-14T14:04:10.945 INFO:journalctl@ceph.node-exporter.a.smithi069.stdout:Apr 14 14:04:10 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-node-exporter-a[58537]: ts=2024-04-14T14:04:10.865Z caller=node_exporter.go:115 level=info collector=udp_queues 2024-04-14T14:04:10.945 INFO:journalctl@ceph.node-exporter.a.smithi069.stdout:Apr 14 14:04:10 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-node-exporter-a[58537]: ts=2024-04-14T14:04:10.865Z caller=node_exporter.go:115 level=info collector=uname 2024-04-14T14:04:10.945 INFO:journalctl@ceph.node-exporter.a.smithi069.stdout:Apr 14 14:04:10 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-node-exporter-a[58537]: ts=2024-04-14T14:04:10.865Z caller=node_exporter.go:115 level=info collector=vmstat 2024-04-14T14:04:10.946 INFO:journalctl@ceph.node-exporter.a.smithi069.stdout:Apr 14 14:04:10 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-node-exporter-a[58537]: ts=2024-04-14T14:04:10.865Z caller=node_exporter.go:115 level=info collector=xfs 2024-04-14T14:04:10.946 INFO:journalctl@ceph.node-exporter.a.smithi069.stdout:Apr 14 14:04:10 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-node-exporter-a[58537]: ts=2024-04-14T14:04:10.865Z caller=node_exporter.go:115 level=info collector=zfs 2024-04-14T14:04:10.946 INFO:journalctl@ceph.node-exporter.a.smithi069.stdout:Apr 14 14:04:10 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-node-exporter-a[58537]: ts=2024-04-14T14:04:10.865Z caller=node_exporter.go:199 level=info msg="Listening on" address=:9100 2024-04-14T14:04:10.946 INFO:journalctl@ceph.node-exporter.a.smithi069.stdout:Apr 14 14:04:10 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-node-exporter-a[58537]: ts=2024-04-14T14:04:10.865Z caller=tls_config.go:195 level=info msg="TLS is disabled." http2=false 2024-04-14T14:04:10.946 INFO:journalctl@ceph.node-exporter.a.smithi069.stdout:Apr 14 14:04:10 smithi069 bash[58523]: ca074142ab73b87b0f942e4242375445f3e1852cd0f6dbcba0fbb79b81e1027b 2024-04-14T14:04:11.316 INFO:journalctl@ceph.node-exporter.a.smithi069.stdout:Apr 14 14:04:10 smithi069 systemd[1]: Started Ceph node-exporter.a for dc326fc2-fa65-11ee-bc8e-c7b262605968. 2024-04-14T14:04:12.317 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:04:11 smithi069 ceph-mon[24818]: from='mgr.24749 ' entity='mgr.x' 2024-04-14T14:04:12.317 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:04:11 smithi069 ceph-mon[24818]: from='mgr.24749 ' entity='mgr.x' 2024-04-14T14:04:12.317 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:04:11 smithi069 ceph-mon[24818]: Reconfiguring alertmanager.a (dependencies changed)... 2024-04-14T14:04:12.317 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:04:11 smithi069 ceph-mon[24818]: Reconfiguring daemon alertmanager.a on smithi069 2024-04-14T14:04:12.317 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:04:11 smithi069 ceph-mon[24818]: pgmap v10: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 10 op/s 2024-04-14T14:04:12.317 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:04:11 smithi069 ceph-mon[29472]: from='mgr.24749 ' entity='mgr.x' 2024-04-14T14:04:12.317 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:04:11 smithi069 ceph-mon[29472]: from='mgr.24749 ' entity='mgr.x' 2024-04-14T14:04:12.317 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:04:11 smithi069 ceph-mon[29472]: Reconfiguring alertmanager.a (dependencies changed)... 2024-04-14T14:04:12.318 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:04:11 smithi069 ceph-mon[29472]: Reconfiguring daemon alertmanager.a on smithi069 2024-04-14T14:04:12.318 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:04:11 smithi069 ceph-mon[29472]: pgmap v10: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 10 op/s 2024-04-14T14:04:12.344 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:04:11 smithi122 ceph-mon[25721]: from='mgr.24749 ' entity='mgr.x' 2024-04-14T14:04:12.344 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:04:11 smithi122 ceph-mon[25721]: from='mgr.24749 ' entity='mgr.x' 2024-04-14T14:04:12.345 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:04:11 smithi122 ceph-mon[25721]: Reconfiguring alertmanager.a (dependencies changed)... 2024-04-14T14:04:12.345 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:04:11 smithi122 ceph-mon[25721]: Reconfiguring daemon alertmanager.a on smithi069 2024-04-14T14:04:12.345 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:04:11 smithi122 ceph-mon[25721]: pgmap v10: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 10 op/s 2024-04-14T14:04:13.237 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:04:12 smithi069 ceph-mon[24818]: from='mgr.24749 172.21.15.122:0/1892077634' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T14:04:13.238 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:04:12 smithi069 ceph-mon[29472]: from='mgr.24749 172.21.15.122:0/1892077634' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T14:04:13.344 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:04:12 smithi122 ceph-mon[25721]: from='mgr.24749 172.21.15.122:0/1892077634' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T14:04:13.817 INFO:journalctl@ceph.alertmanager.a.smithi069.stdout:Apr 14 14:04:13 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-alertmanager-a[49465]: level=error ts=2024-04-14T14:04:13.521Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=6 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://172.21.15.122:8443/api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.122 because it doesn't contain any IP SANs; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://172.21.15.69:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.69 because it doesn't contain any IP SANs" 2024-04-14T14:04:13.817 INFO:journalctl@ceph.alertmanager.a.smithi069.stdout:Apr 14 14:04:13 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-alertmanager-a[49465]: level=warn ts=2024-04-14T14:04:13.523Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.69:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.69 because it doesn't contain any IP SANs" 2024-04-14T14:04:13.817 INFO:journalctl@ceph.alertmanager.a.smithi069.stdout:Apr 14 14:04:13 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-alertmanager-a[49465]: level=warn ts=2024-04-14T14:04:13.524Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.122:8443/api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.122 because it doesn't contain any IP SANs" 2024-04-14T14:04:14.316 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:04:13 smithi069 ceph-mon[24818]: pgmap v11: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 10 op/s 2024-04-14T14:04:14.317 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:04:13 smithi069 ceph-mon[24818]: from='client.14766 -' entity='client.iscsi.foo.smithi069.yuumtr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T14:04:14.317 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:04:13 smithi069 ceph-mon[29472]: pgmap v11: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 10 op/s 2024-04-14T14:04:14.317 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:04:13 smithi069 ceph-mon[29472]: from='client.14766 -' entity='client.iscsi.foo.smithi069.yuumtr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T14:04:14.344 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:04:13 smithi122 ceph-mon[25721]: pgmap v11: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 10 op/s 2024-04-14T14:04:14.344 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:04:13 smithi122 ceph-mon[25721]: from='client.14766 -' entity='client.iscsi.foo.smithi069.yuumtr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T14:04:16.674 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:04:16 smithi069 ceph-mon[24818]: pgmap v12: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 10 op/s 2024-04-14T14:04:16.674 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:04:16 smithi069 ceph-mon[29472]: pgmap v12: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 10 op/s 2024-04-14T14:04:16.844 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:04:16 smithi122 ceph-mon[25721]: pgmap v12: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 10 op/s 2024-04-14T14:04:17.594 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Apr 14 14:04:17 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-x[45487]: ::ffff:172.21.15.122 - - [14/Apr/2024:14:04:17] "GET /metrics HTTP/1.1" 200 37890 "" "Prometheus/2.33.4" 2024-04-14T14:04:18.261 INFO:journalctl@ceph.alertmanager.a.smithi069.stdout:Apr 14 14:04:18 smithi069 systemd[1]: Stopping Ceph alertmanager.a for dc326fc2-fa65-11ee-bc8e-c7b262605968... 2024-04-14T14:04:18.261 INFO:journalctl@ceph.alertmanager.a.smithi069.stdout:Apr 14 14:04:18 smithi069 bash[58798]: Error: no container with name or ID "ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-alertmanager.a" found: no such container 2024-04-14T14:04:18.544 INFO:journalctl@ceph.alertmanager.a.smithi069.stdout:Apr 14 14:04:18 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-alertmanager-a[49465]: level=info ts=2024-04-14T14:04:18.260Z caller=main.go:557 msg="Received SIGTERM, exiting gracefully..." 2024-04-14T14:04:18.545 INFO:journalctl@ceph.alertmanager.a.smithi069.stdout:Apr 14 14:04:18 smithi069 podman[58811]: 2024-04-14 14:04:18.280599984 +0000 UTC m=+0.120718406 container died 148fbcb552325f752f07b182e98ec63a0e7d0dd169222d4e66f95febb6fde931 (image=quay.io/prometheus/alertmanager:v0.23.0, name=ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-alertmanager-a, maintainer=The Prometheus Authors ) 2024-04-14T14:04:18.817 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:04:18 smithi069 ceph-mon[24818]: pgmap v13: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:04:18.817 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:04:18 smithi069 ceph-mon[29472]: pgmap v13: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:04:18.817 INFO:journalctl@ceph.alertmanager.a.smithi069.stdout:Apr 14 14:04:18 smithi069 podman[58811]: 2024-04-14 14:04:18.659800066 +0000 UTC m=+0.499918531 container cleanup 148fbcb552325f752f07b182e98ec63a0e7d0dd169222d4e66f95febb6fde931 (image=quay.io/prometheus/alertmanager:v0.23.0, name=ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-alertmanager-a, maintainer=The Prometheus Authors ) 2024-04-14T14:04:18.817 INFO:journalctl@ceph.alertmanager.a.smithi069.stdout:Apr 14 14:04:18 smithi069 bash[58811]: ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-alertmanager-a 2024-04-14T14:04:18.817 INFO:journalctl@ceph.alertmanager.a.smithi069.stdout:Apr 14 14:04:18 smithi069 bash[58837]: Error: no container with name or ID "ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-alertmanager.a" found: no such container 2024-04-14T14:04:18.844 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:04:18 smithi122 ceph-mon[25721]: pgmap v13: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:04:19.262 INFO:journalctl@ceph.alertmanager.a.smithi069.stdout:Apr 14 14:04:18 smithi069 podman[58825]: 2024-04-14 14:04:18.919308531 +0000 UTC m=+0.637714837 container remove 148fbcb552325f752f07b182e98ec63a0e7d0dd169222d4e66f95febb6fde931 (image=quay.io/prometheus/alertmanager:v0.23.0, name=ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-alertmanager-a, maintainer=The Prometheus Authors ) 2024-04-14T14:04:19.262 INFO:journalctl@ceph.alertmanager.a.smithi069.stdout:Apr 14 14:04:19 smithi069 podman[58825]: 2024-04-14 14:04:19.020021801 +0000 UTC m=+0.738428105 volume remove 6f15ef4706adf0e8f9a6fdb954c96e3cbcd62daa0b63ba5bb68d2b187f2f4372 2024-04-14T14:04:19.262 INFO:journalctl@ceph.alertmanager.a.smithi069.stdout:Apr 14 14:04:19 smithi069 systemd[1]: ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968@alertmanager.a.service: Deactivated successfully. 2024-04-14T14:04:19.262 INFO:journalctl@ceph.alertmanager.a.smithi069.stdout:Apr 14 14:04:19 smithi069 systemd[1]: Stopped Ceph alertmanager.a for dc326fc2-fa65-11ee-bc8e-c7b262605968. 2024-04-14T14:04:19.262 INFO:journalctl@ceph.alertmanager.a.smithi069.stdout:Apr 14 14:04:19 smithi069 systemd[1]: ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968@alertmanager.a.service: Consumed 1.081s CPU time. 2024-04-14T14:04:19.262 INFO:journalctl@ceph.alertmanager.a.smithi069.stdout:Apr 14 14:04:19 smithi069 systemd[1]: Starting Ceph alertmanager.a for dc326fc2-fa65-11ee-bc8e-c7b262605968... 2024-04-14T14:04:19.546 INFO:journalctl@ceph.alertmanager.a.smithi069.stdout:Apr 14 14:04:19 smithi069 podman[58908]: 2024-04-14 14:04:19.163750993 +0000 UTC m=+0.017951687 image pull ba2b418f427c0636d654de8757e830c80168e76482bcc46bb2138e569d6c91d4 quay.io/prometheus/alertmanager:v0.23.0 2024-04-14T14:04:19.546 INFO:journalctl@ceph.alertmanager.a.smithi069.stdout:Apr 14 14:04:19 smithi069 podman[58908]: 2024-04-14 14:04:19.287645595 +0000 UTC m=+0.141846290 volume create 4191fea74c2099335b5c03e4f08bfdefab90e7651f56bed2ceed4994d25dd4d1 2024-04-14T14:04:19.546 INFO:journalctl@ceph.alertmanager.a.smithi069.stdout:Apr 14 14:04:19 smithi069 podman[58908]: 2024-04-14 14:04:19.371507789 +0000 UTC m=+0.225708525 container create e300fc36e997f965f8a8646eeb7d84bec0d051e4a005771ba12d0a7572eea343 (image=quay.io/prometheus/alertmanager:v0.23.0, name=ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-alertmanager-a, maintainer=The Prometheus Authors ) 2024-04-14T14:04:19.817 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 14 14:04:19 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-y[25054]: ::ffff:172.21.15.122 - - [14/Apr/2024:14:04:19] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-04-14T14:04:19.817 INFO:journalctl@ceph.alertmanager.a.smithi069.stdout:Apr 14 14:04:19 smithi069 podman[58908]: 2024-04-14 14:04:19.632491504 +0000 UTC m=+0.486692198 container init e300fc36e997f965f8a8646eeb7d84bec0d051e4a005771ba12d0a7572eea343 (image=quay.io/prometheus/alertmanager:v0.23.0, name=ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-alertmanager-a, maintainer=The Prometheus Authors ) 2024-04-14T14:04:19.817 INFO:journalctl@ceph.alertmanager.a.smithi069.stdout:Apr 14 14:04:19 smithi069 podman[58908]: 2024-04-14 14:04:19.635343083 +0000 UTC m=+0.489543778 container start e300fc36e997f965f8a8646eeb7d84bec0d051e4a005771ba12d0a7572eea343 (image=quay.io/prometheus/alertmanager:v0.23.0, name=ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-alertmanager-a, maintainer=The Prometheus Authors ) 2024-04-14T14:04:19.817 INFO:journalctl@ceph.alertmanager.a.smithi069.stdout:Apr 14 14:04:19 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-alertmanager-a[58923]: level=info ts=2024-04-14T14:04:19.645Z caller=main.go:225 msg="Starting Alertmanager" version="(version=0.23.0, branch=HEAD, revision=61046b17771a57cfd4c4a51be370ab930a4d7d54)" 2024-04-14T14:04:19.817 INFO:journalctl@ceph.alertmanager.a.smithi069.stdout:Apr 14 14:04:19 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-alertmanager-a[58923]: level=info ts=2024-04-14T14:04:19.645Z caller=main.go:226 build_context="(go=go1.16.7, user=root@e21a959be8d2, date=20210825-10:48:55)" 2024-04-14T14:04:19.818 INFO:journalctl@ceph.alertmanager.a.smithi069.stdout:Apr 14 14:04:19 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-alertmanager-a[58923]: level=info ts=2024-04-14T14:04:19.647Z caller=cluster.go:184 component=cluster msg="setting advertise address explicitly" addr=172.21.15.69 port=9094 2024-04-14T14:04:19.818 INFO:journalctl@ceph.alertmanager.a.smithi069.stdout:Apr 14 14:04:19 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-alertmanager-a[58923]: level=info ts=2024-04-14T14:04:19.648Z caller=cluster.go:671 component=cluster msg="Waiting for gossip to settle..." interval=2s 2024-04-14T14:04:19.818 INFO:journalctl@ceph.alertmanager.a.smithi069.stdout:Apr 14 14:04:19 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-alertmanager-a[58923]: level=info ts=2024-04-14T14:04:19.674Z caller=coordinator.go:113 component=configuration msg="Loading configuration file" file=/etc/alertmanager/alertmanager.yml 2024-04-14T14:04:19.818 INFO:journalctl@ceph.alertmanager.a.smithi069.stdout:Apr 14 14:04:19 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-alertmanager-a[58923]: level=info ts=2024-04-14T14:04:19.674Z caller=coordinator.go:126 component=configuration msg="Completed loading of configuration file" file=/etc/alertmanager/alertmanager.yml 2024-04-14T14:04:19.818 INFO:journalctl@ceph.alertmanager.a.smithi069.stdout:Apr 14 14:04:19 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-alertmanager-a[58923]: level=info ts=2024-04-14T14:04:19.676Z caller=main.go:518 msg=Listening address=:9093 2024-04-14T14:04:19.818 INFO:journalctl@ceph.alertmanager.a.smithi069.stdout:Apr 14 14:04:19 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-alertmanager-a[58923]: level=info ts=2024-04-14T14:04:19.676Z caller=tls_config.go:191 msg="TLS is disabled." http2=false 2024-04-14T14:04:19.818 INFO:journalctl@ceph.alertmanager.a.smithi069.stdout:Apr 14 14:04:19 smithi069 bash[58908]: e300fc36e997f965f8a8646eeb7d84bec0d051e4a005771ba12d0a7572eea343 2024-04-14T14:04:19.818 INFO:journalctl@ceph.alertmanager.a.smithi069.stdout:Apr 14 14:04:19 smithi069 systemd[1]: Started Ceph alertmanager.a for dc326fc2-fa65-11ee-bc8e-c7b262605968. 2024-04-14T14:04:21.067 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:04:20 smithi069 ceph-mon[24818]: pgmap v14: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:04:21.067 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:04:20 smithi069 ceph-mon[24818]: from='mgr.24749 ' entity='mgr.x' 2024-04-14T14:04:21.067 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:04:20 smithi069 ceph-mon[24818]: from='mgr.24749 ' entity='mgr.x' 2024-04-14T14:04:21.067 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:04:20 smithi069 ceph-mon[24818]: Reconfiguring iscsi.foo.smithi069.yuumtr (dependencies changed)... 2024-04-14T14:04:21.067 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:04:20 smithi069 ceph-mon[24818]: from='mgr.24749 172.21.15.122:0/1892077634' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi069.yuumtr", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-04-14T14:04:21.067 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:04:20 smithi069 ceph-mon[24818]: from='mgr.24749 ' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi069.yuumtr", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-04-14T14:04:21.067 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:04:20 smithi069 ceph-mon[24818]: from='mgr.24749 172.21.15.122:0/1892077634' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T14:04:21.067 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:04:20 smithi069 ceph-mon[24818]: Reconfiguring daemon iscsi.foo.smithi069.yuumtr on smithi069 2024-04-14T14:04:21.068 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:04:20 smithi069 ceph-mon[29472]: pgmap v14: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:04:21.068 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:04:20 smithi069 ceph-mon[29472]: from='mgr.24749 ' entity='mgr.x' 2024-04-14T14:04:21.068 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:04:20 smithi069 ceph-mon[29472]: from='mgr.24749 ' entity='mgr.x' 2024-04-14T14:04:21.068 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:04:20 smithi069 ceph-mon[29472]: Reconfiguring iscsi.foo.smithi069.yuumtr (dependencies changed)... 2024-04-14T14:04:21.068 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:04:20 smithi069 ceph-mon[29472]: from='mgr.24749 172.21.15.122:0/1892077634' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi069.yuumtr", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-04-14T14:04:21.068 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:04:20 smithi069 ceph-mon[29472]: from='mgr.24749 ' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi069.yuumtr", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-04-14T14:04:21.068 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:04:20 smithi069 ceph-mon[29472]: from='mgr.24749 172.21.15.122:0/1892077634' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T14:04:21.068 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:04:20 smithi069 ceph-mon[29472]: Reconfiguring daemon iscsi.foo.smithi069.yuumtr on smithi069 2024-04-14T14:04:21.094 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:04:20 smithi122 ceph-mon[25721]: pgmap v14: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:04:21.094 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:04:20 smithi122 ceph-mon[25721]: from='mgr.24749 ' entity='mgr.x' 2024-04-14T14:04:21.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:04:20 smithi122 ceph-mon[25721]: from='mgr.24749 ' entity='mgr.x' 2024-04-14T14:04:21.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:04:20 smithi122 ceph-mon[25721]: Reconfiguring iscsi.foo.smithi069.yuumtr (dependencies changed)... 2024-04-14T14:04:21.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:04:20 smithi122 ceph-mon[25721]: from='mgr.24749 172.21.15.122:0/1892077634' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi069.yuumtr", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-04-14T14:04:21.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:04:20 smithi122 ceph-mon[25721]: from='mgr.24749 ' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi069.yuumtr", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-04-14T14:04:21.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:04:20 smithi122 ceph-mon[25721]: from='mgr.24749 172.21.15.122:0/1892077634' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T14:04:21.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:04:20 smithi122 ceph-mon[25721]: Reconfiguring daemon iscsi.foo.smithi069.yuumtr on smithi069 2024-04-14T14:04:22.067 INFO:journalctl@ceph.alertmanager.a.smithi069.stdout:Apr 14 14:04:21 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-alertmanager-a[58923]: level=info ts=2024-04-14T14:04:21.649Z caller=cluster.go:696 component=cluster msg="gossip not settled" polls=0 before=0 now=1 elapsed=2.000670556s 2024-04-14T14:04:22.595 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Apr 14 14:04:22 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-prometheus-a[42660]: ts=2024-04-14T14:04:22.314Z caller=manager.go:609 level=warn component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.69:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-04-14T14:04:23.067 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:04:22 smithi069 ceph-mon[24818]: pgmap v15: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:04:23.067 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:04:22 smithi069 ceph-mon[29472]: pgmap v15: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:04:23.094 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:04:22 smithi122 ceph-mon[25721]: pgmap v15: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:04:23.095 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Apr 14 14:04:22 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-x[45487]: 2024-04-14T14:04:22.908+0000 7f461db9a700 -1 log_channel(cephadm) log [ERR] : cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi069.yuumtr ... 2024-04-14T14:04:23.095 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Apr 14 14:04:22 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-x[45487]: Creating ceph-iscsi config... 2024-04-14T14:04:23.095 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Apr 14 14:04:22 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-x[45487]: Write file: /var/lib/ceph/dc326fc2-fa65-11ee-bc8e-c7b262605968/iscsi.foo.smithi069.yuumtr/iscsi-gateway.cfg 2024-04-14T14:04:23.095 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Apr 14 14:04:22 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-x[45487]: Write file: /var/lib/ceph/dc326fc2-fa65-11ee-bc8e-c7b262605968/iscsi.foo.smithi069.yuumtr/tcmu-runner-entrypoint.sh 2024-04-14T14:04:23.095 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Apr 14 14:04:22 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-x[45487]: Non-zero exit code 1 from systemctl restart ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968@iscsi.foo.smithi069.yuumtr 2024-04-14T14:04:23.095 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Apr 14 14:04:22 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-x[45487]: systemctl: stderr Job for ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968@iscsi.foo.smithi069.yuumtr.service failed because the control process exited with error code. 2024-04-14T14:04:23.095 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Apr 14 14:04:22 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-x[45487]: systemctl: stderr See "systemctl status ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968@iscsi.foo.smithi069.yuumtr.service" and "journalctl -xeu ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968@iscsi.foo.smithi069.yuumtr.service" for details. 2024-04-14T14:04:23.095 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Apr 14 14:04:22 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-x[45487]: Traceback (most recent call last): 2024-04-14T14:04:23.095 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Apr 14 14:04:22 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-x[45487]: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-04-14T14:04:23.096 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Apr 14 14:04:22 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-x[45487]: return _run_code(code, main_globals, None, 2024-04-14T14:04:23.096 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Apr 14 14:04:22 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-x[45487]: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-04-14T14:04:23.096 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Apr 14 14:04:22 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-x[45487]: exec(code, run_globals) 2024-04-14T14:04:23.096 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Apr 14 14:04:22 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-x[45487]: File "/var/lib/ceph/dc326fc2-fa65-11ee-bc8e-c7b262605968/cephadm.de2a8faf91349910526b51da93d8bd42d57efdd2e93c32cd09ae64ecc1748ae9/__main__.py", line 5608, in 2024-04-14T14:04:23.096 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Apr 14 14:04:22 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-x[45487]: File "/var/lib/ceph/dc326fc2-fa65-11ee-bc8e-c7b262605968/cephadm.de2a8faf91349910526b51da93d8bd42d57efdd2e93c32cd09ae64ecc1748ae9/__main__.py", line 5596, in main 2024-04-14T14:04:23.096 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Apr 14 14:04:22 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-x[45487]: File "/var/lib/ceph/dc326fc2-fa65-11ee-bc8e-c7b262605968/cephadm.de2a8faf91349910526b51da93d8bd42d57efdd2e93c32cd09ae64ecc1748ae9/__main__.py", line 3096, in command_deploy_from 2024-04-14T14:04:23.096 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Apr 14 14:04:22 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-x[45487]: File "/var/lib/ceph/dc326fc2-fa65-11ee-bc8e-c7b262605968/cephadm.de2a8faf91349910526b51da93d8bd42d57efdd2e93c32cd09ae64ecc1748ae9/__main__.py", line 3131, in _common_deploy 2024-04-14T14:04:23.096 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Apr 14 14:04:22 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-x[45487]: File "/var/lib/ceph/dc326fc2-fa65-11ee-bc8e-c7b262605968/cephadm.de2a8faf91349910526b51da93d8bd42d57efdd2e93c32cd09ae64ecc1748ae9/__main__.py", line 3151, in _deploy_daemon_container 2024-04-14T14:04:23.096 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Apr 14 14:04:22 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-x[45487]: File "/var/lib/ceph/dc326fc2-fa65-11ee-bc8e-c7b262605968/cephadm.de2a8faf91349910526b51da93d8bd42d57efdd2e93c32cd09ae64ecc1748ae9/__main__.py", line 1123, in deploy_daemon 2024-04-14T14:04:23.096 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Apr 14 14:04:22 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-x[45487]: File "/var/lib/ceph/dc326fc2-fa65-11ee-bc8e-c7b262605968/cephadm.de2a8faf91349910526b51da93d8bd42d57efdd2e93c32cd09ae64ecc1748ae9/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-04-14T14:04:23.097 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Apr 14 14:04:22 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-x[45487]: RuntimeError: Failed command: systemctl restart ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968@iscsi.foo.smithi069.yuumtr: Job for ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968@iscsi.foo.smithi069.yuumtr.service failed because the control process exited with error code. 2024-04-14T14:04:23.097 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Apr 14 14:04:22 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-x[45487]: See "systemctl status ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968@iscsi.foo.smithi069.yuumtr.service" and "journalctl -xeu ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968@iscsi.foo.smithi069.yuumtr.service" for details. 2024-04-14T14:04:23.097 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Apr 14 14:04:22 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-x[45487]: Traceback (most recent call last): 2024-04-14T14:04:23.097 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Apr 14 14:04:22 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-x[45487]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1114, in _check_daemons 2024-04-14T14:04:23.097 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Apr 14 14:04:22 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-x[45487]: self.mgr._daemon_action(daemon_spec, action=action) 2024-04-14T14:04:23.097 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Apr 14 14:04:22 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-x[45487]: File "/usr/share/ceph/mgr/cephadm/module.py", line 2424, in _daemon_action 2024-04-14T14:04:23.097 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Apr 14 14:04:22 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-x[45487]: CephadmServe(self)._create_daemon(daemon_spec, reconfig=(action == 'reconfig'))) 2024-04-14T14:04:23.097 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Apr 14 14:04:22 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-x[45487]: File "/usr/share/ceph/mgr/cephadm/module.py", line 779, in wait_async 2024-04-14T14:04:23.097 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Apr 14 14:04:22 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-x[45487]: return self.event_loop.get_result(coro, timeout) 2024-04-14T14:04:23.097 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Apr 14 14:04:22 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-x[45487]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 136, in get_result 2024-04-14T14:04:23.097 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Apr 14 14:04:22 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-x[45487]: return future.result(timeout) 2024-04-14T14:04:23.098 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Apr 14 14:04:22 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-x[45487]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2024-04-14T14:04:23.098 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Apr 14 14:04:22 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-x[45487]: return self.__get_result() 2024-04-14T14:04:23.098 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Apr 14 14:04:22 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-x[45487]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2024-04-14T14:04:23.098 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Apr 14 14:04:22 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-x[45487]: raise self._exception 2024-04-14T14:04:23.098 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Apr 14 14:04:22 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-x[45487]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1374, in _create_daemon 2024-04-14T14:04:23.098 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Apr 14 14:04:22 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-x[45487]: config_blobs=daemon_spec.final_config, 2024-04-14T14:04:23.098 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Apr 14 14:04:22 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-x[45487]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1685, in _run_cephadm 2024-04-14T14:04:23.098 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Apr 14 14:04:22 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-x[45487]: f'cephadm exited with an error code: {code}, stderr: {err}') 2024-04-14T14:04:23.098 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Apr 14 14:04:22 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-x[45487]: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi069.yuumtr ... 2024-04-14T14:04:23.098 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Apr 14 14:04:22 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-x[45487]: Creating ceph-iscsi config... 2024-04-14T14:04:23.098 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Apr 14 14:04:22 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-x[45487]: Write file: /var/lib/ceph/dc326fc2-fa65-11ee-bc8e-c7b262605968/iscsi.foo.smithi069.yuumtr/iscsi-gateway.cfg 2024-04-14T14:04:23.098 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Apr 14 14:04:22 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-x[45487]: Write file: /var/lib/ceph/dc326fc2-fa65-11ee-bc8e-c7b262605968/iscsi.foo.smithi069.yuumtr/tcmu-runner-entrypoint.sh 2024-04-14T14:04:23.098 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Apr 14 14:04:22 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-x[45487]: Non-zero exit code 1 from systemctl restart ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968@iscsi.foo.smithi069.yuumtr 2024-04-14T14:04:23.099 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Apr 14 14:04:22 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-x[45487]: systemctl: stderr Job for ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968@iscsi.foo.smithi069.yuumtr.service failed because the control process exited with error code. 2024-04-14T14:04:23.099 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Apr 14 14:04:22 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-x[45487]: systemctl: stderr See "systemctl status ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968@iscsi.foo.smithi069.yuumtr.service" and "journalctl -xeu ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968@iscsi.foo.smithi069.yuumtr.service" for details. 2024-04-14T14:04:23.099 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Apr 14 14:04:22 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-x[45487]: Traceback (most recent call last): 2024-04-14T14:04:23.099 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Apr 14 14:04:22 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-x[45487]: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-04-14T14:04:23.099 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Apr 14 14:04:22 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-x[45487]: return _run_code(code, main_globals, None, 2024-04-14T14:04:23.099 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Apr 14 14:04:22 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-x[45487]: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-04-14T14:04:23.099 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Apr 14 14:04:22 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-x[45487]: exec(code, run_globals) 2024-04-14T14:04:23.099 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Apr 14 14:04:22 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-x[45487]: File "/var/lib/ceph/dc326fc2-fa65-11ee-bc8e-c7b262605968/cephadm.de2a8faf91349910526b51da93d8bd42d57efdd2e93c32cd09ae64ecc1748ae9/__main__.py", line 5608, in 2024-04-14T14:04:23.099 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Apr 14 14:04:22 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-x[45487]: File "/var/lib/ceph/dc326fc2-fa65-11ee-bc8e-c7b262605968/cephadm.de2a8faf91349910526b51da93d8bd42d57efdd2e93c32cd09ae64ecc1748ae9/__main__.py", line 5596, in main 2024-04-14T14:04:23.099 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Apr 14 14:04:22 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-x[45487]: File "/var/lib/ceph/dc326fc2-fa65-11ee-bc8e-c7b262605968/cephadm.de2a8faf91349910526b51da93d8bd42d57efdd2e93c32cd09ae64ecc1748ae9/__main__.py", line 3096, in command_deploy_from 2024-04-14T14:04:23.099 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Apr 14 14:04:22 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-x[45487]: File "/var/lib/ceph/dc326fc2-fa65-11ee-bc8e-c7b262605968/cephadm.de2a8faf91349910526b51da93d8bd42d57efdd2e93c32cd09ae64ecc1748ae9/__main__.py", line 3131, in _common_deploy 2024-04-14T14:04:23.099 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Apr 14 14:04:22 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-x[45487]: File "/var/lib/ceph/dc326fc2-fa65-11ee-bc8e-c7b262605968/cephadm.de2a8faf91349910526b51da93d8bd42d57efdd2e93c32cd09ae64ecc1748ae9/__main__.py", line 3151, in _deploy_daemon_container 2024-04-14T14:04:23.099 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Apr 14 14:04:22 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-x[45487]: File "/var/lib/ceph/dc326fc2-fa65-11ee-bc8e-c7b262605968/cephadm.de2a8faf91349910526b51da93d8bd42d57efdd2e93c32cd09ae64ecc1748ae9/__main__.py", line 1123, in deploy_daemon 2024-04-14T14:04:23.099 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Apr 14 14:04:22 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-x[45487]: File "/var/lib/ceph/dc326fc2-fa65-11ee-bc8e-c7b262605968/cephadm.de2a8faf91349910526b51da93d8bd42d57efdd2e93c32cd09ae64ecc1748ae9/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-04-14T14:04:23.100 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Apr 14 14:04:22 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-x[45487]: RuntimeError: Failed command: systemctl restart ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968@iscsi.foo.smithi069.yuumtr: Job for ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968@iscsi.foo.smithi069.yuumtr.service failed because the control process exited with error code. 2024-04-14T14:04:23.100 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Apr 14 14:04:22 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-x[45487]: See "systemctl status ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968@iscsi.foo.smithi069.yuumtr.service" and "journalctl -xeu ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968@iscsi.foo.smithi069.yuumtr.service" for details. 2024-04-14T14:04:25.067 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:04:24 smithi069 ceph-mon[24818]: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi069.yuumtr ... 2024-04-14T14:04:25.067 INFO:journalctl@ceph.mon.a.smithi069.stdout: Creating ceph-iscsi config... 2024-04-14T14:04:25.067 INFO:journalctl@ceph.mon.a.smithi069.stdout: Write file: /var/lib/ceph/dc326fc2-fa65-11ee-bc8e-c7b262605968/iscsi.foo.smithi069.yuumtr/iscsi-gateway.cfg 2024-04-14T14:04:25.067 INFO:journalctl@ceph.mon.a.smithi069.stdout: Write file: /var/lib/ceph/dc326fc2-fa65-11ee-bc8e-c7b262605968/iscsi.foo.smithi069.yuumtr/tcmu-runner-entrypoint.sh 2024-04-14T14:04:25.067 INFO:journalctl@ceph.mon.a.smithi069.stdout: Non-zero exit code 1 from systemctl restart ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968@iscsi.foo.smithi069.yuumtr 2024-04-14T14:04:25.067 INFO:journalctl@ceph.mon.a.smithi069.stdout: systemctl: stderr Job for ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968@iscsi.foo.smithi069.yuumtr.service failed because the control process exited with error code. 2024-04-14T14:04:25.067 INFO:journalctl@ceph.mon.a.smithi069.stdout: systemctl: stderr See "systemctl status ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968@iscsi.foo.smithi069.yuumtr.service" and "journalctl -xeu ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968@iscsi.foo.smithi069.yuumtr.service" for details. 2024-04-14T14:04:25.067 INFO:journalctl@ceph.mon.a.smithi069.stdout: Traceback (most recent call last): 2024-04-14T14:04:25.067 INFO:journalctl@ceph.mon.a.smithi069.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-04-14T14:04:25.068 INFO:journalctl@ceph.mon.a.smithi069.stdout: return _run_code(code, main_globals, None, 2024-04-14T14:04:25.068 INFO:journalctl@ceph.mon.a.smithi069.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-04-14T14:04:25.068 INFO:journalctl@ceph.mon.a.smithi069.stdout: exec(code, run_globals) 2024-04-14T14:04:25.068 INFO:journalctl@ceph.mon.a.smithi069.stdout: File "/var/lib/ceph/dc326fc2-fa65-11ee-bc8e-c7b262605968/cephadm.de2a8faf91349910526b51da93d8bd42d57efdd2e93c32cd09ae64ecc1748ae9/__main__.py", line 5608, in 2024-04-14T14:04:25.068 INFO:journalctl@ceph.mon.a.smithi069.stdout: File "/var/lib/ceph/dc326fc2-fa65-11ee-bc8e-c7b262605968/cephadm.de2a8faf91349910526b51da93d8bd42d57efdd2e93c32cd09ae64ecc1748ae9/__main__.py", line 5596, in main 2024-04-14T14:04:25.068 INFO:journalctl@ceph.mon.a.smithi069.stdout: File "/var/lib/ceph/dc326fc2-fa65-11ee-bc8e-c7b262605968/cephadm.de2a8faf91349910526b51da93d8bd42d57efdd2e93c32cd09ae64ecc1748ae9/__main__.py", line 3096, in command_deploy_from 2024-04-14T14:04:25.068 INFO:journalctl@ceph.mon.a.smithi069.stdout: File "/var/lib/ceph/dc326fc2-fa65-11ee-bc8e-c7b262605968/cephadm.de2a8faf91349910526b51da93d8bd42d57efdd2e93c32cd09ae64ecc1748ae9/__main__.py", line 3131, in _common_deploy 2024-04-14T14:04:25.068 INFO:journalctl@ceph.mon.a.smithi069.stdout: File "/var/lib/ceph/dc326fc2-fa65-11ee-bc8e-c7b262605968/cephadm.de2a8faf91349910526b51da93d8bd42d57efdd2e93c32cd09ae64ecc1748ae9/__main__.py", line 3151, in _deploy_daemon_container 2024-04-14T14:04:25.068 INFO:journalctl@ceph.mon.a.smithi069.stdout: File "/var/lib/ceph/dc326fc2-fa65-11ee-bc8e-c7b262605968/cephadm.de2a8faf91349910526b51da93d8bd42d57efdd2e93c32cd09ae64ecc1748ae9/__main__.py", line 1123, in deploy_daemon 2024-04-14T14:04:25.068 INFO:journalctl@ceph.mon.a.smithi069.stdout: File "/var/lib/ceph/dc326fc2-fa65-11ee-bc8e-c7b262605968/cephadm.de2a8faf91349910526b51da93d8bd42d57efdd2e93c32cd09ae64ecc1748ae9/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-04-14T14:04:25.068 INFO:journalctl@ceph.mon.a.smithi069.stdout: RuntimeError: Failed command: systemctl restart ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968@iscsi.foo.smithi069.yuumtr: Job for ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968@iscsi.foo.smithi069.yuumtr.service failed because the control process exited with error code. 2024-04-14T14:04:25.068 INFO:journalctl@ceph.mon.a.smithi069.stdout: See "systemctl status ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968@iscsi.foo.smithi069.yuumtr.service" and "journalctl -xeu ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968@iscsi.foo.smithi069.yuumtr.service" for details. 2024-04-14T14:04:25.068 INFO:journalctl@ceph.mon.a.smithi069.stdout: Traceback (most recent call last): 2024-04-14T14:04:25.068 INFO:journalctl@ceph.mon.a.smithi069.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1114, in _check_daemons 2024-04-14T14:04:25.068 INFO:journalctl@ceph.mon.a.smithi069.stdout: self.mgr._daemon_action(daemon_spec, action=action) 2024-04-14T14:04:25.069 INFO:journalctl@ceph.mon.a.smithi069.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 2424, in _daemon_action 2024-04-14T14:04:25.069 INFO:journalctl@ceph.mon.a.smithi069.stdout: CephadmServe(self)._create_daemon(daemon_spec, reconfig=(action == 'reconfig'))) 2024-04-14T14:04:25.069 INFO:journalctl@ceph.mon.a.smithi069.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 779, in wait_async 2024-04-14T14:04:25.069 INFO:journalctl@ceph.mon.a.smithi069.stdout: return self.event_loop.get_result(coro, timeout) 2024-04-14T14:04:25.069 INFO:journalctl@ceph.mon.a.smithi069.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 136, in get_result 2024-04-14T14:04:25.069 INFO:journalctl@ceph.mon.a.smithi069.stdout: return future.result(timeout) 2024-04-14T14:04:25.069 INFO:journalctl@ceph.mon.a.smithi069.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2024-04-14T14:04:25.069 INFO:journalctl@ceph.mon.a.smithi069.stdout: return self.__get_result() 2024-04-14T14:04:25.069 INFO:journalctl@ceph.mon.a.smithi069.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2024-04-14T14:04:25.069 INFO:journalctl@ceph.mon.a.smithi069.stdout: raise self._exception 2024-04-14T14:04:25.069 INFO:journalctl@ceph.mon.a.smithi069.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1374, in _create_daemon 2024-04-14T14:04:25.069 INFO:journalctl@ceph.mon.a.smithi069.stdout: config_blobs=daemon_spec.final_config, 2024-04-14T14:04:25.069 INFO:journalctl@ceph.mon.a.smithi069.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1685, in _run_cephadm 2024-04-14T14:04:25.069 INFO:journalctl@ceph.mon.a.smithi069.stdout: f'cephadm exited with an error code: {code}, stderr: {err}') 2024-04-14T14:04:25.069 INFO:journalctl@ceph.mon.a.smithi069.stdout: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi069.yuumtr ... 2024-04-14T14:04:25.070 INFO:journalctl@ceph.mon.a.smithi069.stdout: Creating ceph-iscsi config... 2024-04-14T14:04:25.070 INFO:journalctl@ceph.mon.a.smithi069.stdout: Write file: /var/lib/ceph/dc326fc2-fa65-11ee-bc8e-c7b262605968/iscsi.foo.smithi069.yuumtr/iscsi-gateway.cfg 2024-04-14T14:04:25.070 INFO:journalctl@ceph.mon.a.smithi069.stdout: Write file: /var/lib/ceph/dc326fc2-fa65-11ee-bc8e-c7b262605968/iscsi.foo.smithi069.yuumtr/tcmu-runner-entrypoint.sh 2024-04-14T14:04:25.070 INFO:journalctl@ceph.mon.a.smithi069.stdout: Non-zero exit code 1 from systemctl restart ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968@iscsi.foo.smithi069.yuumtr 2024-04-14T14:04:25.070 INFO:journalctl@ceph.mon.a.smithi069.stdout: systemctl: stderr Job for ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968@iscsi.foo.smithi069.yuumtr.service failed because the control process exited with error code. 2024-04-14T14:04:25.070 INFO:journalctl@ceph.mon.a.smithi069.stdout: systemctl: stderr See "systemctl status ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968@iscsi.foo.smithi069.yuumtr.service" and "journalctl -xeu ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968@iscsi.foo.smithi069.yuumtr.service" for details. 2024-04-14T14:04:25.070 INFO:journalctl@ceph.mon.a.smithi069.stdout: Traceback (most recent call last): 2024-04-14T14:04:25.070 INFO:journalctl@ceph.mon.a.smithi069.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-04-14T14:04:25.070 INFO:journalctl@ceph.mon.a.smithi069.stdout: return _run_code(code, main_globals, None, 2024-04-14T14:04:25.070 INFO:journalctl@ceph.mon.a.smithi069.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-04-14T14:04:25.070 INFO:journalctl@ceph.mon.a.smithi069.stdout: exec(code, run_globals) 2024-04-14T14:04:25.070 INFO:journalctl@ceph.mon.a.smithi069.stdout: File "/var/lib/ceph/dc326fc2-fa65-11ee-bc8e-c7b262605968/cephadm.de2a8faf91349910526b51da93d8bd42d57efdd2e93c32cd09ae64ecc1748ae9/__main__.py", line 5608, in 2024-04-14T14:04:25.070 INFO:journalctl@ceph.mon.a.smithi069.stdout: File "/var/lib/ceph/dc326fc2-fa65-11ee-bc8e-c7b262605968/cephadm.de2a8faf91349910526b51da93d8bd42d57efdd2e93c32cd09ae64ecc1748ae9/__main__.py", line 5596, in main 2024-04-14T14:04:25.070 INFO:journalctl@ceph.mon.a.smithi069.stdout: File "/var/lib/ceph/dc326fc2-fa65-11ee-bc8e-c7b262605968/cephadm.de2a8faf91349910526b51da93d8bd42d57efdd2e93c32cd09ae64ecc1748ae9/__main__.py", line 3096, in command_deploy_from 2024-04-14T14:04:25.070 INFO:journalctl@ceph.mon.a.smithi069.stdout: File "/var/lib/ceph/dc326fc2-fa65-11ee-bc8e-c7b262605968/cephadm.de2a8faf91349910526b51da93d8bd42d57efdd2e93c32cd09ae64ecc1748ae9/__main__.py", line 3131, in _common_deploy 2024-04-14T14:04:25.071 INFO:journalctl@ceph.mon.a.smithi069.stdout: File "/var/lib/ceph/dc326fc2-fa65-11ee-bc8e-c7b262605968/cephadm.de2a8faf91349910526b51da93d8bd42d57efdd2e93c32cd09ae64ecc1748ae9/__main__.py", line 3151, in _deploy_daemon_container 2024-04-14T14:04:25.071 INFO:journalctl@ceph.mon.a.smithi069.stdout: File "/var/lib/ceph/dc326fc2-fa65-11ee-bc8e-c7b262605968/cephadm.de2a8faf91349910526b51da93d8bd42d57efdd2e93c32cd09ae64ecc1748ae9/__main__.py", line 1123, in deploy_daemon 2024-04-14T14:04:25.071 INFO:journalctl@ceph.mon.a.smithi069.stdout: File "/var/lib/ceph/dc326fc2-fa65-11ee-bc8e-c7b262605968/cephadm.de2a8faf91349910526b51da93d8bd42d57efdd2e93c32cd09ae64ecc1748ae9/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-04-14T14:04:25.071 INFO:journalctl@ceph.mon.a.smithi069.stdout: RuntimeError: Failed command: systemctl restart ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968@iscsi.foo.smithi069.yuumtr: Job for ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968@iscsi.foo.smithi069.yuumtr.service failed because the control process exited with error code. 2024-04-14T14:04:25.071 INFO:journalctl@ceph.mon.a.smithi069.stdout: See "systemctl status ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968@iscsi.foo.smithi069.yuumtr.service" and "journalctl -xeu ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968@iscsi.foo.smithi069.yuumtr.service" for details. 2024-04-14T14:04:25.071 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:04:24 smithi069 ceph-mon[24818]: Reconfiguring prometheus.a (dependencies changed)... 2024-04-14T14:04:25.071 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:04:24 smithi069 ceph-mon[24818]: Reconfiguring daemon prometheus.a on smithi122 2024-04-14T14:04:25.071 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:04:24 smithi069 ceph-mon[24818]: pgmap v16: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:04:25.072 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:04:24 smithi069 ceph-mon[29472]: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi069.yuumtr ... 2024-04-14T14:04:25.072 INFO:journalctl@ceph.mon.c.smithi069.stdout: Creating ceph-iscsi config... 2024-04-14T14:04:25.072 INFO:journalctl@ceph.mon.c.smithi069.stdout: Write file: /var/lib/ceph/dc326fc2-fa65-11ee-bc8e-c7b262605968/iscsi.foo.smithi069.yuumtr/iscsi-gateway.cfg 2024-04-14T14:04:25.072 INFO:journalctl@ceph.mon.c.smithi069.stdout: Write file: /var/lib/ceph/dc326fc2-fa65-11ee-bc8e-c7b262605968/iscsi.foo.smithi069.yuumtr/tcmu-runner-entrypoint.sh 2024-04-14T14:04:25.072 INFO:journalctl@ceph.mon.c.smithi069.stdout: Non-zero exit code 1 from systemctl restart ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968@iscsi.foo.smithi069.yuumtr 2024-04-14T14:04:25.072 INFO:journalctl@ceph.mon.c.smithi069.stdout: systemctl: stderr Job for ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968@iscsi.foo.smithi069.yuumtr.service failed because the control process exited with error code. 2024-04-14T14:04:25.072 INFO:journalctl@ceph.mon.c.smithi069.stdout: systemctl: stderr See "systemctl status ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968@iscsi.foo.smithi069.yuumtr.service" and "journalctl -xeu ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968@iscsi.foo.smithi069.yuumtr.service" for details. 2024-04-14T14:04:25.072 INFO:journalctl@ceph.mon.c.smithi069.stdout: Traceback (most recent call last): 2024-04-14T14:04:25.072 INFO:journalctl@ceph.mon.c.smithi069.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-04-14T14:04:25.072 INFO:journalctl@ceph.mon.c.smithi069.stdout: return _run_code(code, main_globals, None, 2024-04-14T14:04:25.072 INFO:journalctl@ceph.mon.c.smithi069.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-04-14T14:04:25.072 INFO:journalctl@ceph.mon.c.smithi069.stdout: exec(code, run_globals) 2024-04-14T14:04:25.073 INFO:journalctl@ceph.mon.c.smithi069.stdout: File "/var/lib/ceph/dc326fc2-fa65-11ee-bc8e-c7b262605968/cephadm.de2a8faf91349910526b51da93d8bd42d57efdd2e93c32cd09ae64ecc1748ae9/__main__.py", line 5608, in 2024-04-14T14:04:25.073 INFO:journalctl@ceph.mon.c.smithi069.stdout: File "/var/lib/ceph/dc326fc2-fa65-11ee-bc8e-c7b262605968/cephadm.de2a8faf91349910526b51da93d8bd42d57efdd2e93c32cd09ae64ecc1748ae9/__main__.py", line 5596, in main 2024-04-14T14:04:25.073 INFO:journalctl@ceph.mon.c.smithi069.stdout: File "/var/lib/ceph/dc326fc2-fa65-11ee-bc8e-c7b262605968/cephadm.de2a8faf91349910526b51da93d8bd42d57efdd2e93c32cd09ae64ecc1748ae9/__main__.py", line 3096, in command_deploy_from 2024-04-14T14:04:25.073 INFO:journalctl@ceph.mon.c.smithi069.stdout: File "/var/lib/ceph/dc326fc2-fa65-11ee-bc8e-c7b262605968/cephadm.de2a8faf91349910526b51da93d8bd42d57efdd2e93c32cd09ae64ecc1748ae9/__main__.py", line 3131, in _common_deploy 2024-04-14T14:04:25.073 INFO:journalctl@ceph.mon.c.smithi069.stdout: File "/var/lib/ceph/dc326fc2-fa65-11ee-bc8e-c7b262605968/cephadm.de2a8faf91349910526b51da93d8bd42d57efdd2e93c32cd09ae64ecc1748ae9/__main__.py", line 3151, in _deploy_daemon_container 2024-04-14T14:04:25.073 INFO:journalctl@ceph.mon.c.smithi069.stdout: File "/var/lib/ceph/dc326fc2-fa65-11ee-bc8e-c7b262605968/cephadm.de2a8faf91349910526b51da93d8bd42d57efdd2e93c32cd09ae64ecc1748ae9/__main__.py", line 1123, in deploy_daemon 2024-04-14T14:04:25.073 INFO:journalctl@ceph.mon.c.smithi069.stdout: File "/var/lib/ceph/dc326fc2-fa65-11ee-bc8e-c7b262605968/cephadm.de2a8faf91349910526b51da93d8bd42d57efdd2e93c32cd09ae64ecc1748ae9/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-04-14T14:04:25.073 INFO:journalctl@ceph.mon.c.smithi069.stdout: RuntimeError: Failed command: systemctl restart ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968@iscsi.foo.smithi069.yuumtr: Job for ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968@iscsi.foo.smithi069.yuumtr.service failed because the control process exited with error code. 2024-04-14T14:04:25.073 INFO:journalctl@ceph.mon.c.smithi069.stdout: See "systemctl status ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968@iscsi.foo.smithi069.yuumtr.service" and "journalctl -xeu ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968@iscsi.foo.smithi069.yuumtr.service" for details. 2024-04-14T14:04:25.073 INFO:journalctl@ceph.mon.c.smithi069.stdout: Traceback (most recent call last): 2024-04-14T14:04:25.073 INFO:journalctl@ceph.mon.c.smithi069.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1114, in _check_daemons 2024-04-14T14:04:25.073 INFO:journalctl@ceph.mon.c.smithi069.stdout: self.mgr._daemon_action(daemon_spec, action=action) 2024-04-14T14:04:25.073 INFO:journalctl@ceph.mon.c.smithi069.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 2424, in _daemon_action 2024-04-14T14:04:25.074 INFO:journalctl@ceph.mon.c.smithi069.stdout: CephadmServe(self)._create_daemon(daemon_spec, reconfig=(action == 'reconfig'))) 2024-04-14T14:04:25.074 INFO:journalctl@ceph.mon.c.smithi069.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 779, in wait_async 2024-04-14T14:04:25.074 INFO:journalctl@ceph.mon.c.smithi069.stdout: return self.event_loop.get_result(coro, timeout) 2024-04-14T14:04:25.074 INFO:journalctl@ceph.mon.c.smithi069.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 136, in get_result 2024-04-14T14:04:25.074 INFO:journalctl@ceph.mon.c.smithi069.stdout: return future.result(timeout) 2024-04-14T14:04:25.074 INFO:journalctl@ceph.mon.c.smithi069.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2024-04-14T14:04:25.074 INFO:journalctl@ceph.mon.c.smithi069.stdout: return self.__get_result() 2024-04-14T14:04:25.074 INFO:journalctl@ceph.mon.c.smithi069.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2024-04-14T14:04:25.074 INFO:journalctl@ceph.mon.c.smithi069.stdout: raise self._exception 2024-04-14T14:04:25.074 INFO:journalctl@ceph.mon.c.smithi069.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1374, in _create_daemon 2024-04-14T14:04:25.074 INFO:journalctl@ceph.mon.c.smithi069.stdout: config_blobs=daemon_spec.final_config, 2024-04-14T14:04:25.074 INFO:journalctl@ceph.mon.c.smithi069.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1685, in _run_cephadm 2024-04-14T14:04:25.074 INFO:journalctl@ceph.mon.c.smithi069.stdout: f'cephadm exited with an error code: {code}, stderr: {err}') 2024-04-14T14:04:25.074 INFO:journalctl@ceph.mon.c.smithi069.stdout: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi069.yuumtr ... 2024-04-14T14:04:25.075 INFO:journalctl@ceph.mon.c.smithi069.stdout: Creating ceph-iscsi config... 2024-04-14T14:04:25.075 INFO:journalctl@ceph.mon.c.smithi069.stdout: Write file: /var/lib/ceph/dc326fc2-fa65-11ee-bc8e-c7b262605968/iscsi.foo.smithi069.yuumtr/iscsi-gateway.cfg 2024-04-14T14:04:25.075 INFO:journalctl@ceph.mon.c.smithi069.stdout: Write file: /var/lib/ceph/dc326fc2-fa65-11ee-bc8e-c7b262605968/iscsi.foo.smithi069.yuumtr/tcmu-runner-entrypoint.sh 2024-04-14T14:04:25.075 INFO:journalctl@ceph.mon.c.smithi069.stdout: Non-zero exit code 1 from systemctl restart ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968@iscsi.foo.smithi069.yuumtr 2024-04-14T14:04:25.075 INFO:journalctl@ceph.mon.c.smithi069.stdout: systemctl: stderr Job for ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968@iscsi.foo.smithi069.yuumtr.service failed because the control process exited with error code. 2024-04-14T14:04:25.075 INFO:journalctl@ceph.mon.c.smithi069.stdout: systemctl: stderr See "systemctl status ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968@iscsi.foo.smithi069.yuumtr.service" and "journalctl -xeu ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968@iscsi.foo.smithi069.yuumtr.service" for details. 2024-04-14T14:04:25.075 INFO:journalctl@ceph.mon.c.smithi069.stdout: Traceback (most recent call last): 2024-04-14T14:04:25.075 INFO:journalctl@ceph.mon.c.smithi069.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-04-14T14:04:25.075 INFO:journalctl@ceph.mon.c.smithi069.stdout: return _run_code(code, main_globals, None, 2024-04-14T14:04:25.075 INFO:journalctl@ceph.mon.c.smithi069.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-04-14T14:04:25.075 INFO:journalctl@ceph.mon.c.smithi069.stdout: exec(code, run_globals) 2024-04-14T14:04:25.075 INFO:journalctl@ceph.mon.c.smithi069.stdout: File "/var/lib/ceph/dc326fc2-fa65-11ee-bc8e-c7b262605968/cephadm.de2a8faf91349910526b51da93d8bd42d57efdd2e93c32cd09ae64ecc1748ae9/__main__.py", line 5608, in 2024-04-14T14:04:25.075 INFO:journalctl@ceph.mon.c.smithi069.stdout: File "/var/lib/ceph/dc326fc2-fa65-11ee-bc8e-c7b262605968/cephadm.de2a8faf91349910526b51da93d8bd42d57efdd2e93c32cd09ae64ecc1748ae9/__main__.py", line 5596, in main 2024-04-14T14:04:25.075 INFO:journalctl@ceph.mon.c.smithi069.stdout: File "/var/lib/ceph/dc326fc2-fa65-11ee-bc8e-c7b262605968/cephadm.de2a8faf91349910526b51da93d8bd42d57efdd2e93c32cd09ae64ecc1748ae9/__main__.py", line 3096, in command_deploy_from 2024-04-14T14:04:25.076 INFO:journalctl@ceph.mon.c.smithi069.stdout: File "/var/lib/ceph/dc326fc2-fa65-11ee-bc8e-c7b262605968/cephadm.de2a8faf91349910526b51da93d8bd42d57efdd2e93c32cd09ae64ecc1748ae9/__main__.py", line 3131, in _common_deploy 2024-04-14T14:04:25.076 INFO:journalctl@ceph.mon.c.smithi069.stdout: File "/var/lib/ceph/dc326fc2-fa65-11ee-bc8e-c7b262605968/cephadm.de2a8faf91349910526b51da93d8bd42d57efdd2e93c32cd09ae64ecc1748ae9/__main__.py", line 3151, in _deploy_daemon_container 2024-04-14T14:04:25.076 INFO:journalctl@ceph.mon.c.smithi069.stdout: File "/var/lib/ceph/dc326fc2-fa65-11ee-bc8e-c7b262605968/cephadm.de2a8faf91349910526b51da93d8bd42d57efdd2e93c32cd09ae64ecc1748ae9/__main__.py", line 1123, in deploy_daemon 2024-04-14T14:04:25.076 INFO:journalctl@ceph.mon.c.smithi069.stdout: File "/var/lib/ceph/dc326fc2-fa65-11ee-bc8e-c7b262605968/cephadm.de2a8faf91349910526b51da93d8bd42d57efdd2e93c32cd09ae64ecc1748ae9/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-04-14T14:04:25.076 INFO:journalctl@ceph.mon.c.smithi069.stdout: RuntimeError: Failed command: systemctl restart ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968@iscsi.foo.smithi069.yuumtr: Job for ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968@iscsi.foo.smithi069.yuumtr.service failed because the control process exited with error code. 2024-04-14T14:04:25.076 INFO:journalctl@ceph.mon.c.smithi069.stdout: See "systemctl status ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968@iscsi.foo.smithi069.yuumtr.service" and "journalctl -xeu ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968@iscsi.foo.smithi069.yuumtr.service" for details. 2024-04-14T14:04:25.076 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:04:24 smithi069 ceph-mon[29472]: Reconfiguring prometheus.a (dependencies changed)... 2024-04-14T14:04:25.076 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:04:24 smithi069 ceph-mon[29472]: Reconfiguring daemon prometheus.a on smithi122 2024-04-14T14:04:25.076 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:04:24 smithi069 ceph-mon[29472]: pgmap v16: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:04:25.094 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:04:24 smithi122 ceph-mon[25721]: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi069.yuumtr ... 2024-04-14T14:04:25.095 INFO:journalctl@ceph.mon.b.smithi122.stdout: Creating ceph-iscsi config... 2024-04-14T14:04:25.095 INFO:journalctl@ceph.mon.b.smithi122.stdout: Write file: /var/lib/ceph/dc326fc2-fa65-11ee-bc8e-c7b262605968/iscsi.foo.smithi069.yuumtr/iscsi-gateway.cfg 2024-04-14T14:04:25.095 INFO:journalctl@ceph.mon.b.smithi122.stdout: Write file: /var/lib/ceph/dc326fc2-fa65-11ee-bc8e-c7b262605968/iscsi.foo.smithi069.yuumtr/tcmu-runner-entrypoint.sh 2024-04-14T14:04:25.095 INFO:journalctl@ceph.mon.b.smithi122.stdout: Non-zero exit code 1 from systemctl restart ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968@iscsi.foo.smithi069.yuumtr 2024-04-14T14:04:25.095 INFO:journalctl@ceph.mon.b.smithi122.stdout: systemctl: stderr Job for ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968@iscsi.foo.smithi069.yuumtr.service failed because the control process exited with error code. 2024-04-14T14:04:25.095 INFO:journalctl@ceph.mon.b.smithi122.stdout: systemctl: stderr See "systemctl status ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968@iscsi.foo.smithi069.yuumtr.service" and "journalctl -xeu ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968@iscsi.foo.smithi069.yuumtr.service" for details. 2024-04-14T14:04:25.095 INFO:journalctl@ceph.mon.b.smithi122.stdout: Traceback (most recent call last): 2024-04-14T14:04:25.095 INFO:journalctl@ceph.mon.b.smithi122.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-04-14T14:04:25.095 INFO:journalctl@ceph.mon.b.smithi122.stdout: return _run_code(code, main_globals, None, 2024-04-14T14:04:25.095 INFO:journalctl@ceph.mon.b.smithi122.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-04-14T14:04:25.095 INFO:journalctl@ceph.mon.b.smithi122.stdout: exec(code, run_globals) 2024-04-14T14:04:25.095 INFO:journalctl@ceph.mon.b.smithi122.stdout: File "/var/lib/ceph/dc326fc2-fa65-11ee-bc8e-c7b262605968/cephadm.de2a8faf91349910526b51da93d8bd42d57efdd2e93c32cd09ae64ecc1748ae9/__main__.py", line 5608, in 2024-04-14T14:04:25.095 INFO:journalctl@ceph.mon.b.smithi122.stdout: File "/var/lib/ceph/dc326fc2-fa65-11ee-bc8e-c7b262605968/cephadm.de2a8faf91349910526b51da93d8bd42d57efdd2e93c32cd09ae64ecc1748ae9/__main__.py", line 5596, in main 2024-04-14T14:04:25.096 INFO:journalctl@ceph.mon.b.smithi122.stdout: File "/var/lib/ceph/dc326fc2-fa65-11ee-bc8e-c7b262605968/cephadm.de2a8faf91349910526b51da93d8bd42d57efdd2e93c32cd09ae64ecc1748ae9/__main__.py", line 3096, in command_deploy_from 2024-04-14T14:04:25.096 INFO:journalctl@ceph.mon.b.smithi122.stdout: File "/var/lib/ceph/dc326fc2-fa65-11ee-bc8e-c7b262605968/cephadm.de2a8faf91349910526b51da93d8bd42d57efdd2e93c32cd09ae64ecc1748ae9/__main__.py", line 3131, in _common_deploy 2024-04-14T14:04:25.096 INFO:journalctl@ceph.mon.b.smithi122.stdout: File "/var/lib/ceph/dc326fc2-fa65-11ee-bc8e-c7b262605968/cephadm.de2a8faf91349910526b51da93d8bd42d57efdd2e93c32cd09ae64ecc1748ae9/__main__.py", line 3151, in _deploy_daemon_container 2024-04-14T14:04:25.096 INFO:journalctl@ceph.mon.b.smithi122.stdout: File "/var/lib/ceph/dc326fc2-fa65-11ee-bc8e-c7b262605968/cephadm.de2a8faf91349910526b51da93d8bd42d57efdd2e93c32cd09ae64ecc1748ae9/__main__.py", line 1123, in deploy_daemon 2024-04-14T14:04:25.096 INFO:journalctl@ceph.mon.b.smithi122.stdout: File "/var/lib/ceph/dc326fc2-fa65-11ee-bc8e-c7b262605968/cephadm.de2a8faf91349910526b51da93d8bd42d57efdd2e93c32cd09ae64ecc1748ae9/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-04-14T14:04:25.096 INFO:journalctl@ceph.mon.b.smithi122.stdout: RuntimeError: Failed command: systemctl restart ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968@iscsi.foo.smithi069.yuumtr: Job for ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968@iscsi.foo.smithi069.yuumtr.service failed because the control process exited with error code. 2024-04-14T14:04:25.096 INFO:journalctl@ceph.mon.b.smithi122.stdout: See "systemctl status ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968@iscsi.foo.smithi069.yuumtr.service" and "journalctl -xeu ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968@iscsi.foo.smithi069.yuumtr.service" for details. 2024-04-14T14:04:25.096 INFO:journalctl@ceph.mon.b.smithi122.stdout: Traceback (most recent call last): 2024-04-14T14:04:25.096 INFO:journalctl@ceph.mon.b.smithi122.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1114, in _check_daemons 2024-04-14T14:04:25.096 INFO:journalctl@ceph.mon.b.smithi122.stdout: self.mgr._daemon_action(daemon_spec, action=action) 2024-04-14T14:04:25.096 INFO:journalctl@ceph.mon.b.smithi122.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 2424, in _daemon_action 2024-04-14T14:04:25.096 INFO:journalctl@ceph.mon.b.smithi122.stdout: CephadmServe(self)._create_daemon(daemon_spec, reconfig=(action == 'reconfig'))) 2024-04-14T14:04:25.096 INFO:journalctl@ceph.mon.b.smithi122.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 779, in wait_async 2024-04-14T14:04:25.096 INFO:journalctl@ceph.mon.b.smithi122.stdout: return self.event_loop.get_result(coro, timeout) 2024-04-14T14:04:25.097 INFO:journalctl@ceph.mon.b.smithi122.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 136, in get_result 2024-04-14T14:04:25.097 INFO:journalctl@ceph.mon.b.smithi122.stdout: return future.result(timeout) 2024-04-14T14:04:25.097 INFO:journalctl@ceph.mon.b.smithi122.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2024-04-14T14:04:25.097 INFO:journalctl@ceph.mon.b.smithi122.stdout: return self.__get_result() 2024-04-14T14:04:25.097 INFO:journalctl@ceph.mon.b.smithi122.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2024-04-14T14:04:25.097 INFO:journalctl@ceph.mon.b.smithi122.stdout: raise self._exception 2024-04-14T14:04:25.097 INFO:journalctl@ceph.mon.b.smithi122.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1374, in _create_daemon 2024-04-14T14:04:25.097 INFO:journalctl@ceph.mon.b.smithi122.stdout: config_blobs=daemon_spec.final_config, 2024-04-14T14:04:25.097 INFO:journalctl@ceph.mon.b.smithi122.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1685, in _run_cephadm 2024-04-14T14:04:25.097 INFO:journalctl@ceph.mon.b.smithi122.stdout: f'cephadm exited with an error code: {code}, stderr: {err}') 2024-04-14T14:04:25.097 INFO:journalctl@ceph.mon.b.smithi122.stdout: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi069.yuumtr ... 2024-04-14T14:04:25.097 INFO:journalctl@ceph.mon.b.smithi122.stdout: Creating ceph-iscsi config... 2024-04-14T14:04:25.097 INFO:journalctl@ceph.mon.b.smithi122.stdout: Write file: /var/lib/ceph/dc326fc2-fa65-11ee-bc8e-c7b262605968/iscsi.foo.smithi069.yuumtr/iscsi-gateway.cfg 2024-04-14T14:04:25.097 INFO:journalctl@ceph.mon.b.smithi122.stdout: Write file: /var/lib/ceph/dc326fc2-fa65-11ee-bc8e-c7b262605968/iscsi.foo.smithi069.yuumtr/tcmu-runner-entrypoint.sh 2024-04-14T14:04:25.097 INFO:journalctl@ceph.mon.b.smithi122.stdout: Non-zero exit code 1 from systemctl restart ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968@iscsi.foo.smithi069.yuumtr 2024-04-14T14:04:25.098 INFO:journalctl@ceph.mon.b.smithi122.stdout: systemctl: stderr Job for ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968@iscsi.foo.smithi069.yuumtr.service failed because the control process exited with error code. 2024-04-14T14:04:25.098 INFO:journalctl@ceph.mon.b.smithi122.stdout: systemctl: stderr See "systemctl status ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968@iscsi.foo.smithi069.yuumtr.service" and "journalctl -xeu ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968@iscsi.foo.smithi069.yuumtr.service" for details. 2024-04-14T14:04:25.098 INFO:journalctl@ceph.mon.b.smithi122.stdout: Traceback (most recent call last): 2024-04-14T14:04:25.098 INFO:journalctl@ceph.mon.b.smithi122.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-04-14T14:04:25.098 INFO:journalctl@ceph.mon.b.smithi122.stdout: return _run_code(code, main_globals, None, 2024-04-14T14:04:25.098 INFO:journalctl@ceph.mon.b.smithi122.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-04-14T14:04:25.098 INFO:journalctl@ceph.mon.b.smithi122.stdout: exec(code, run_globals) 2024-04-14T14:04:25.098 INFO:journalctl@ceph.mon.b.smithi122.stdout: File "/var/lib/ceph/dc326fc2-fa65-11ee-bc8e-c7b262605968/cephadm.de2a8faf91349910526b51da93d8bd42d57efdd2e93c32cd09ae64ecc1748ae9/__main__.py", line 5608, in 2024-04-14T14:04:25.098 INFO:journalctl@ceph.mon.b.smithi122.stdout: File "/var/lib/ceph/dc326fc2-fa65-11ee-bc8e-c7b262605968/cephadm.de2a8faf91349910526b51da93d8bd42d57efdd2e93c32cd09ae64ecc1748ae9/__main__.py", line 5596, in main 2024-04-14T14:04:25.098 INFO:journalctl@ceph.mon.b.smithi122.stdout: File "/var/lib/ceph/dc326fc2-fa65-11ee-bc8e-c7b262605968/cephadm.de2a8faf91349910526b51da93d8bd42d57efdd2e93c32cd09ae64ecc1748ae9/__main__.py", line 3096, in command_deploy_from 2024-04-14T14:04:25.098 INFO:journalctl@ceph.mon.b.smithi122.stdout: File "/var/lib/ceph/dc326fc2-fa65-11ee-bc8e-c7b262605968/cephadm.de2a8faf91349910526b51da93d8bd42d57efdd2e93c32cd09ae64ecc1748ae9/__main__.py", line 3131, in _common_deploy 2024-04-14T14:04:25.098 INFO:journalctl@ceph.mon.b.smithi122.stdout: File "/var/lib/ceph/dc326fc2-fa65-11ee-bc8e-c7b262605968/cephadm.de2a8faf91349910526b51da93d8bd42d57efdd2e93c32cd09ae64ecc1748ae9/__main__.py", line 3151, in _deploy_daemon_container 2024-04-14T14:04:25.098 INFO:journalctl@ceph.mon.b.smithi122.stdout: File "/var/lib/ceph/dc326fc2-fa65-11ee-bc8e-c7b262605968/cephadm.de2a8faf91349910526b51da93d8bd42d57efdd2e93c32cd09ae64ecc1748ae9/__main__.py", line 1123, in deploy_daemon 2024-04-14T14:04:25.098 INFO:journalctl@ceph.mon.b.smithi122.stdout: File "/var/lib/ceph/dc326fc2-fa65-11ee-bc8e-c7b262605968/cephadm.de2a8faf91349910526b51da93d8bd42d57efdd2e93c32cd09ae64ecc1748ae9/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-04-14T14:04:25.099 INFO:journalctl@ceph.mon.b.smithi122.stdout: RuntimeError: Failed command: systemctl restart ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968@iscsi.foo.smithi069.yuumtr: Job for ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968@iscsi.foo.smithi069.yuumtr.service failed because the control process exited with error code. 2024-04-14T14:04:25.099 INFO:journalctl@ceph.mon.b.smithi122.stdout: See "systemctl status ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968@iscsi.foo.smithi069.yuumtr.service" and "journalctl -xeu ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968@iscsi.foo.smithi069.yuumtr.service" for details. 2024-04-14T14:04:25.099 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:04:24 smithi122 ceph-mon[25721]: Reconfiguring prometheus.a (dependencies changed)... 2024-04-14T14:04:25.099 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:04:24 smithi122 ceph-mon[25721]: Reconfiguring daemon prometheus.a on smithi122 2024-04-14T14:04:25.099 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:04:24 smithi122 ceph-mon[25721]: pgmap v16: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:04:26.816 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:04:26 smithi069 ceph-mon[24818]: pgmap v17: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:04:26.817 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:04:26 smithi069 ceph-mon[29472]: pgmap v17: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:04:26.844 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:04:26 smithi122 ceph-mon[25721]: pgmap v17: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:04:27.594 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Apr 14 14:04:27 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-x[45487]: ::ffff:172.21.15.122 - - [14/Apr/2024:14:04:27] "GET /metrics HTTP/1.1" 200 37890 "" "Prometheus/2.33.4" 2024-04-14T14:04:28.844 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:04:28 smithi122 ceph-mon[25721]: pgmap v18: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-04-14T14:04:28.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:04:28 smithi122 ceph-mon[25721]: from='mgr.24749 172.21.15.122:0/1892077634' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T14:04:29.066 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:04:28 smithi069 ceph-mon[24818]: pgmap v18: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-04-14T14:04:29.067 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:04:28 smithi069 ceph-mon[24818]: from='mgr.24749 172.21.15.122:0/1892077634' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T14:04:29.067 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:04:28 smithi069 ceph-mon[29472]: pgmap v18: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-04-14T14:04:29.067 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:04:28 smithi069 ceph-mon[29472]: from='mgr.24749 172.21.15.122:0/1892077634' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T14:04:29.817 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 14 14:04:29 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-y[25054]: ::ffff:172.21.15.122 - - [14/Apr/2024:14:04:29] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-04-14T14:04:29.817 INFO:journalctl@ceph.alertmanager.a.smithi069.stdout:Apr 14 14:04:29 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-alertmanager-a[58923]: level=info ts=2024-04-14T14:04:29.651Z caller=cluster.go:688 component=cluster msg="gossip settled; proceeding" elapsed=10.00337988s 2024-04-14T14:04:30.816 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:04:30 smithi069 ceph-mon[24818]: pgmap v19: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-04-14T14:04:30.817 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:04:30 smithi069 ceph-mon[29472]: pgmap v19: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-04-14T14:04:30.846 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:04:30 smithi122 ceph-mon[25721]: pgmap v19: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-04-14T14:04:32.439 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Apr 14 14:04:32 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-prometheus-a[42660]: ts=2024-04-14T14:04:32.315Z caller=manager.go:609 level=warn component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.69:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.122:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-04-14T14:04:32.844 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:04:32 smithi122 ceph-mon[25721]: pgmap v20: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2024-04-14T14:04:33.067 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:04:32 smithi069 ceph-mon[24818]: pgmap v20: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2024-04-14T14:04:33.067 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:04:32 smithi069 ceph-mon[29472]: pgmap v20: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2024-04-14T14:04:33.496 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Apr 14 14:04:33 smithi122 systemd[1]: Stopping Ceph prometheus.a for dc326fc2-fa65-11ee-bc8e-c7b262605968... 2024-04-14T14:04:33.496 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Apr 14 14:04:33 smithi122 bash[49150]: Error: no container with name or ID "ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-prometheus.a" found: no such container 2024-04-14T14:04:33.781 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Apr 14 14:04:33 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-prometheus-a[42660]: ts=2024-04-14T14:04:33.495Z caller=main.go:775 level=warn msg="Received SIGTERM, exiting gracefully..." 2024-04-14T14:04:33.782 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Apr 14 14:04:33 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-prometheus-a[42660]: ts=2024-04-14T14:04:33.495Z caller=main.go:798 level=info msg="Stopping scrape discovery manager..." 2024-04-14T14:04:33.782 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Apr 14 14:04:33 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-prometheus-a[42660]: ts=2024-04-14T14:04:33.495Z caller=main.go:812 level=info msg="Stopping notify discovery manager..." 2024-04-14T14:04:33.782 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Apr 14 14:04:33 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-prometheus-a[42660]: ts=2024-04-14T14:04:33.495Z caller=main.go:834 level=info msg="Stopping scrape manager..." 2024-04-14T14:04:33.782 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Apr 14 14:04:33 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-prometheus-a[42660]: ts=2024-04-14T14:04:33.495Z caller=main.go:794 level=info msg="Scrape discovery manager stopped" 2024-04-14T14:04:33.782 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Apr 14 14:04:33 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-prometheus-a[42660]: ts=2024-04-14T14:04:33.495Z caller=main.go:808 level=info msg="Notify discovery manager stopped" 2024-04-14T14:04:33.782 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Apr 14 14:04:33 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-prometheus-a[42660]: ts=2024-04-14T14:04:33.495Z caller=manager.go:945 level=info component="rule manager" msg="Stopping rule manager..." 2024-04-14T14:04:33.782 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Apr 14 14:04:33 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-prometheus-a[42660]: ts=2024-04-14T14:04:33.495Z caller=main.go:828 level=info msg="Scrape manager stopped" 2024-04-14T14:04:33.782 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Apr 14 14:04:33 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-prometheus-a[42660]: ts=2024-04-14T14:04:33.495Z caller=manager.go:955 level=info component="rule manager" msg="Rule manager stopped" 2024-04-14T14:04:33.782 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Apr 14 14:04:33 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-prometheus-a[42660]: ts=2024-04-14T14:04:33.496Z caller=notifier.go:600 level=info component=notifier msg="Stopping notification manager..." 2024-04-14T14:04:33.782 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Apr 14 14:04:33 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-prometheus-a[42660]: ts=2024-04-14T14:04:33.496Z caller=main.go:1054 level=info msg="Notifier manager stopped" 2024-04-14T14:04:33.782 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Apr 14 14:04:33 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-prometheus-a[42660]: ts=2024-04-14T14:04:33.496Z caller=main.go:1066 level=info msg="See you next time!" 2024-04-14T14:04:33.782 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Apr 14 14:04:33 smithi122 podman[49164]: 2024-04-14 14:04:33.505672708 +0000 UTC m=+0.111806717 container died 59e61af03fdf3105d92fb2c73df6a42a6ef10a5eddbf11d7ef945aac315c06eb (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-prometheus-a, maintainer=The Prometheus Authors ) 2024-04-14T14:04:34.094 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Apr 14 14:04:33 smithi122 podman[49164]: 2024-04-14 14:04:33.878821532 +0000 UTC m=+0.484955548 container cleanup 59e61af03fdf3105d92fb2c73df6a42a6ef10a5eddbf11d7ef945aac315c06eb (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-prometheus-a, maintainer=The Prometheus Authors ) 2024-04-14T14:04:34.095 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Apr 14 14:04:33 smithi122 bash[49164]: ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-prometheus-a 2024-04-14T14:04:34.095 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Apr 14 14:04:33 smithi122 bash[49191]: Error: no container with name or ID "ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-prometheus.a" found: no such container 2024-04-14T14:04:34.595 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Apr 14 14:04:34 smithi122 podman[49178]: 2024-04-14 14:04:34.129874001 +0000 UTC m=+0.626204002 container remove 59e61af03fdf3105d92fb2c73df6a42a6ef10a5eddbf11d7ef945aac315c06eb (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-prometheus-a, maintainer=The Prometheus Authors ) 2024-04-14T14:04:34.595 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Apr 14 14:04:34 smithi122 systemd[1]: ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968@prometheus.a.service: Deactivated successfully. 2024-04-14T14:04:34.595 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Apr 14 14:04:34 smithi122 systemd[1]: Stopped Ceph prometheus.a for dc326fc2-fa65-11ee-bc8e-c7b262605968. 2024-04-14T14:04:34.595 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Apr 14 14:04:34 smithi122 systemd[1]: ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968@prometheus.a.service: Consumed 1.996s CPU time. 2024-04-14T14:04:34.595 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Apr 14 14:04:34 smithi122 systemd[1]: Starting Ceph prometheus.a for dc326fc2-fa65-11ee-bc8e-c7b262605968... 2024-04-14T14:04:34.595 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Apr 14 14:04:34 smithi122 podman[49263]: 2024-04-14 14:04:34.273676042 +0000 UTC m=+0.019339296 image pull 514e6a882f6e74806a5856468489eeff8d7106095557578da96935e4d0ba4d9d quay.io/prometheus/prometheus:v2.33.4 2024-04-14T14:04:35.066 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:04:34 smithi069 ceph-mon[24818]: pgmap v21: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2024-04-14T14:04:35.067 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:04:34 smithi069 ceph-mon[24818]: from='client.? 172.21.15.69:0/2416216888' entity='client.iscsi.foo.smithi069.yuumtr' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-04-14T14:04:35.067 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:04:34 smithi069 ceph-mon[29472]: pgmap v21: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2024-04-14T14:04:35.067 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:04:34 smithi069 ceph-mon[29472]: from='client.? 172.21.15.69:0/2416216888' entity='client.iscsi.foo.smithi069.yuumtr' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-04-14T14:04:35.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:04:34 smithi122 ceph-mon[25721]: pgmap v21: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2024-04-14T14:04:35.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:04:34 smithi122 ceph-mon[25721]: from='client.? 172.21.15.69:0/2416216888' entity='client.iscsi.foo.smithi069.yuumtr' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-04-14T14:04:35.095 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Apr 14 14:04:34 smithi122 podman[49263]: 2024-04-14 14:04:34.656165295 +0000 UTC m=+0.401828535 container create 2bdf6dae6e1e624add99fc15caba8a64ab8495e8227a255fdce337ca01270932 (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-prometheus-a, maintainer=The Prometheus Authors ) 2024-04-14T14:04:35.996 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:04:35 smithi122 ceph-mon[25721]: from='client.? 172.21.15.69:0/205063227' entity='client.iscsi.foo.smithi069.yuumtr' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.69:0/1033838063"}]: dispatch 2024-04-14T14:04:35.996 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:04:35 smithi122 ceph-mon[25721]: from='client.? ' entity='client.iscsi.foo.smithi069.yuumtr' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.69:0/1033838063"}]: dispatch 2024-04-14T14:04:36.066 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:04:35 smithi069 ceph-mon[24818]: from='client.? 172.21.15.69:0/205063227' entity='client.iscsi.foo.smithi069.yuumtr' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.69:0/1033838063"}]: dispatch 2024-04-14T14:04:36.067 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:04:35 smithi069 ceph-mon[24818]: from='client.? ' entity='client.iscsi.foo.smithi069.yuumtr' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.69:0/1033838063"}]: dispatch 2024-04-14T14:04:36.067 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:04:35 smithi069 ceph-mon[29472]: from='client.? 172.21.15.69:0/205063227' entity='client.iscsi.foo.smithi069.yuumtr' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.69:0/1033838063"}]: dispatch 2024-04-14T14:04:36.067 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:04:35 smithi069 ceph-mon[29472]: from='client.? ' entity='client.iscsi.foo.smithi069.yuumtr' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.69:0/1033838063"}]: dispatch 2024-04-14T14:04:36.272 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Apr 14 14:04:36 smithi122 podman[49263]: 2024-04-14 14:04:36.15162845 +0000 UTC m=+1.897291690 container init 2bdf6dae6e1e624add99fc15caba8a64ab8495e8227a255fdce337ca01270932 (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-prometheus-a, maintainer=The Prometheus Authors ) 2024-04-14T14:04:36.273 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Apr 14 14:04:36 smithi122 podman[49263]: 2024-04-14 14:04:36.155416357 +0000 UTC m=+1.901079601 container start 2bdf6dae6e1e624add99fc15caba8a64ab8495e8227a255fdce337ca01270932 (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-prometheus-a, maintainer=The Prometheus Authors ) 2024-04-14T14:04:36.273 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Apr 14 14:04:36 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-prometheus-a[49277]: ts=2024-04-14T14:04:36.182Z caller=main.go:475 level=info msg="No time or size retention was set so using the default time retention" duration=15d 2024-04-14T14:04:36.273 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Apr 14 14:04:36 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-prometheus-a[49277]: ts=2024-04-14T14:04:36.182Z caller=main.go:512 level=info msg="Starting Prometheus" version="(version=2.33.4, branch=HEAD, revision=83032011a5d3e6102624fe58241a374a7201fee8)" 2024-04-14T14:04:36.273 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Apr 14 14:04:36 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-prometheus-a[49277]: ts=2024-04-14T14:04:36.182Z caller=main.go:517 level=info build_context="(go=go1.17.7, user=root@d13bf69e7be8, date=20220222-16:51:28)" 2024-04-14T14:04:36.273 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Apr 14 14:04:36 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-prometheus-a[49277]: ts=2024-04-14T14:04:36.182Z caller=main.go:518 level=info host_details="(Linux 5.14.0-435.el9.x86_64 #1 SMP PREEMPT_DYNAMIC Fri Mar 29 18:46:37 UTC 2024 x86_64 smithi122 (none))" 2024-04-14T14:04:36.273 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Apr 14 14:04:36 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-prometheus-a[49277]: ts=2024-04-14T14:04:36.182Z caller=main.go:519 level=info fd_limits="(soft=1048576, hard=1048576)" 2024-04-14T14:04:36.273 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Apr 14 14:04:36 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-prometheus-a[49277]: ts=2024-04-14T14:04:36.182Z caller=main.go:520 level=info vm_limits="(soft=unlimited, hard=unlimited)" 2024-04-14T14:04:36.273 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Apr 14 14:04:36 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-prometheus-a[49277]: ts=2024-04-14T14:04:36.184Z caller=web.go:570 level=info component=web msg="Start listening for connections" address=:9095 2024-04-14T14:04:36.273 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Apr 14 14:04:36 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-prometheus-a[49277]: ts=2024-04-14T14:04:36.184Z caller=main.go:923 level=info msg="Starting TSDB ..." 2024-04-14T14:04:36.273 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Apr 14 14:04:36 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-prometheus-a[49277]: ts=2024-04-14T14:04:36.185Z caller=tls_config.go:195 level=info component=web msg="TLS is disabled." http2=false 2024-04-14T14:04:36.273 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Apr 14 14:04:36 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-prometheus-a[49277]: ts=2024-04-14T14:04:36.188Z caller=head.go:493 level=info component=tsdb msg="Replaying on-disk memory mappable chunks if any" 2024-04-14T14:04:36.273 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Apr 14 14:04:36 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-prometheus-a[49277]: ts=2024-04-14T14:04:36.189Z caller=head.go:527 level=info component=tsdb msg="On-disk memory mappable chunks replay completed" duration=1.289518ms 2024-04-14T14:04:36.273 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Apr 14 14:04:36 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-prometheus-a[49277]: ts=2024-04-14T14:04:36.189Z caller=head.go:533 level=info component=tsdb msg="Replaying WAL, this may take a while" 2024-04-14T14:04:36.273 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Apr 14 14:04:36 smithi122 bash[49263]: 2bdf6dae6e1e624add99fc15caba8a64ab8495e8227a255fdce337ca01270932 2024-04-14T14:04:36.274 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Apr 14 14:04:36 smithi122 systemd[1]: Started Ceph prometheus.a for dc326fc2-fa65-11ee-bc8e-c7b262605968. 2024-04-14T14:04:36.594 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Apr 14 14:04:36 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-prometheus-a[49277]: ts=2024-04-14T14:04:36.402Z caller=head.go:604 level=info component=tsdb msg="WAL segment loaded" segment=0 maxSegment=2 2024-04-14T14:04:37.067 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:04:36 smithi069 ceph-mon[29472]: pgmap v22: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2024-04-14T14:04:37.067 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:04:36 smithi069 ceph-mon[29472]: from='client.? ' entity='client.iscsi.foo.smithi069.yuumtr' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.69:0/1033838063"}]': finished 2024-04-14T14:04:37.067 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:04:36 smithi069 ceph-mon[29472]: osdmap e95: 8 total, 8 up, 8 in 2024-04-14T14:04:37.067 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:04:36 smithi069 ceph-mon[29472]: from='client.? 172.21.15.69:0/394999926' entity='client.iscsi.foo.smithi069.yuumtr' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.69:0/1711970283"}]: dispatch 2024-04-14T14:04:37.067 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:04:36 smithi069 ceph-mon[29472]: from='mgr.24749 ' entity='mgr.x' 2024-04-14T14:04:37.067 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:04:36 smithi069 ceph-mon[29472]: from='mgr.24749 ' entity='mgr.x' 2024-04-14T14:04:37.067 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:04:36 smithi069 ceph-mon[24818]: pgmap v22: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2024-04-14T14:04:37.067 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:04:36 smithi069 ceph-mon[24818]: from='client.? ' entity='client.iscsi.foo.smithi069.yuumtr' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.69:0/1033838063"}]': finished 2024-04-14T14:04:37.067 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:04:36 smithi069 ceph-mon[24818]: osdmap e95: 8 total, 8 up, 8 in 2024-04-14T14:04:37.067 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:04:36 smithi069 ceph-mon[24818]: from='client.? 172.21.15.69:0/394999926' entity='client.iscsi.foo.smithi069.yuumtr' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.69:0/1711970283"}]: dispatch 2024-04-14T14:04:37.068 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:04:36 smithi069 ceph-mon[24818]: from='mgr.24749 ' entity='mgr.x' 2024-04-14T14:04:37.068 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:04:36 smithi069 ceph-mon[24818]: from='mgr.24749 ' entity='mgr.x' 2024-04-14T14:04:37.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:04:36 smithi122 ceph-mon[25721]: pgmap v22: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2024-04-14T14:04:37.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:04:36 smithi122 ceph-mon[25721]: from='client.? ' entity='client.iscsi.foo.smithi069.yuumtr' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.69:0/1033838063"}]': finished 2024-04-14T14:04:37.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:04:36 smithi122 ceph-mon[25721]: osdmap e95: 8 total, 8 up, 8 in 2024-04-14T14:04:37.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:04:36 smithi122 ceph-mon[25721]: from='client.? 172.21.15.69:0/394999926' entity='client.iscsi.foo.smithi069.yuumtr' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.69:0/1711970283"}]: dispatch 2024-04-14T14:04:37.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:04:36 smithi122 ceph-mon[25721]: from='mgr.24749 ' entity='mgr.x' 2024-04-14T14:04:37.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:04:36 smithi122 ceph-mon[25721]: from='mgr.24749 ' entity='mgr.x' 2024-04-14T14:04:37.095 INFO:journalctl@ceph.node-exporter.b.smithi122.stdout:Apr 14 14:04:36 smithi122 systemd[1]: Stopping Ceph node-exporter.b for dc326fc2-fa65-11ee-bc8e-c7b262605968... 2024-04-14T14:04:37.096 INFO:journalctl@ceph.node-exporter.b.smithi122.stdout:Apr 14 14:04:36 smithi122 bash[49398]: Error: no container with name or ID "ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-node-exporter.b" found: no such container 2024-04-14T14:04:37.096 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Apr 14 14:04:36 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-prometheus-a[49277]: ts=2024-04-14T14:04:36.645Z caller=head.go:604 level=info component=tsdb msg="WAL segment loaded" segment=1 maxSegment=2 2024-04-14T14:04:37.096 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Apr 14 14:04:36 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-prometheus-a[49277]: ts=2024-04-14T14:04:36.646Z caller=head.go:604 level=info component=tsdb msg="WAL segment loaded" segment=2 maxSegment=2 2024-04-14T14:04:37.096 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Apr 14 14:04:36 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-prometheus-a[49277]: ts=2024-04-14T14:04:36.646Z caller=head.go:610 level=info component=tsdb msg="WAL replay completed" checkpoint_replay_duration=35.144µs wal_replay_duration=456.503271ms total_replay_duration=457.847466ms 2024-04-14T14:04:37.096 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Apr 14 14:04:36 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-prometheus-a[49277]: ts=2024-04-14T14:04:36.649Z caller=main.go:944 level=info fs_type=XFS_SUPER_MAGIC 2024-04-14T14:04:37.096 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Apr 14 14:04:36 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-prometheus-a[49277]: ts=2024-04-14T14:04:36.649Z caller=main.go:947 level=info msg="TSDB started" 2024-04-14T14:04:37.096 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Apr 14 14:04:36 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-prometheus-a[49277]: ts=2024-04-14T14:04:36.649Z caller=main.go:1128 level=info msg="Loading configuration file" filename=/etc/prometheus/prometheus.yml 2024-04-14T14:04:37.096 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Apr 14 14:04:36 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-prometheus-a[49277]: ts=2024-04-14T14:04:36.678Z caller=main.go:1165 level=info msg="Completed loading of configuration file" filename=/etc/prometheus/prometheus.yml totalDuration=28.887268ms db_storage=734ns remote_storage=1.296µs web_handler=377ns query_engine=693ns scrape=372.136µs scrape_sd=102.461µs notify=17.934µs notify_sd=17.311µs rules=28.054224ms 2024-04-14T14:04:37.096 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Apr 14 14:04:36 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-prometheus-a[49277]: ts=2024-04-14T14:04:36.678Z caller=main.go:896 level=info msg="Server is ready to receive web requests." 2024-04-14T14:04:37.443 INFO:journalctl@ceph.node-exporter.b.smithi122.stdout:Apr 14 14:04:37 smithi122 podman[49411]: 2024-04-14 14:04:37.166395255 +0000 UTC m=+0.328656491 container died e57f3d5931fbce91c3bf4d2422435c6b3ea9b89efe9f0b019b7925d0cac49251 (image=quay.io/prometheus/node-exporter:v1.3.1, name=ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-node-exporter-b, maintainer=The Prometheus Authors ) 2024-04-14T14:04:37.783 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:04:37 smithi122 ceph-mon[25721]: Reconfiguring node-exporter.b (dependencies changed)... 2024-04-14T14:04:37.783 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:04:37 smithi122 ceph-mon[25721]: Reconfiguring daemon node-exporter.b on smithi122 2024-04-14T14:04:37.783 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:04:37 smithi122 ceph-mon[25721]: from='client.? 172.21.15.69:0/394999926' entity='client.iscsi.foo.smithi069.yuumtr' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.69:0/1711970283"}]': finished 2024-04-14T14:04:37.783 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:04:37 smithi122 ceph-mon[25721]: osdmap e96: 8 total, 8 up, 8 in 2024-04-14T14:04:37.783 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:04:37 smithi122 ceph-mon[25721]: from='client.? 172.21.15.69:0/1157801709' entity='client.iscsi.foo.smithi069.yuumtr' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.69:0/3960312281"}]: dispatch 2024-04-14T14:04:37.783 INFO:journalctl@ceph.node-exporter.b.smithi122.stdout:Apr 14 14:04:37 smithi122 podman[49411]: 2024-04-14 14:04:37.555498482 +0000 UTC m=+0.717759725 container cleanup e57f3d5931fbce91c3bf4d2422435c6b3ea9b89efe9f0b019b7925d0cac49251 (image=quay.io/prometheus/node-exporter:v1.3.1, name=ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-node-exporter-b, maintainer=The Prometheus Authors ) 2024-04-14T14:04:37.783 INFO:journalctl@ceph.node-exporter.b.smithi122.stdout:Apr 14 14:04:37 smithi122 bash[49411]: ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-node-exporter-b 2024-04-14T14:04:37.783 INFO:journalctl@ceph.node-exporter.b.smithi122.stdout:Apr 14 14:04:37 smithi122 bash[49439]: Error: no container with name or ID "ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-node-exporter.b" found: no such container 2024-04-14T14:04:38.059 INFO:journalctl@ceph.node-exporter.b.smithi122.stdout:Apr 14 14:04:37 smithi122 podman[49426]: 2024-04-14 14:04:37.782126678 +0000 UTC m=+0.619916576 container remove e57f3d5931fbce91c3bf4d2422435c6b3ea9b89efe9f0b019b7925d0cac49251 (image=quay.io/prometheus/node-exporter:v1.3.1, name=ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-node-exporter-b, maintainer=The Prometheus Authors ) 2024-04-14T14:04:38.059 INFO:journalctl@ceph.node-exporter.b.smithi122.stdout:Apr 14 14:04:37 smithi122 systemd[1]: ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968@node-exporter.b.service: Main process exited, code=exited, status=143/n/a 2024-04-14T14:04:38.059 INFO:journalctl@ceph.node-exporter.b.smithi122.stdout:Apr 14 14:04:37 smithi122 systemd[1]: ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968@node-exporter.b.service: Failed with result 'exit-code'. 2024-04-14T14:04:38.059 INFO:journalctl@ceph.node-exporter.b.smithi122.stdout:Apr 14 14:04:37 smithi122 systemd[1]: Stopped Ceph node-exporter.b for dc326fc2-fa65-11ee-bc8e-c7b262605968. 2024-04-14T14:04:38.059 INFO:journalctl@ceph.node-exporter.b.smithi122.stdout:Apr 14 14:04:37 smithi122 systemd[1]: ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968@node-exporter.b.service: Consumed 3.895s CPU time. 2024-04-14T14:04:38.059 INFO:journalctl@ceph.node-exporter.b.smithi122.stdout:Apr 14 14:04:37 smithi122 systemd[1]: Starting Ceph node-exporter.b for dc326fc2-fa65-11ee-bc8e-c7b262605968... 2024-04-14T14:04:38.059 INFO:journalctl@ceph.node-exporter.b.smithi122.stdout:Apr 14 14:04:38 smithi122 podman[49504]: 2024-04-14 14:04:37.906261073 +0000 UTC m=+0.017276174 image pull 1dbe0e931976487e20e5cfb272087e08a9779c88fd5e9617ed7042dd9751ec26 quay.io/prometheus/node-exporter:v1.3.1 2024-04-14T14:04:38.059 INFO:journalctl@ceph.node-exporter.b.smithi122.stdout:Apr 14 14:04:38 smithi122 podman[49504]: 2024-04-14 14:04:38.058208863 +0000 UTC m=+0.169223949 container create 996115353a0b0ad63833e7ac18a29d257d8b9f2ae3c8ab1a0efe8d419490c6ad (image=quay.io/prometheus/node-exporter:v1.3.1, name=ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-node-exporter-b, maintainer=The Prometheus Authors ) 2024-04-14T14:04:38.066 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:04:37 smithi069 ceph-mon[24818]: Reconfiguring node-exporter.b (dependencies changed)... 2024-04-14T14:04:38.066 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:04:37 smithi069 ceph-mon[24818]: Reconfiguring daemon node-exporter.b on smithi122 2024-04-14T14:04:38.066 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:04:37 smithi069 ceph-mon[24818]: from='client.? 172.21.15.69:0/394999926' entity='client.iscsi.foo.smithi069.yuumtr' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.69:0/1711970283"}]': finished 2024-04-14T14:04:38.067 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:04:37 smithi069 ceph-mon[24818]: osdmap e96: 8 total, 8 up, 8 in 2024-04-14T14:04:38.067 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:04:37 smithi069 ceph-mon[24818]: from='client.? 172.21.15.69:0/1157801709' entity='client.iscsi.foo.smithi069.yuumtr' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.69:0/3960312281"}]: dispatch 2024-04-14T14:04:38.067 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:04:37 smithi069 ceph-mon[29472]: Reconfiguring node-exporter.b (dependencies changed)... 2024-04-14T14:04:38.067 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:04:37 smithi069 ceph-mon[29472]: Reconfiguring daemon node-exporter.b on smithi122 2024-04-14T14:04:38.067 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:04:37 smithi069 ceph-mon[29472]: from='client.? 172.21.15.69:0/394999926' entity='client.iscsi.foo.smithi069.yuumtr' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.69:0/1711970283"}]': finished 2024-04-14T14:04:38.067 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:04:37 smithi069 ceph-mon[29472]: osdmap e96: 8 total, 8 up, 8 in 2024-04-14T14:04:38.067 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:04:37 smithi069 ceph-mon[29472]: from='client.? 172.21.15.69:0/1157801709' entity='client.iscsi.foo.smithi069.yuumtr' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.69:0/3960312281"}]: dispatch 2024-04-14T14:04:38.345 INFO:journalctl@ceph.node-exporter.b.smithi122.stdout:Apr 14 14:04:38 smithi122 podman[49504]: 2024-04-14 14:04:38.285871268 +0000 UTC m=+0.396886353 container init 996115353a0b0ad63833e7ac18a29d257d8b9f2ae3c8ab1a0efe8d419490c6ad (image=quay.io/prometheus/node-exporter:v1.3.1, name=ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-node-exporter-b, maintainer=The Prometheus Authors ) 2024-04-14T14:04:38.346 INFO:journalctl@ceph.node-exporter.b.smithi122.stdout:Apr 14 14:04:38 smithi122 podman[49504]: 2024-04-14 14:04:38.288598773 +0000 UTC m=+0.399613859 container start 996115353a0b0ad63833e7ac18a29d257d8b9f2ae3c8ab1a0efe8d419490c6ad (image=quay.io/prometheus/node-exporter:v1.3.1, name=ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-node-exporter-b, maintainer=The Prometheus Authors ) 2024-04-14T14:04:38.346 INFO:journalctl@ceph.node-exporter.b.smithi122.stdout:Apr 14 14:04:38 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-node-exporter-b[49519]: ts=2024-04-14T14:04:38.292Z caller=node_exporter.go:182 level=info msg="Starting node_exporter" version="(version=1.3.1, branch=HEAD, revision=a2321e7b940ddcff26873612bccdf7cd4c42b6b6)" 2024-04-14T14:04:38.346 INFO:journalctl@ceph.node-exporter.b.smithi122.stdout:Apr 14 14:04:38 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-node-exporter-b[49519]: ts=2024-04-14T14:04:38.292Z caller=node_exporter.go:183 level=info msg="Build context" build_context="(go=go1.17.3, user=root@243aafa5525c, date=20211205-11:09:49)" 2024-04-14T14:04:38.346 INFO:journalctl@ceph.node-exporter.b.smithi122.stdout:Apr 14 14:04:38 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-node-exporter-b[49519]: ts=2024-04-14T14:04:38.292Z caller=filesystem_common.go:111 level=info collector=filesystem msg="Parsed flag --collector.filesystem.mount-points-exclude" flag=^/(dev|proc|run/credentials/.+|sys|var/lib/docker/.+)($|/) 2024-04-14T14:04:38.346 INFO:journalctl@ceph.node-exporter.b.smithi122.stdout:Apr 14 14:04:38 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-node-exporter-b[49519]: ts=2024-04-14T14:04:38.292Z caller=filesystem_common.go:113 level=info collector=filesystem msg="Parsed flag --collector.filesystem.fs-types-exclude" flag=^(autofs|binfmt_misc|bpf|cgroup2?|configfs|debugfs|devpts|devtmpfs|fusectl|hugetlbfs|iso9660|mqueue|nsfs|overlay|proc|procfs|pstore|rpc_pipefs|securityfs|selinuxfs|squashfs|sysfs|tracefs)$ 2024-04-14T14:04:38.346 INFO:journalctl@ceph.node-exporter.b.smithi122.stdout:Apr 14 14:04:38 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-node-exporter-b[49519]: ts=2024-04-14T14:04:38.293Z caller=node_exporter.go:108 level=info msg="Enabled collectors" 2024-04-14T14:04:38.346 INFO:journalctl@ceph.node-exporter.b.smithi122.stdout:Apr 14 14:04:38 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-node-exporter-b[49519]: ts=2024-04-14T14:04:38.293Z caller=node_exporter.go:115 level=info collector=arp 2024-04-14T14:04:38.346 INFO:journalctl@ceph.node-exporter.b.smithi122.stdout:Apr 14 14:04:38 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-node-exporter-b[49519]: ts=2024-04-14T14:04:38.293Z caller=node_exporter.go:115 level=info collector=bcache 2024-04-14T14:04:38.346 INFO:journalctl@ceph.node-exporter.b.smithi122.stdout:Apr 14 14:04:38 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-node-exporter-b[49519]: ts=2024-04-14T14:04:38.293Z caller=node_exporter.go:115 level=info collector=bonding 2024-04-14T14:04:38.346 INFO:journalctl@ceph.node-exporter.b.smithi122.stdout:Apr 14 14:04:38 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-node-exporter-b[49519]: ts=2024-04-14T14:04:38.293Z caller=node_exporter.go:115 level=info collector=btrfs 2024-04-14T14:04:38.347 INFO:journalctl@ceph.node-exporter.b.smithi122.stdout:Apr 14 14:04:38 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-node-exporter-b[49519]: ts=2024-04-14T14:04:38.293Z caller=node_exporter.go:115 level=info collector=conntrack 2024-04-14T14:04:38.347 INFO:journalctl@ceph.node-exporter.b.smithi122.stdout:Apr 14 14:04:38 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-node-exporter-b[49519]: ts=2024-04-14T14:04:38.293Z caller=node_exporter.go:115 level=info collector=cpu 2024-04-14T14:04:38.347 INFO:journalctl@ceph.node-exporter.b.smithi122.stdout:Apr 14 14:04:38 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-node-exporter-b[49519]: ts=2024-04-14T14:04:38.293Z caller=node_exporter.go:115 level=info collector=cpufreq 2024-04-14T14:04:38.347 INFO:journalctl@ceph.node-exporter.b.smithi122.stdout:Apr 14 14:04:38 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-node-exporter-b[49519]: ts=2024-04-14T14:04:38.293Z caller=node_exporter.go:115 level=info collector=diskstats 2024-04-14T14:04:38.347 INFO:journalctl@ceph.node-exporter.b.smithi122.stdout:Apr 14 14:04:38 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-node-exporter-b[49519]: ts=2024-04-14T14:04:38.293Z caller=node_exporter.go:115 level=info collector=dmi 2024-04-14T14:04:38.347 INFO:journalctl@ceph.node-exporter.b.smithi122.stdout:Apr 14 14:04:38 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-node-exporter-b[49519]: ts=2024-04-14T14:04:38.293Z caller=node_exporter.go:115 level=info collector=edac 2024-04-14T14:04:38.347 INFO:journalctl@ceph.node-exporter.b.smithi122.stdout:Apr 14 14:04:38 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-node-exporter-b[49519]: ts=2024-04-14T14:04:38.293Z caller=node_exporter.go:115 level=info collector=entropy 2024-04-14T14:04:38.347 INFO:journalctl@ceph.node-exporter.b.smithi122.stdout:Apr 14 14:04:38 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-node-exporter-b[49519]: ts=2024-04-14T14:04:38.293Z caller=node_exporter.go:115 level=info collector=fibrechannel 2024-04-14T14:04:38.347 INFO:journalctl@ceph.node-exporter.b.smithi122.stdout:Apr 14 14:04:38 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-node-exporter-b[49519]: ts=2024-04-14T14:04:38.293Z caller=node_exporter.go:115 level=info collector=filefd 2024-04-14T14:04:38.347 INFO:journalctl@ceph.node-exporter.b.smithi122.stdout:Apr 14 14:04:38 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-node-exporter-b[49519]: ts=2024-04-14T14:04:38.293Z caller=node_exporter.go:115 level=info collector=filesystem 2024-04-14T14:04:38.347 INFO:journalctl@ceph.node-exporter.b.smithi122.stdout:Apr 14 14:04:38 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-node-exporter-b[49519]: ts=2024-04-14T14:04:38.293Z caller=node_exporter.go:115 level=info collector=hwmon 2024-04-14T14:04:38.347 INFO:journalctl@ceph.node-exporter.b.smithi122.stdout:Apr 14 14:04:38 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-node-exporter-b[49519]: ts=2024-04-14T14:04:38.293Z caller=node_exporter.go:115 level=info collector=infiniband 2024-04-14T14:04:38.348 INFO:journalctl@ceph.node-exporter.b.smithi122.stdout:Apr 14 14:04:38 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-node-exporter-b[49519]: ts=2024-04-14T14:04:38.293Z caller=node_exporter.go:115 level=info collector=ipvs 2024-04-14T14:04:38.348 INFO:journalctl@ceph.node-exporter.b.smithi122.stdout:Apr 14 14:04:38 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-node-exporter-b[49519]: ts=2024-04-14T14:04:38.293Z caller=node_exporter.go:115 level=info collector=loadavg 2024-04-14T14:04:38.348 INFO:journalctl@ceph.node-exporter.b.smithi122.stdout:Apr 14 14:04:38 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-node-exporter-b[49519]: ts=2024-04-14T14:04:38.293Z caller=node_exporter.go:115 level=info collector=mdadm 2024-04-14T14:04:38.348 INFO:journalctl@ceph.node-exporter.b.smithi122.stdout:Apr 14 14:04:38 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-node-exporter-b[49519]: ts=2024-04-14T14:04:38.293Z caller=node_exporter.go:115 level=info collector=meminfo 2024-04-14T14:04:38.348 INFO:journalctl@ceph.node-exporter.b.smithi122.stdout:Apr 14 14:04:38 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-node-exporter-b[49519]: ts=2024-04-14T14:04:38.293Z caller=node_exporter.go:115 level=info collector=netclass 2024-04-14T14:04:38.348 INFO:journalctl@ceph.node-exporter.b.smithi122.stdout:Apr 14 14:04:38 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-node-exporter-b[49519]: ts=2024-04-14T14:04:38.293Z caller=node_exporter.go:115 level=info collector=netdev 2024-04-14T14:04:38.348 INFO:journalctl@ceph.node-exporter.b.smithi122.stdout:Apr 14 14:04:38 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-node-exporter-b[49519]: ts=2024-04-14T14:04:38.293Z caller=node_exporter.go:115 level=info collector=netstat 2024-04-14T14:04:38.348 INFO:journalctl@ceph.node-exporter.b.smithi122.stdout:Apr 14 14:04:38 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-node-exporter-b[49519]: ts=2024-04-14T14:04:38.293Z caller=node_exporter.go:115 level=info collector=nfs 2024-04-14T14:04:38.348 INFO:journalctl@ceph.node-exporter.b.smithi122.stdout:Apr 14 14:04:38 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-node-exporter-b[49519]: ts=2024-04-14T14:04:38.293Z caller=node_exporter.go:115 level=info collector=nfsd 2024-04-14T14:04:38.348 INFO:journalctl@ceph.node-exporter.b.smithi122.stdout:Apr 14 14:04:38 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-node-exporter-b[49519]: ts=2024-04-14T14:04:38.293Z caller=node_exporter.go:115 level=info collector=nvme 2024-04-14T14:04:38.348 INFO:journalctl@ceph.node-exporter.b.smithi122.stdout:Apr 14 14:04:38 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-node-exporter-b[49519]: ts=2024-04-14T14:04:38.293Z caller=node_exporter.go:115 level=info collector=os 2024-04-14T14:04:38.348 INFO:journalctl@ceph.node-exporter.b.smithi122.stdout:Apr 14 14:04:38 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-node-exporter-b[49519]: ts=2024-04-14T14:04:38.293Z caller=node_exporter.go:115 level=info collector=powersupplyclass 2024-04-14T14:04:38.348 INFO:journalctl@ceph.node-exporter.b.smithi122.stdout:Apr 14 14:04:38 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-node-exporter-b[49519]: ts=2024-04-14T14:04:38.293Z caller=node_exporter.go:115 level=info collector=pressure 2024-04-14T14:04:38.348 INFO:journalctl@ceph.node-exporter.b.smithi122.stdout:Apr 14 14:04:38 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-node-exporter-b[49519]: ts=2024-04-14T14:04:38.293Z caller=node_exporter.go:115 level=info collector=rapl 2024-04-14T14:04:38.349 INFO:journalctl@ceph.node-exporter.b.smithi122.stdout:Apr 14 14:04:38 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-node-exporter-b[49519]: ts=2024-04-14T14:04:38.293Z caller=node_exporter.go:115 level=info collector=schedstat 2024-04-14T14:04:38.349 INFO:journalctl@ceph.node-exporter.b.smithi122.stdout:Apr 14 14:04:38 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-node-exporter-b[49519]: ts=2024-04-14T14:04:38.293Z caller=node_exporter.go:115 level=info collector=sockstat 2024-04-14T14:04:38.349 INFO:journalctl@ceph.node-exporter.b.smithi122.stdout:Apr 14 14:04:38 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-node-exporter-b[49519]: ts=2024-04-14T14:04:38.293Z caller=node_exporter.go:115 level=info collector=softnet 2024-04-14T14:04:38.349 INFO:journalctl@ceph.node-exporter.b.smithi122.stdout:Apr 14 14:04:38 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-node-exporter-b[49519]: ts=2024-04-14T14:04:38.293Z caller=node_exporter.go:115 level=info collector=stat 2024-04-14T14:04:38.349 INFO:journalctl@ceph.node-exporter.b.smithi122.stdout:Apr 14 14:04:38 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-node-exporter-b[49519]: ts=2024-04-14T14:04:38.293Z caller=node_exporter.go:115 level=info collector=tapestats 2024-04-14T14:04:38.349 INFO:journalctl@ceph.node-exporter.b.smithi122.stdout:Apr 14 14:04:38 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-node-exporter-b[49519]: ts=2024-04-14T14:04:38.293Z caller=node_exporter.go:115 level=info collector=textfile 2024-04-14T14:04:38.349 INFO:journalctl@ceph.node-exporter.b.smithi122.stdout:Apr 14 14:04:38 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-node-exporter-b[49519]: ts=2024-04-14T14:04:38.293Z caller=node_exporter.go:115 level=info collector=thermal_zone 2024-04-14T14:04:38.349 INFO:journalctl@ceph.node-exporter.b.smithi122.stdout:Apr 14 14:04:38 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-node-exporter-b[49519]: ts=2024-04-14T14:04:38.293Z caller=node_exporter.go:115 level=info collector=time 2024-04-14T14:04:38.349 INFO:journalctl@ceph.node-exporter.b.smithi122.stdout:Apr 14 14:04:38 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-node-exporter-b[49519]: ts=2024-04-14T14:04:38.293Z caller=node_exporter.go:115 level=info collector=udp_queues 2024-04-14T14:04:38.349 INFO:journalctl@ceph.node-exporter.b.smithi122.stdout:Apr 14 14:04:38 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-node-exporter-b[49519]: ts=2024-04-14T14:04:38.293Z caller=node_exporter.go:115 level=info collector=uname 2024-04-14T14:04:38.349 INFO:journalctl@ceph.node-exporter.b.smithi122.stdout:Apr 14 14:04:38 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-node-exporter-b[49519]: ts=2024-04-14T14:04:38.293Z caller=node_exporter.go:115 level=info collector=vmstat 2024-04-14T14:04:38.349 INFO:journalctl@ceph.node-exporter.b.smithi122.stdout:Apr 14 14:04:38 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-node-exporter-b[49519]: ts=2024-04-14T14:04:38.293Z caller=node_exporter.go:115 level=info collector=xfs 2024-04-14T14:04:38.349 INFO:journalctl@ceph.node-exporter.b.smithi122.stdout:Apr 14 14:04:38 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-node-exporter-b[49519]: ts=2024-04-14T14:04:38.293Z caller=node_exporter.go:115 level=info collector=zfs 2024-04-14T14:04:38.349 INFO:journalctl@ceph.node-exporter.b.smithi122.stdout:Apr 14 14:04:38 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-node-exporter-b[49519]: ts=2024-04-14T14:04:38.293Z caller=node_exporter.go:199 level=info msg="Listening on" address=:9100 2024-04-14T14:04:38.350 INFO:journalctl@ceph.node-exporter.b.smithi122.stdout:Apr 14 14:04:38 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-node-exporter-b[49519]: ts=2024-04-14T14:04:38.293Z caller=tls_config.go:195 level=info msg="TLS is disabled." http2=false 2024-04-14T14:04:38.628 INFO:journalctl@ceph.node-exporter.b.smithi122.stdout:Apr 14 14:04:38 smithi122 bash[49504]: 996115353a0b0ad63833e7ac18a29d257d8b9f2ae3c8ab1a0efe8d419490c6ad 2024-04-14T14:04:38.628 INFO:journalctl@ceph.node-exporter.b.smithi122.stdout:Apr 14 14:04:38 smithi122 systemd[1]: Started Ceph node-exporter.b for dc326fc2-fa65-11ee-bc8e-c7b262605968. 2024-04-14T14:04:39.067 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:04:38 smithi069 ceph-mon[24818]: pgmap v25: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail 2024-04-14T14:04:39.067 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:04:38 smithi069 ceph-mon[24818]: from='client.? 172.21.15.69:0/1157801709' entity='client.iscsi.foo.smithi069.yuumtr' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.69:0/3960312281"}]': finished 2024-04-14T14:04:39.067 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:04:38 smithi069 ceph-mon[24818]: osdmap e97: 8 total, 8 up, 8 in 2024-04-14T14:04:39.067 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:04:38 smithi069 ceph-mon[24818]: from='client.? 172.21.15.69:0/2228573292' entity='client.iscsi.foo.smithi069.yuumtr' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.69:0/3671841875"}]: dispatch 2024-04-14T14:04:39.067 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:04:38 smithi069 ceph-mon[24818]: from='mgr.24749 ' entity='mgr.x' 2024-04-14T14:04:39.067 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:04:38 smithi069 ceph-mon[24818]: from='mgr.24749 ' entity='mgr.x' 2024-04-14T14:04:39.067 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:04:38 smithi069 ceph-mon[24818]: from='mgr.24749 ' entity='mgr.x' 2024-04-14T14:04:39.067 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:04:38 smithi069 ceph-mon[24818]: from='mgr.24749 ' entity='mgr.x' 2024-04-14T14:04:39.067 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:04:38 smithi069 ceph-mon[24818]: from='mgr.24749 172.21.15.122:0/1892077634' entity='mgr.x' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-04-14T14:04:39.067 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:04:38 smithi069 ceph-mon[29472]: pgmap v25: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail 2024-04-14T14:04:39.068 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:04:38 smithi069 ceph-mon[29472]: from='client.? 172.21.15.69:0/1157801709' entity='client.iscsi.foo.smithi069.yuumtr' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.69:0/3960312281"}]': finished 2024-04-14T14:04:39.068 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:04:38 smithi069 ceph-mon[29472]: osdmap e97: 8 total, 8 up, 8 in 2024-04-14T14:04:39.068 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:04:38 smithi069 ceph-mon[29472]: from='client.? 172.21.15.69:0/2228573292' entity='client.iscsi.foo.smithi069.yuumtr' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.69:0/3671841875"}]: dispatch 2024-04-14T14:04:39.068 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:04:38 smithi069 ceph-mon[29472]: from='mgr.24749 ' entity='mgr.x' 2024-04-14T14:04:39.068 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:04:38 smithi069 ceph-mon[29472]: from='mgr.24749 ' entity='mgr.x' 2024-04-14T14:04:39.068 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:04:38 smithi069 ceph-mon[29472]: from='mgr.24749 ' entity='mgr.x' 2024-04-14T14:04:39.068 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:04:38 smithi069 ceph-mon[29472]: from='mgr.24749 ' entity='mgr.x' 2024-04-14T14:04:39.068 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:04:38 smithi069 ceph-mon[29472]: from='mgr.24749 172.21.15.122:0/1892077634' entity='mgr.x' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-04-14T14:04:39.094 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:04:38 smithi122 ceph-mon[25721]: pgmap v25: 161 pgs: 161 active+clean; 585 KiB data, 67 MiB used, 715 GiB / 715 GiB avail 2024-04-14T14:04:39.094 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:04:38 smithi122 ceph-mon[25721]: from='client.? 172.21.15.69:0/1157801709' entity='client.iscsi.foo.smithi069.yuumtr' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.69:0/3960312281"}]': finished 2024-04-14T14:04:39.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:04:38 smithi122 ceph-mon[25721]: osdmap e97: 8 total, 8 up, 8 in 2024-04-14T14:04:39.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:04:38 smithi122 ceph-mon[25721]: from='client.? 172.21.15.69:0/2228573292' entity='client.iscsi.foo.smithi069.yuumtr' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.69:0/3671841875"}]: dispatch 2024-04-14T14:04:39.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:04:38 smithi122 ceph-mon[25721]: from='mgr.24749 ' entity='mgr.x' 2024-04-14T14:04:39.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:04:38 smithi122 ceph-mon[25721]: from='mgr.24749 ' entity='mgr.x' 2024-04-14T14:04:39.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:04:38 smithi122 ceph-mon[25721]: from='mgr.24749 ' entity='mgr.x' 2024-04-14T14:04:39.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:04:38 smithi122 ceph-mon[25721]: from='mgr.24749 ' entity='mgr.x' 2024-04-14T14:04:39.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:04:38 smithi122 ceph-mon[25721]: from='mgr.24749 172.21.15.122:0/1892077634' entity='mgr.x' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-04-14T14:04:40.067 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:04:39 smithi069 ceph-mon[24818]: Reconfiguring grafana.a (dependencies changed)... 2024-04-14T14:04:40.067 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:04:39 smithi069 ceph-mon[24818]: Regenerating cephadm self-signed grafana TLS certificates 2024-04-14T14:04:40.067 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:04:39 smithi069 ceph-mon[24818]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-04-14T14:04:40.067 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:04:39 smithi069 ceph-mon[24818]: Reconfiguring daemon grafana.a on smithi122 2024-04-14T14:04:40.067 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:04:39 smithi069 ceph-mon[24818]: from='client.? 172.21.15.69:0/2228573292' entity='client.iscsi.foo.smithi069.yuumtr' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.69:0/3671841875"}]': finished 2024-04-14T14:04:40.067 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:04:39 smithi069 ceph-mon[24818]: osdmap e98: 8 total, 8 up, 8 in 2024-04-14T14:04:40.067 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:04:39 smithi069 ceph-mon[24818]: from='client.? 172.21.15.69:0/173422370' entity='client.iscsi.foo.smithi069.yuumtr' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.69:0/4037048393"}]: dispatch 2024-04-14T14:04:40.068 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:04:39 smithi069 ceph-mon[24818]: from='client.? ' entity='client.iscsi.foo.smithi069.yuumtr' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.69:0/4037048393"}]: dispatch 2024-04-14T14:04:40.068 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:04:39 smithi069 ceph-mon[29472]: Reconfiguring grafana.a (dependencies changed)... 2024-04-14T14:04:40.068 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:04:39 smithi069 ceph-mon[29472]: Regenerating cephadm self-signed grafana TLS certificates 2024-04-14T14:04:40.068 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:04:39 smithi069 ceph-mon[29472]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-04-14T14:04:40.068 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:04:39 smithi069 ceph-mon[29472]: Reconfiguring daemon grafana.a on smithi122 2024-04-14T14:04:40.068 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:04:39 smithi069 ceph-mon[29472]: from='client.? 172.21.15.69:0/2228573292' entity='client.iscsi.foo.smithi069.yuumtr' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.69:0/3671841875"}]': finished 2024-04-14T14:04:40.068 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:04:39 smithi069 ceph-mon[29472]: osdmap e98: 8 total, 8 up, 8 in 2024-04-14T14:04:40.068 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:04:39 smithi069 ceph-mon[29472]: from='client.? 172.21.15.69:0/173422370' entity='client.iscsi.foo.smithi069.yuumtr' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.69:0/4037048393"}]: dispatch 2024-04-14T14:04:40.068 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:04:39 smithi069 ceph-mon[29472]: from='client.? ' entity='client.iscsi.foo.smithi069.yuumtr' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.69:0/4037048393"}]: dispatch 2024-04-14T14:04:40.094 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:04:39 smithi122 ceph-mon[25721]: Reconfiguring grafana.a (dependencies changed)... 2024-04-14T14:04:40.094 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:04:39 smithi122 ceph-mon[25721]: Regenerating cephadm self-signed grafana TLS certificates 2024-04-14T14:04:40.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:04:39 smithi122 ceph-mon[25721]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-04-14T14:04:40.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:04:39 smithi122 ceph-mon[25721]: Reconfiguring daemon grafana.a on smithi122 2024-04-14T14:04:40.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:04:39 smithi122 ceph-mon[25721]: from='client.? 172.21.15.69:0/2228573292' entity='client.iscsi.foo.smithi069.yuumtr' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.69:0/3671841875"}]': finished 2024-04-14T14:04:40.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:04:39 smithi122 ceph-mon[25721]: osdmap e98: 8 total, 8 up, 8 in 2024-04-14T14:04:40.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:04:39 smithi122 ceph-mon[25721]: from='client.? 172.21.15.69:0/173422370' entity='client.iscsi.foo.smithi069.yuumtr' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.69:0/4037048393"}]: dispatch 2024-04-14T14:04:40.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:04:39 smithi122 ceph-mon[25721]: from='client.? ' entity='client.iscsi.foo.smithi069.yuumtr' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.69:0/4037048393"}]: dispatch 2024-04-14T14:04:41.067 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:04:40 smithi069 ceph-mon[24818]: pgmap v28: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:04:41.067 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:04:40 smithi069 ceph-mon[24818]: from='client.? ' entity='client.iscsi.foo.smithi069.yuumtr' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.69:0/4037048393"}]': finished 2024-04-14T14:04:41.067 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:04:40 smithi069 ceph-mon[24818]: osdmap e99: 8 total, 8 up, 8 in 2024-04-14T14:04:41.067 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:04:40 smithi069 ceph-mon[24818]: from='client.? 172.21.15.69:0/477960801' entity='client.iscsi.foo.smithi069.yuumtr' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.69:6801/3122031884"}]: dispatch 2024-04-14T14:04:41.067 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:04:40 smithi069 ceph-mon[24818]: from='client.? ' entity='client.iscsi.foo.smithi069.yuumtr' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.69:6801/3122031884"}]: dispatch 2024-04-14T14:04:41.067 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:04:40 smithi069 ceph-mon[29472]: pgmap v28: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:04:41.068 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:04:40 smithi069 ceph-mon[29472]: from='client.? ' entity='client.iscsi.foo.smithi069.yuumtr' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.69:0/4037048393"}]': finished 2024-04-14T14:04:41.068 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:04:40 smithi069 ceph-mon[29472]: osdmap e99: 8 total, 8 up, 8 in 2024-04-14T14:04:41.068 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:04:40 smithi069 ceph-mon[29472]: from='client.? 172.21.15.69:0/477960801' entity='client.iscsi.foo.smithi069.yuumtr' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.69:6801/3122031884"}]: dispatch 2024-04-14T14:04:41.068 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:04:40 smithi069 ceph-mon[29472]: from='client.? ' entity='client.iscsi.foo.smithi069.yuumtr' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.69:6801/3122031884"}]: dispatch 2024-04-14T14:04:41.094 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:04:40 smithi122 ceph-mon[25721]: pgmap v28: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:04:41.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:04:40 smithi122 ceph-mon[25721]: from='client.? ' entity='client.iscsi.foo.smithi069.yuumtr' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.69:0/4037048393"}]': finished 2024-04-14T14:04:41.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:04:40 smithi122 ceph-mon[25721]: osdmap e99: 8 total, 8 up, 8 in 2024-04-14T14:04:41.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:04:40 smithi122 ceph-mon[25721]: from='client.? 172.21.15.69:0/477960801' entity='client.iscsi.foo.smithi069.yuumtr' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.69:6801/3122031884"}]: dispatch 2024-04-14T14:04:41.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:04:40 smithi122 ceph-mon[25721]: from='client.? ' entity='client.iscsi.foo.smithi069.yuumtr' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.69:6801/3122031884"}]: dispatch 2024-04-14T14:04:42.067 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:04:41 smithi069 ceph-mon[24818]: from='client.? ' entity='client.iscsi.foo.smithi069.yuumtr' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.69:6801/3122031884"}]': finished 2024-04-14T14:04:42.067 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:04:41 smithi069 ceph-mon[24818]: osdmap e100: 8 total, 8 up, 8 in 2024-04-14T14:04:42.067 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:04:41 smithi069 ceph-mon[24818]: from='client.? 172.21.15.69:0/1401755452' entity='client.iscsi.foo.smithi069.yuumtr' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.69:6800/3122031884"}]: dispatch 2024-04-14T14:04:42.067 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:04:41 smithi069 ceph-mon[24818]: from='client.? ' entity='client.iscsi.foo.smithi069.yuumtr' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.69:6800/3122031884"}]: dispatch 2024-04-14T14:04:42.067 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:04:41 smithi069 ceph-mon[29472]: from='client.? ' entity='client.iscsi.foo.smithi069.yuumtr' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.69:6801/3122031884"}]': finished 2024-04-14T14:04:42.067 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:04:41 smithi069 ceph-mon[29472]: osdmap e100: 8 total, 8 up, 8 in 2024-04-14T14:04:42.067 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:04:41 smithi069 ceph-mon[29472]: from='client.? 172.21.15.69:0/1401755452' entity='client.iscsi.foo.smithi069.yuumtr' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.69:6800/3122031884"}]: dispatch 2024-04-14T14:04:42.067 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:04:41 smithi069 ceph-mon[29472]: from='client.? ' entity='client.iscsi.foo.smithi069.yuumtr' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.69:6800/3122031884"}]: dispatch 2024-04-14T14:04:42.094 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:04:41 smithi122 ceph-mon[25721]: from='client.? ' entity='client.iscsi.foo.smithi069.yuumtr' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.69:6801/3122031884"}]': finished 2024-04-14T14:04:42.094 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:04:41 smithi122 ceph-mon[25721]: osdmap e100: 8 total, 8 up, 8 in 2024-04-14T14:04:42.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:04:41 smithi122 ceph-mon[25721]: from='client.? 172.21.15.69:0/1401755452' entity='client.iscsi.foo.smithi069.yuumtr' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.69:6800/3122031884"}]: dispatch 2024-04-14T14:04:42.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:04:41 smithi122 ceph-mon[25721]: from='client.? ' entity='client.iscsi.foo.smithi069.yuumtr' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.69:6800/3122031884"}]: dispatch 2024-04-14T14:04:43.067 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:04:42 smithi069 ceph-mon[24818]: pgmap v31: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:04:43.067 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:04:42 smithi069 ceph-mon[24818]: from='client.? ' entity='client.iscsi.foo.smithi069.yuumtr' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.69:6800/3122031884"}]': finished 2024-04-14T14:04:43.067 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:04:42 smithi069 ceph-mon[24818]: osdmap e101: 8 total, 8 up, 8 in 2024-04-14T14:04:43.067 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:04:42 smithi069 ceph-mon[24818]: from='mgr.24749 172.21.15.122:0/1892077634' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T14:04:43.067 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:04:42 smithi069 ceph-mon[29472]: pgmap v31: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:04:43.067 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:04:42 smithi069 ceph-mon[29472]: from='client.? ' entity='client.iscsi.foo.smithi069.yuumtr' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.69:6800/3122031884"}]': finished 2024-04-14T14:04:43.067 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:04:42 smithi069 ceph-mon[29472]: osdmap e101: 8 total, 8 up, 8 in 2024-04-14T14:04:43.067 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:04:42 smithi069 ceph-mon[29472]: from='mgr.24749 172.21.15.122:0/1892077634' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T14:04:43.094 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Apr 14 14:04:42 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-x[45487]: ::ffff:172.21.15.122 - - [14/Apr/2024:14:04:42] "GET /metrics HTTP/1.1" 200 37891 "" "Prometheus/2.33.4" 2024-04-14T14:04:43.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:04:42 smithi122 ceph-mon[25721]: pgmap v31: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:04:43.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:04:42 smithi122 ceph-mon[25721]: from='client.? ' entity='client.iscsi.foo.smithi069.yuumtr' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.69:6800/3122031884"}]': finished 2024-04-14T14:04:43.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:04:42 smithi122 ceph-mon[25721]: osdmap e101: 8 total, 8 up, 8 in 2024-04-14T14:04:43.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:04:42 smithi122 ceph-mon[25721]: from='mgr.24749 172.21.15.122:0/1892077634' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T14:04:45.066 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:04:44 smithi069 ceph-mon[24818]: pgmap v33: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail 2024-04-14T14:04:45.067 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:04:44 smithi069 ceph-mon[29472]: pgmap v33: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail 2024-04-14T14:04:45.094 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:04:44 smithi122 ceph-mon[25721]: pgmap v33: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail 2024-04-14T14:04:45.937 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:04:45 smithi122 ceph-mon[25721]: from='client.24985 -' entity='client.iscsi.foo.smithi069.yuumtr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T14:04:45.983 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:04:45 smithi069 ceph-mon[29472]: from='client.24985 -' entity='client.iscsi.foo.smithi069.yuumtr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T14:04:45.984 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:04:45 smithi069 ceph-mon[24818]: from='client.24985 -' entity='client.iscsi.foo.smithi069.yuumtr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T14:04:46.974 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:04:46 smithi122 ceph-mon[25721]: pgmap v34: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:04:47.066 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:04:46 smithi069 ceph-mon[24818]: pgmap v34: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:04:47.067 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:04:46 smithi069 ceph-mon[29472]: pgmap v34: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:04:48.345 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Apr 14 14:04:48 smithi122 systemd[1]: Stopping Ceph grafana.a for dc326fc2-fa65-11ee-bc8e-c7b262605968... 2024-04-14T14:04:48.345 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Apr 14 14:04:48 smithi122 bash[49799]: Error: no container with name or ID "ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-grafana.a" found: no such container 2024-04-14T14:04:48.345 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Apr 14 14:04:48 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-grafana-a[41857]: t=2024-04-14T14:04:48+0000 lvl=info msg="Shutdown started" logger=server reason="System signal: terminated" 2024-04-14T14:04:48.345 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Apr 14 14:04:48 smithi122 podman[49812]: 2024-04-14 14:04:48.278292261 +0000 UTC m=+0.138085536 container died 6c0d806d20ea2df3985053e7ce0ab16ace983a6480fb12b94b7b65c9fa414f72 (image=quay.io/ceph/ceph-grafana:8.3.5, name=ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-grafana-a, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-236.1648460182, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, io.buildah.version=1.24.2, vcs-type=git, description=Ceph Grafana Container, io.openshift.tags=base rhel8, com.redhat.component=ubi8-container, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, architecture=x86_64, distribution-scope=public, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., summary=Grafana Container configured for Ceph mgr/dashboard integration, build-date=2022-03-28T10:36:18.413762, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, vendor=Red Hat, Inc., maintainer=Paul Cuzner , release=236.1648460182, name=ubi8, io.k8s.display-name=Red Hat Universal Base Image 8, io.openshift.expose-services=, version=8.5) 2024-04-14T14:04:48.828 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:04:48 smithi122 ceph-mon[25721]: pgmap v35: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 647 B/s rd, 0 op/s 2024-04-14T14:04:48.828 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Apr 14 14:04:48 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-prometheus-a[49277]: ts=2024-04-14T14:04:48.659Z caller=manager.go:609 level=warn component="rule manager" group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on(ceph_daemon) group_left(hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.69\", device_class=\"ssd\", hostname=\"smithi069\", instance=\"172.21.15.122:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.69\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster=\"dc326fc2-fa65-11ee-bc8e-c7b262605968\", cluster_addr=\"172.21.15.69\", device_class=\"ssd\", hostname=\"smithi069\", instance=\"smithi122:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.69\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-04-14T14:04:48.829 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Apr 14 14:04:48 smithi122 podman[49812]: 2024-04-14 14:04:48.592497878 +0000 UTC m=+0.452291153 container cleanup 6c0d806d20ea2df3985053e7ce0ab16ace983a6480fb12b94b7b65c9fa414f72 (image=quay.io/ceph/ceph-grafana:8.3.5, name=ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-grafana-a, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, com.redhat.component=ubi8-container, vcs-type=git, distribution-scope=public, name=ubi8, architecture=x86_64, io.k8s.display-name=Red Hat Universal Base Image 8, io.openshift.expose-services=, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, maintainer=Paul Cuzner , build-date=2022-03-28T10:36:18.413762, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., summary=Grafana Container configured for Ceph mgr/dashboard integration, io.buildah.version=1.24.2, release=236.1648460182, description=Ceph Grafana Container, io.openshift.tags=base rhel8, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-236.1648460182, vendor=Red Hat, Inc., version=8.5) 2024-04-14T14:04:48.829 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Apr 14 14:04:48 smithi122 bash[49812]: ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-grafana-a 2024-04-14T14:04:48.829 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Apr 14 14:04:48 smithi122 bash[49838]: Error: no container with name or ID "ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-grafana.a" found: no such container 2024-04-14T14:04:49.066 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:04:48 smithi069 ceph-mon[24818]: pgmap v35: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 647 B/s rd, 0 op/s 2024-04-14T14:04:49.067 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:04:48 smithi069 ceph-mon[29472]: pgmap v35: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 647 B/s rd, 0 op/s 2024-04-14T14:04:49.095 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Apr 14 14:04:48 smithi122 podman[49826]: 2024-04-14 14:04:48.826882244 +0000 UTC m=+0.549332950 container remove 6c0d806d20ea2df3985053e7ce0ab16ace983a6480fb12b94b7b65c9fa414f72 (image=quay.io/ceph/ceph-grafana:8.3.5, name=ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-grafana-a, build-date=2022-03-28T10:36:18.413762, description=Ceph Grafana Container, name=ubi8, com.redhat.component=ubi8-container, architecture=x86_64, version=8.5, release=236.1648460182, distribution-scope=public, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., vcs-type=git, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, io.buildah.version=1.24.2, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, summary=Grafana Container configured for Ceph mgr/dashboard integration, maintainer=Paul Cuzner , vendor=Red Hat, Inc., io.openshift.expose-services=, io.openshift.tags=base rhel8, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, io.k8s.display-name=Red Hat Universal Base Image 8, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-236.1648460182) 2024-04-14T14:04:49.095 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Apr 14 14:04:48 smithi122 systemd[1]: ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968@grafana.a.service: Deactivated successfully. 2024-04-14T14:04:49.095 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Apr 14 14:04:48 smithi122 systemd[1]: Stopped Ceph grafana.a for dc326fc2-fa65-11ee-bc8e-c7b262605968. 2024-04-14T14:04:49.095 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Apr 14 14:04:48 smithi122 systemd[1]: ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968@grafana.a.service: Consumed 3.293s CPU time. 2024-04-14T14:04:49.095 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Apr 14 14:04:48 smithi122 systemd[1]: Starting Ceph grafana.a for dc326fc2-fa65-11ee-bc8e-c7b262605968... 2024-04-14T14:04:49.095 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Apr 14 14:04:49 smithi122 podman[49906]: 2024-04-14 14:04:48.954159866 +0000 UTC m=+0.017704807 image pull dad864ee21e98e69f4029d1e417aa085001566be0d322fbc75bc6f29b0050c01 quay.io/ceph/ceph-grafana:8.3.5 2024-04-14T14:04:49.095 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Apr 14 14:04:49 smithi122 podman[49906]: 2024-04-14 14:04:49.069557738 +0000 UTC m=+0.133102666 container create 9a309479c12b8a1470533e5e24faf0160e665c3a3825a22401aeb85fb40a9539 (image=quay.io/ceph/ceph-grafana:8.3.5, name=ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-grafana-a, name=ubi8, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, com.redhat.component=ubi8-container, io.openshift.tags=base rhel8, io.openshift.expose-services=, io.k8s.display-name=Red Hat Universal Base Image 8, summary=Grafana Container configured for Ceph mgr/dashboard integration, description=Ceph Grafana Container, vcs-type=git, vendor=Red Hat, Inc., io.buildah.version=1.24.2, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-236.1648460182, maintainer=Paul Cuzner , release=236.1648460182, distribution-scope=public, architecture=x86_64, build-date=2022-03-28T10:36:18.413762, version=8.5, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly.) 2024-04-14T14:04:49.454 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Apr 14 14:04:49 smithi122 podman[49906]: 2024-04-14 14:04:49.305373454 +0000 UTC m=+0.368918381 container init 9a309479c12b8a1470533e5e24faf0160e665c3a3825a22401aeb85fb40a9539 (image=quay.io/ceph/ceph-grafana:8.3.5, name=ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-grafana-a, distribution-scope=public, vendor=Red Hat, Inc., com.redhat.component=ubi8-container, io.buildah.version=1.24.2, build-date=2022-03-28T10:36:18.413762, version=8.5, description=Ceph Grafana Container, release=236.1648460182, name=ubi8, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, io.openshift.tags=base rhel8, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-236.1648460182, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, io.k8s.display-name=Red Hat Universal Base Image 8, summary=Grafana Container configured for Ceph mgr/dashboard integration, vcs-type=git, io.openshift.expose-services=, architecture=x86_64, maintainer=Paul Cuzner ) 2024-04-14T14:04:49.454 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Apr 14 14:04:49 smithi122 podman[49906]: 2024-04-14 14:04:49.307908258 +0000 UTC m=+0.371453187 container start 9a309479c12b8a1470533e5e24faf0160e665c3a3825a22401aeb85fb40a9539 (image=quay.io/ceph/ceph-grafana:8.3.5, name=ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-grafana-a, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, build-date=2022-03-28T10:36:18.413762, summary=Grafana Container configured for Ceph mgr/dashboard integration, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-236.1648460182, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, io.buildah.version=1.24.2, name=ubi8, release=236.1648460182, architecture=x86_64, io.openshift.expose-services=, io.k8s.display-name=Red Hat Universal Base Image 8, maintainer=Paul Cuzner , distribution-scope=public, io.openshift.tags=base rhel8, description=Ceph Grafana Container, vendor=Red Hat, Inc., com.redhat.component=ubi8-container, version=8.5, vcs-type=git) 2024-04-14T14:04:49.454 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Apr 14 14:04:49 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-grafana-a[49921]: Failed to start grafana. error: failed to create log directory "/usr/share/grafana/data/log": mkdir /usr/share/grafana/data: permission denied 2024-04-14T14:04:49.454 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Apr 14 14:04:49 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-grafana-a[49921]: failed to create log directory "/usr/share/grafana/data/log": mkdir /usr/share/grafana/data: permission denied 2024-04-14T14:04:49.454 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Apr 14 14:04:49 smithi122 bash[49906]: 9a309479c12b8a1470533e5e24faf0160e665c3a3825a22401aeb85fb40a9539 2024-04-14T14:04:49.454 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Apr 14 14:04:49 smithi122 systemd[1]: Started Ceph grafana.a for dc326fc2-fa65-11ee-bc8e-c7b262605968. 2024-04-14T14:04:49.455 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Apr 14 14:04:49 smithi122 podman[49937]: 2024-04-14 14:04:49.395584192 +0000 UTC m=+0.052839595 container died 9a309479c12b8a1470533e5e24faf0160e665c3a3825a22401aeb85fb40a9539 (image=quay.io/ceph/ceph-grafana:8.3.5, name=ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-grafana-a, maintainer=Paul Cuzner , vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, name=ubi8, summary=Grafana Container configured for Ceph mgr/dashboard integration, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., release=236.1648460182, io.buildah.version=1.24.2, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, build-date=2022-03-28T10:36:18.413762, com.redhat.component=ubi8-container, io.openshift.expose-services=, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-236.1648460182, description=Ceph Grafana Container, version=8.5, io.k8s.display-name=Red Hat Universal Base Image 8, vendor=Red Hat, Inc., com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, distribution-scope=public, io.openshift.tags=base rhel8, vcs-type=git, architecture=x86_64) 2024-04-14T14:04:49.801 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Apr 14 14:04:49 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-x[45487]: [14/Apr/2024:14:04:49] ENGINE Bus STOPPING 2024-04-14T14:04:49.801 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Apr 14 14:04:49 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-x[45487]: [14/Apr/2024:14:04:49] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-04-14T14:04:49.802 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Apr 14 14:04:49 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-x[45487]: [14/Apr/2024:14:04:49] ENGINE Bus STOPPED 2024-04-14T14:04:49.802 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Apr 14 14:04:49 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-x[45487]: [14/Apr/2024:14:04:49] ENGINE Bus STARTING 2024-04-14T14:04:49.802 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Apr 14 14:04:49 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-x[45487]: [14/Apr/2024:14:04:49] ENGINE Serving on http://:::9283 2024-04-14T14:04:49.802 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Apr 14 14:04:49 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-x[45487]: [14/Apr/2024:14:04:49] ENGINE Bus STARTED 2024-04-14T14:04:49.802 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Apr 14 14:04:49 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-x[45487]: [14/Apr/2024:14:04:49] ENGINE Bus STOPPING 2024-04-14T14:04:49.802 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Apr 14 14:04:49 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-x[45487]: [14/Apr/2024:14:04:49] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-04-14T14:04:49.802 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Apr 14 14:04:49 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-x[45487]: [14/Apr/2024:14:04:49] ENGINE Bus STOPPED 2024-04-14T14:04:49.802 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Apr 14 14:04:49 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-x[45487]: [14/Apr/2024:14:04:49] ENGINE Bus STARTING 2024-04-14T14:04:49.802 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Apr 14 14:04:49 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-x[45487]: [14/Apr/2024:14:04:49] ENGINE Serving on http://:::9283 2024-04-14T14:04:49.802 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Apr 14 14:04:49 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-x[45487]: [14/Apr/2024:14:04:49] ENGINE Bus STARTED 2024-04-14T14:04:49.802 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Apr 14 14:04:49 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-x[45487]: [14/Apr/2024:14:04:49] ENGINE Bus STOPPING 2024-04-14T14:04:49.802 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Apr 14 14:04:49 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-x[45487]: [14/Apr/2024:14:04:49] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-04-14T14:04:49.802 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Apr 14 14:04:49 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-x[45487]: [14/Apr/2024:14:04:49] ENGINE Bus STOPPED 2024-04-14T14:04:49.802 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Apr 14 14:04:49 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-x[45487]: [14/Apr/2024:14:04:49] ENGINE Bus STARTING 2024-04-14T14:04:49.803 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Apr 14 14:04:49 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-x[45487]: [14/Apr/2024:14:04:49] ENGINE Serving on http://:::9283 2024-04-14T14:04:49.803 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Apr 14 14:04:49 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-x[45487]: [14/Apr/2024:14:04:49] ENGINE Bus STARTED 2024-04-14T14:04:50.072 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Apr 14 14:04:49 smithi122 podman[49937]: 2024-04-14 14:04:49.993349703 +0000 UTC m=+0.650605089 container remove 9a309479c12b8a1470533e5e24faf0160e665c3a3825a22401aeb85fb40a9539 (image=quay.io/ceph/ceph-grafana:8.3.5, name=ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-grafana-a, maintainer=Paul Cuzner , name=ubi8, io.openshift.tags=base rhel8, version=8.5, summary=Grafana Container configured for Ceph mgr/dashboard integration, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-236.1648460182, build-date=2022-03-28T10:36:18.413762, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, architecture=x86_64, com.redhat.component=ubi8-container, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., vendor=Red Hat, Inc., release=236.1648460182, vcs-type=git, distribution-scope=public, io.buildah.version=1.24.2, io.k8s.display-name=Red Hat Universal Base Image 8, description=Ceph Grafana Container, io.openshift.expose-services=, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56) 2024-04-14T14:04:50.073 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Apr 14 14:04:49 smithi122 systemd[1]: ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968@grafana.a.service: Main process exited, code=exited, status=1/FAILURE 2024-04-14T14:04:50.073 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Apr 14 14:04:50 smithi122 systemd[1]: ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968@grafana.a.service: Failed with result 'exit-code'. 2024-04-14T14:04:50.567 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:04:50 smithi069 ceph-mon[29472]: from='mgr.24749 ' entity='mgr.x' 2024-04-14T14:04:50.567 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:04:50 smithi069 ceph-mon[29472]: from='mgr.24749 ' entity='mgr.x' 2024-04-14T14:04:50.567 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:04:50 smithi069 ceph-mon[29472]: from='mgr.24749 172.21.15.122:0/1892077634' entity='mgr.x' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-04-14T14:04:50.567 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:04:50 smithi069 ceph-mon[29472]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-04-14T14:04:50.567 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:04:50 smithi069 ceph-mon[29472]: from='mgr.24749 172.21.15.122:0/1892077634' entity='mgr.x' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi069.front.sepia.ceph.com:9093"}]: dispatch 2024-04-14T14:04:50.567 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:04:50 smithi069 ceph-mon[29472]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi069.front.sepia.ceph.com:9093"}]: dispatch 2024-04-14T14:04:50.567 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:04:50 smithi069 ceph-mon[29472]: from='mgr.24749 ' entity='mgr.x' 2024-04-14T14:04:50.568 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:04:50 smithi069 ceph-mon[29472]: from='mgr.24749 172.21.15.122:0/1892077634' entity='mgr.x' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-04-14T14:04:50.568 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:04:50 smithi069 ceph-mon[29472]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-04-14T14:04:50.568 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:04:50 smithi069 ceph-mon[29472]: from='mgr.24749 172.21.15.122:0/1892077634' entity='mgr.x' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi122.front.sepia.ceph.com:9095"}]: dispatch 2024-04-14T14:04:50.568 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:04:50 smithi069 ceph-mon[29472]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi122.front.sepia.ceph.com:9095"}]: dispatch 2024-04-14T14:04:50.568 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:04:50 smithi069 ceph-mon[29472]: from='mgr.24749 ' entity='mgr.x' 2024-04-14T14:04:50.568 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:04:50 smithi069 ceph-mon[29472]: from='mgr.24749 172.21.15.122:0/1892077634' entity='mgr.x' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-04-14T14:04:50.568 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:04:50 smithi069 ceph-mon[29472]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-04-14T14:04:50.568 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:04:50 smithi069 ceph-mon[29472]: from='mgr.24749 172.21.15.122:0/1892077634' entity='mgr.x' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi122.front.sepia.ceph.com:3000"}]: dispatch 2024-04-14T14:04:50.568 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:04:50 smithi069 ceph-mon[29472]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi122.front.sepia.ceph.com:3000"}]: dispatch 2024-04-14T14:04:50.568 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:04:50 smithi069 ceph-mon[29472]: from='mgr.24749 ' entity='mgr.x' 2024-04-14T14:04:50.568 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:04:50 smithi069 ceph-mon[29472]: pgmap v36: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-04-14T14:04:50.569 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:04:50 smithi069 ceph-mon[24818]: from='mgr.24749 ' entity='mgr.x' 2024-04-14T14:04:50.569 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:04:50 smithi069 ceph-mon[24818]: from='mgr.24749 ' entity='mgr.x' 2024-04-14T14:04:50.569 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:04:50 smithi069 ceph-mon[24818]: from='mgr.24749 172.21.15.122:0/1892077634' entity='mgr.x' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-04-14T14:04:50.569 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:04:50 smithi069 ceph-mon[24818]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-04-14T14:04:50.569 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:04:50 smithi069 ceph-mon[24818]: from='mgr.24749 172.21.15.122:0/1892077634' entity='mgr.x' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi069.front.sepia.ceph.com:9093"}]: dispatch 2024-04-14T14:04:50.569 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:04:50 smithi069 ceph-mon[24818]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi069.front.sepia.ceph.com:9093"}]: dispatch 2024-04-14T14:04:50.569 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:04:50 smithi069 ceph-mon[24818]: from='mgr.24749 ' entity='mgr.x' 2024-04-14T14:04:50.569 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:04:50 smithi069 ceph-mon[24818]: from='mgr.24749 172.21.15.122:0/1892077634' entity='mgr.x' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-04-14T14:04:50.569 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:04:50 smithi069 ceph-mon[24818]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-04-14T14:04:50.569 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:04:50 smithi069 ceph-mon[24818]: from='mgr.24749 172.21.15.122:0/1892077634' entity='mgr.x' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi122.front.sepia.ceph.com:9095"}]: dispatch 2024-04-14T14:04:50.569 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:04:50 smithi069 ceph-mon[24818]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi122.front.sepia.ceph.com:9095"}]: dispatch 2024-04-14T14:04:50.569 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:04:50 smithi069 ceph-mon[24818]: from='mgr.24749 ' entity='mgr.x' 2024-04-14T14:04:50.569 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:04:50 smithi069 ceph-mon[24818]: from='mgr.24749 172.21.15.122:0/1892077634' entity='mgr.x' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-04-14T14:04:50.570 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:04:50 smithi069 ceph-mon[24818]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-04-14T14:04:50.570 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:04:50 smithi069 ceph-mon[24818]: from='mgr.24749 172.21.15.122:0/1892077634' entity='mgr.x' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi122.front.sepia.ceph.com:3000"}]: dispatch 2024-04-14T14:04:50.570 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:04:50 smithi069 ceph-mon[24818]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi122.front.sepia.ceph.com:3000"}]: dispatch 2024-04-14T14:04:50.570 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:04:50 smithi069 ceph-mon[24818]: from='mgr.24749 ' entity='mgr.x' 2024-04-14T14:04:50.570 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:04:50 smithi069 ceph-mon[24818]: pgmap v36: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-04-14T14:04:50.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:04:50 smithi122 ceph-mon[25721]: from='mgr.24749 ' entity='mgr.x' 2024-04-14T14:04:50.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:04:50 smithi122 ceph-mon[25721]: from='mgr.24749 ' entity='mgr.x' 2024-04-14T14:04:50.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:04:50 smithi122 ceph-mon[25721]: from='mgr.24749 172.21.15.122:0/1892077634' entity='mgr.x' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-04-14T14:04:50.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:04:50 smithi122 ceph-mon[25721]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-04-14T14:04:50.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:04:50 smithi122 ceph-mon[25721]: from='mgr.24749 172.21.15.122:0/1892077634' entity='mgr.x' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi069.front.sepia.ceph.com:9093"}]: dispatch 2024-04-14T14:04:50.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:04:50 smithi122 ceph-mon[25721]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi069.front.sepia.ceph.com:9093"}]: dispatch 2024-04-14T14:04:50.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:04:50 smithi122 ceph-mon[25721]: from='mgr.24749 ' entity='mgr.x' 2024-04-14T14:04:50.846 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:04:50 smithi122 ceph-mon[25721]: from='mgr.24749 172.21.15.122:0/1892077634' entity='mgr.x' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-04-14T14:04:50.846 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:04:50 smithi122 ceph-mon[25721]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-04-14T14:04:50.846 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:04:50 smithi122 ceph-mon[25721]: from='mgr.24749 172.21.15.122:0/1892077634' entity='mgr.x' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi122.front.sepia.ceph.com:9095"}]: dispatch 2024-04-14T14:04:50.846 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:04:50 smithi122 ceph-mon[25721]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi122.front.sepia.ceph.com:9095"}]: dispatch 2024-04-14T14:04:50.846 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:04:50 smithi122 ceph-mon[25721]: from='mgr.24749 ' entity='mgr.x' 2024-04-14T14:04:50.846 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:04:50 smithi122 ceph-mon[25721]: from='mgr.24749 172.21.15.122:0/1892077634' entity='mgr.x' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-04-14T14:04:50.846 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:04:50 smithi122 ceph-mon[25721]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-04-14T14:04:50.846 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:04:50 smithi122 ceph-mon[25721]: from='mgr.24749 172.21.15.122:0/1892077634' entity='mgr.x' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi122.front.sepia.ceph.com:3000"}]: dispatch 2024-04-14T14:04:50.846 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:04:50 smithi122 ceph-mon[25721]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi122.front.sepia.ceph.com:3000"}]: dispatch 2024-04-14T14:04:50.846 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:04:50 smithi122 ceph-mon[25721]: from='mgr.24749 ' entity='mgr.x' 2024-04-14T14:04:50.846 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:04:50 smithi122 ceph-mon[25721]: pgmap v36: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-04-14T14:04:52.803 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:04:52 smithi122 ceph-mon[25721]: pgmap v37: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-14T14:04:53.066 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:04:52 smithi069 ceph-mon[24818]: pgmap v37: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-14T14:04:53.067 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:04:52 smithi069 ceph-mon[29472]: pgmap v37: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-14T14:04:53.094 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Apr 14 14:04:52 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-x[45487]: ::ffff:172.21.15.122 - - [14/Apr/2024:14:04:52] "GET /metrics HTTP/1.1" 200 37891 "" "Prometheus/2.33.4" 2024-04-14T14:04:53.844 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Apr 14 14:04:53 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-prometheus-a[49277]: ts=2024-04-14T14:04:53.504Z caller=manager.go:609 level=warn component="rule manager" group=nodes msg="Evaluating rule failed" rule="alert: CephNodeDiskspaceWarning\nexpr: predict_linear(node_filesystem_free_bytes{device=~\"/.*\"}[2d], 3600 * 24 * 5)\n * on(instance) group_left(nodename) node_uname_info < 0\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.8.4\n severity: warning\n type: ceph_default\nannotations:\n description: Mountpoint {{ $labels.mountpoint }} on {{ $labels.nodename }} will\n be full in less than 5 days based on the 48 hour trailing fill rate.\n summary: Host filesystem free space is getting low\n" err="found duplicate series for the match group {instance=\"smithi069\"} on the right hand-side of the operation: [{__name__=\"node_uname_info\", domainname=\"(none)\", instance=\"smithi069\", job=\"node\", machine=\"x86_64\", nodename=\"smithi069\", release=\"5.14.0-435.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Fri Mar 29 18:46:37 UTC 2024\"}, {__name__=\"node_uname_info\", cluster=\"dc326fc2-fa65-11ee-bc8e-c7b262605968\", domainname=\"(none)\", instance=\"smithi069\", job=\"node\", machine=\"x86_64\", nodename=\"smithi069\", release=\"5.14.0-435.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Fri Mar 29 18:46:37 UTC 2024\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-04-14T14:04:54.280 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:04:54 smithi069 ceph-mon[24818]: from='mgr.24749 ' entity='mgr.x' 2024-04-14T14:04:54.280 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:04:54 smithi069 ceph-mon[24818]: from='mgr.24749 ' entity='mgr.x' 2024-04-14T14:04:54.280 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:04:54 smithi069 ceph-mon[24818]: from='mgr.24749 ' entity='mgr.x' 2024-04-14T14:04:54.281 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:04:54 smithi069 ceph-mon[24818]: from='mgr.24749 ' entity='mgr.x' 2024-04-14T14:04:54.281 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:04:54 smithi069 ceph-mon[24818]: from='mgr.24749 172.21.15.122:0/1892077634' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T14:04:54.281 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:04:54 smithi069 ceph-mon[24818]: from='mgr.24749 172.21.15.122:0/1892077634' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-14T14:04:54.281 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:04:54 smithi069 ceph-mon[24818]: pgmap v38: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 876 B/s rd, 0 op/s 2024-04-14T14:04:54.281 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:04:54 smithi069 ceph-mon[24818]: from='mgr.24749 ' entity='mgr.x' 2024-04-14T14:04:54.281 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:04:54 smithi069 ceph-mon[24818]: Reconfiguring iscsi.foo.smithi069.yuumtr (dependencies changed)... 2024-04-14T14:04:54.281 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:04:54 smithi069 ceph-mon[24818]: from='mgr.24749 172.21.15.122:0/1892077634' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi069.yuumtr", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-04-14T14:04:54.281 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:04:54 smithi069 ceph-mon[24818]: from='mgr.24749 ' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi069.yuumtr", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-04-14T14:04:54.281 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:04:54 smithi069 ceph-mon[24818]: from='mgr.24749 172.21.15.122:0/1892077634' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T14:04:54.281 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:04:54 smithi069 ceph-mon[24818]: Reconfiguring daemon iscsi.foo.smithi069.yuumtr on smithi069 2024-04-14T14:04:54.281 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:04:54 smithi069 ceph-mon[29472]: from='mgr.24749 ' entity='mgr.x' 2024-04-14T14:04:54.281 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:04:54 smithi069 ceph-mon[29472]: from='mgr.24749 ' entity='mgr.x' 2024-04-14T14:04:54.282 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:04:54 smithi069 ceph-mon[29472]: from='mgr.24749 ' entity='mgr.x' 2024-04-14T14:04:54.282 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:04:54 smithi069 ceph-mon[29472]: from='mgr.24749 ' entity='mgr.x' 2024-04-14T14:04:54.282 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:04:54 smithi069 ceph-mon[29472]: from='mgr.24749 172.21.15.122:0/1892077634' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T14:04:54.282 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:04:54 smithi069 ceph-mon[29472]: from='mgr.24749 172.21.15.122:0/1892077634' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-14T14:04:54.282 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:04:54 smithi069 ceph-mon[29472]: pgmap v38: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 876 B/s rd, 0 op/s 2024-04-14T14:04:54.282 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:04:54 smithi069 ceph-mon[29472]: from='mgr.24749 ' entity='mgr.x' 2024-04-14T14:04:54.282 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:04:54 smithi069 ceph-mon[29472]: Reconfiguring iscsi.foo.smithi069.yuumtr (dependencies changed)... 2024-04-14T14:04:54.282 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:04:54 smithi069 ceph-mon[29472]: from='mgr.24749 172.21.15.122:0/1892077634' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi069.yuumtr", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-04-14T14:04:54.282 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:04:54 smithi069 ceph-mon[29472]: from='mgr.24749 ' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi069.yuumtr", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-04-14T14:04:54.282 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:04:54 smithi069 ceph-mon[29472]: from='mgr.24749 172.21.15.122:0/1892077634' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T14:04:54.282 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:04:54 smithi069 ceph-mon[29472]: Reconfiguring daemon iscsi.foo.smithi069.yuumtr on smithi069 2024-04-14T14:04:54.595 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:04:54 smithi122 ceph-mon[25721]: from='mgr.24749 ' entity='mgr.x' 2024-04-14T14:04:54.595 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:04:54 smithi122 ceph-mon[25721]: from='mgr.24749 ' entity='mgr.x' 2024-04-14T14:04:54.595 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:04:54 smithi122 ceph-mon[25721]: from='mgr.24749 ' entity='mgr.x' 2024-04-14T14:04:54.595 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:04:54 smithi122 ceph-mon[25721]: from='mgr.24749 ' entity='mgr.x' 2024-04-14T14:04:54.595 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:04:54 smithi122 ceph-mon[25721]: from='mgr.24749 172.21.15.122:0/1892077634' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T14:04:54.595 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:04:54 smithi122 ceph-mon[25721]: from='mgr.24749 172.21.15.122:0/1892077634' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-14T14:04:54.596 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:04:54 smithi122 ceph-mon[25721]: pgmap v38: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 876 B/s rd, 0 op/s 2024-04-14T14:04:54.596 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:04:54 smithi122 ceph-mon[25721]: from='mgr.24749 ' entity='mgr.x' 2024-04-14T14:04:54.596 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:04:54 smithi122 ceph-mon[25721]: Reconfiguring iscsi.foo.smithi069.yuumtr (dependencies changed)... 2024-04-14T14:04:54.596 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:04:54 smithi122 ceph-mon[25721]: from='mgr.24749 172.21.15.122:0/1892077634' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi069.yuumtr", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-04-14T14:04:54.596 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:04:54 smithi122 ceph-mon[25721]: from='mgr.24749 ' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi069.yuumtr", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-04-14T14:04:54.596 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:04:54 smithi122 ceph-mon[25721]: from='mgr.24749 172.21.15.122:0/1892077634' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T14:04:54.596 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:04:54 smithi122 ceph-mon[25721]: Reconfiguring daemon iscsi.foo.smithi069.yuumtr on smithi069 2024-04-14T14:04:55.463 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:04:55 smithi069 ceph-mon[24818]: from='client.24985 -' entity='client.iscsi.foo.smithi069.yuumtr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T14:04:55.463 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:04:55 smithi069 ceph-mon[24818]: Health check failed: 1 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON) 2024-04-14T14:04:55.464 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:04:55 smithi069 ceph-mon[29472]: from='client.24985 -' entity='client.iscsi.foo.smithi069.yuumtr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T14:04:55.464 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:04:55 smithi069 ceph-mon[29472]: Health check failed: 1 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON) 2024-04-14T14:04:55.594 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:04:55 smithi122 ceph-mon[25721]: from='client.24985 -' entity='client.iscsi.foo.smithi069.yuumtr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T14:04:55.595 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:04:55 smithi122 ceph-mon[25721]: Health check failed: 1 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON) 2024-04-14T14:04:56.525 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:04:56 smithi069 ceph-mon[24818]: pgmap v39: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-04-14T14:04:56.526 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:04:56 smithi069 ceph-mon[29472]: pgmap v39: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-04-14T14:04:56.844 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:04:56 smithi122 ceph-mon[25721]: pgmap v39: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-04-14T14:04:56.845 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Apr 14 14:04:56 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-x[45487]: 2024-04-14T14:04:56.624+0000 7f461db9a700 -1 log_channel(cephadm) log [ERR] : cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi069.yuumtr ... 2024-04-14T14:04:56.845 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Apr 14 14:04:56 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-x[45487]: Creating ceph-iscsi config... 2024-04-14T14:04:56.845 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Apr 14 14:04:56 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-x[45487]: Write file: /var/lib/ceph/dc326fc2-fa65-11ee-bc8e-c7b262605968/iscsi.foo.smithi069.yuumtr/iscsi-gateway.cfg 2024-04-14T14:04:56.845 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Apr 14 14:04:56 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-x[45487]: Write file: /var/lib/ceph/dc326fc2-fa65-11ee-bc8e-c7b262605968/iscsi.foo.smithi069.yuumtr/tcmu-runner-entrypoint.sh 2024-04-14T14:04:56.845 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Apr 14 14:04:56 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-x[45487]: Non-zero exit code 1 from systemctl restart ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968@iscsi.foo.smithi069.yuumtr 2024-04-14T14:04:56.845 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Apr 14 14:04:56 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-x[45487]: systemctl: stderr Job for ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968@iscsi.foo.smithi069.yuumtr.service failed because the control process exited with error code. 2024-04-14T14:04:56.845 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Apr 14 14:04:56 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-x[45487]: systemctl: stderr See "systemctl status ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968@iscsi.foo.smithi069.yuumtr.service" and "journalctl -xeu ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968@iscsi.foo.smithi069.yuumtr.service" for details. 2024-04-14T14:04:56.846 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Apr 14 14:04:56 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-x[45487]: Traceback (most recent call last): 2024-04-14T14:04:56.846 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Apr 14 14:04:56 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-x[45487]: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-04-14T14:04:56.846 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Apr 14 14:04:56 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-x[45487]: return _run_code(code, main_globals, None, 2024-04-14T14:04:56.846 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Apr 14 14:04:56 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-x[45487]: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-04-14T14:04:56.846 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Apr 14 14:04:56 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-x[45487]: exec(code, run_globals) 2024-04-14T14:04:56.846 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Apr 14 14:04:56 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-x[45487]: File "/var/lib/ceph/dc326fc2-fa65-11ee-bc8e-c7b262605968/cephadm.de2a8faf91349910526b51da93d8bd42d57efdd2e93c32cd09ae64ecc1748ae9/__main__.py", line 5608, in 2024-04-14T14:04:56.846 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Apr 14 14:04:56 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-x[45487]: File "/var/lib/ceph/dc326fc2-fa65-11ee-bc8e-c7b262605968/cephadm.de2a8faf91349910526b51da93d8bd42d57efdd2e93c32cd09ae64ecc1748ae9/__main__.py", line 5596, in main 2024-04-14T14:04:56.846 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Apr 14 14:04:56 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-x[45487]: File "/var/lib/ceph/dc326fc2-fa65-11ee-bc8e-c7b262605968/cephadm.de2a8faf91349910526b51da93d8bd42d57efdd2e93c32cd09ae64ecc1748ae9/__main__.py", line 3096, in command_deploy_from 2024-04-14T14:04:56.846 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Apr 14 14:04:56 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-x[45487]: File "/var/lib/ceph/dc326fc2-fa65-11ee-bc8e-c7b262605968/cephadm.de2a8faf91349910526b51da93d8bd42d57efdd2e93c32cd09ae64ecc1748ae9/__main__.py", line 3131, in _common_deploy 2024-04-14T14:04:56.846 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Apr 14 14:04:56 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-x[45487]: File "/var/lib/ceph/dc326fc2-fa65-11ee-bc8e-c7b262605968/cephadm.de2a8faf91349910526b51da93d8bd42d57efdd2e93c32cd09ae64ecc1748ae9/__main__.py", line 3151, in _deploy_daemon_container 2024-04-14T14:04:56.846 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Apr 14 14:04:56 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-x[45487]: File "/var/lib/ceph/dc326fc2-fa65-11ee-bc8e-c7b262605968/cephadm.de2a8faf91349910526b51da93d8bd42d57efdd2e93c32cd09ae64ecc1748ae9/__main__.py", line 1123, in deploy_daemon 2024-04-14T14:04:56.847 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Apr 14 14:04:56 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-x[45487]: File "/var/lib/ceph/dc326fc2-fa65-11ee-bc8e-c7b262605968/cephadm.de2a8faf91349910526b51da93d8bd42d57efdd2e93c32cd09ae64ecc1748ae9/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-04-14T14:04:56.847 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Apr 14 14:04:56 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-x[45487]: RuntimeError: Failed command: systemctl restart ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968@iscsi.foo.smithi069.yuumtr: Job for ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968@iscsi.foo.smithi069.yuumtr.service failed because the control process exited with error code. 2024-04-14T14:04:56.847 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Apr 14 14:04:56 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-x[45487]: See "systemctl status ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968@iscsi.foo.smithi069.yuumtr.service" and "journalctl -xeu ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968@iscsi.foo.smithi069.yuumtr.service" for details. 2024-04-14T14:04:56.847 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Apr 14 14:04:56 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-x[45487]: Traceback (most recent call last): 2024-04-14T14:04:56.847 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Apr 14 14:04:56 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-x[45487]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1114, in _check_daemons 2024-04-14T14:04:56.847 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Apr 14 14:04:56 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-x[45487]: self.mgr._daemon_action(daemon_spec, action=action) 2024-04-14T14:04:56.847 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Apr 14 14:04:56 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-x[45487]: File "/usr/share/ceph/mgr/cephadm/module.py", line 2424, in _daemon_action 2024-04-14T14:04:56.848 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Apr 14 14:04:56 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-x[45487]: CephadmServe(self)._create_daemon(daemon_spec, reconfig=(action == 'reconfig'))) 2024-04-14T14:04:56.848 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Apr 14 14:04:56 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-x[45487]: File "/usr/share/ceph/mgr/cephadm/module.py", line 779, in wait_async 2024-04-14T14:04:56.848 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Apr 14 14:04:56 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-x[45487]: return self.event_loop.get_result(coro, timeout) 2024-04-14T14:04:56.848 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Apr 14 14:04:56 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-x[45487]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 136, in get_result 2024-04-14T14:04:56.848 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Apr 14 14:04:56 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-x[45487]: return future.result(timeout) 2024-04-14T14:04:56.848 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Apr 14 14:04:56 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-x[45487]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2024-04-14T14:04:56.848 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Apr 14 14:04:56 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-x[45487]: return self.__get_result() 2024-04-14T14:04:56.848 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Apr 14 14:04:56 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-x[45487]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2024-04-14T14:04:56.848 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Apr 14 14:04:56 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-x[45487]: raise self._exception 2024-04-14T14:04:56.848 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Apr 14 14:04:56 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-x[45487]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1374, in _create_daemon 2024-04-14T14:04:56.848 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Apr 14 14:04:56 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-x[45487]: config_blobs=daemon_spec.final_config, 2024-04-14T14:04:56.848 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Apr 14 14:04:56 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-x[45487]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1685, in _run_cephadm 2024-04-14T14:04:56.849 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Apr 14 14:04:56 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-x[45487]: f'cephadm exited with an error code: {code}, stderr: {err}') 2024-04-14T14:04:56.849 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Apr 14 14:04:56 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-x[45487]: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi069.yuumtr ... 2024-04-14T14:04:56.849 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Apr 14 14:04:56 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-x[45487]: Creating ceph-iscsi config... 2024-04-14T14:04:56.849 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Apr 14 14:04:56 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-x[45487]: Write file: /var/lib/ceph/dc326fc2-fa65-11ee-bc8e-c7b262605968/iscsi.foo.smithi069.yuumtr/iscsi-gateway.cfg 2024-04-14T14:04:56.849 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Apr 14 14:04:56 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-x[45487]: Write file: /var/lib/ceph/dc326fc2-fa65-11ee-bc8e-c7b262605968/iscsi.foo.smithi069.yuumtr/tcmu-runner-entrypoint.sh 2024-04-14T14:04:56.849 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Apr 14 14:04:56 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-x[45487]: Non-zero exit code 1 from systemctl restart ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968@iscsi.foo.smithi069.yuumtr 2024-04-14T14:04:56.849 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Apr 14 14:04:56 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-x[45487]: systemctl: stderr Job for ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968@iscsi.foo.smithi069.yuumtr.service failed because the control process exited with error code. 2024-04-14T14:04:56.849 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Apr 14 14:04:56 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-x[45487]: systemctl: stderr See "systemctl status ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968@iscsi.foo.smithi069.yuumtr.service" and "journalctl -xeu ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968@iscsi.foo.smithi069.yuumtr.service" for details. 2024-04-14T14:04:56.849 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Apr 14 14:04:56 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-x[45487]: Traceback (most recent call last): 2024-04-14T14:04:56.849 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Apr 14 14:04:56 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-x[45487]: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-04-14T14:04:56.849 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Apr 14 14:04:56 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-x[45487]: return _run_code(code, main_globals, None, 2024-04-14T14:04:56.849 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Apr 14 14:04:56 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-x[45487]: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-04-14T14:04:56.849 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Apr 14 14:04:56 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-x[45487]: exec(code, run_globals) 2024-04-14T14:04:56.850 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Apr 14 14:04:56 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-x[45487]: File "/var/lib/ceph/dc326fc2-fa65-11ee-bc8e-c7b262605968/cephadm.de2a8faf91349910526b51da93d8bd42d57efdd2e93c32cd09ae64ecc1748ae9/__main__.py", line 5608, in 2024-04-14T14:04:56.850 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Apr 14 14:04:56 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-x[45487]: File "/var/lib/ceph/dc326fc2-fa65-11ee-bc8e-c7b262605968/cephadm.de2a8faf91349910526b51da93d8bd42d57efdd2e93c32cd09ae64ecc1748ae9/__main__.py", line 5596, in main 2024-04-14T14:04:56.850 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Apr 14 14:04:56 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-x[45487]: File "/var/lib/ceph/dc326fc2-fa65-11ee-bc8e-c7b262605968/cephadm.de2a8faf91349910526b51da93d8bd42d57efdd2e93c32cd09ae64ecc1748ae9/__main__.py", line 3096, in command_deploy_from 2024-04-14T14:04:56.850 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Apr 14 14:04:56 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-x[45487]: File "/var/lib/ceph/dc326fc2-fa65-11ee-bc8e-c7b262605968/cephadm.de2a8faf91349910526b51da93d8bd42d57efdd2e93c32cd09ae64ecc1748ae9/__main__.py", line 3131, in _common_deploy 2024-04-14T14:04:56.850 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Apr 14 14:04:56 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-x[45487]: File "/var/lib/ceph/dc326fc2-fa65-11ee-bc8e-c7b262605968/cephadm.de2a8faf91349910526b51da93d8bd42d57efdd2e93c32cd09ae64ecc1748ae9/__main__.py", line 3151, in _deploy_daemon_container 2024-04-14T14:04:56.850 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Apr 14 14:04:56 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-x[45487]: File "/var/lib/ceph/dc326fc2-fa65-11ee-bc8e-c7b262605968/cephadm.de2a8faf91349910526b51da93d8bd42d57efdd2e93c32cd09ae64ecc1748ae9/__main__.py", line 1123, in deploy_daemon 2024-04-14T14:04:56.850 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Apr 14 14:04:56 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-x[45487]: File "/var/lib/ceph/dc326fc2-fa65-11ee-bc8e-c7b262605968/cephadm.de2a8faf91349910526b51da93d8bd42d57efdd2e93c32cd09ae64ecc1748ae9/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-04-14T14:04:56.850 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Apr 14 14:04:56 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-x[45487]: RuntimeError: Failed command: systemctl restart ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968@iscsi.foo.smithi069.yuumtr: Job for ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968@iscsi.foo.smithi069.yuumtr.service failed because the control process exited with error code. 2024-04-14T14:04:56.851 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Apr 14 14:04:56 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-x[45487]: See "systemctl status ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968@iscsi.foo.smithi069.yuumtr.service" and "journalctl -xeu ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968@iscsi.foo.smithi069.yuumtr.service" for details. 2024-04-14T14:04:57.817 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:04:57 smithi069 ceph-mon[24818]: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi069.yuumtr ... 2024-04-14T14:04:57.818 INFO:journalctl@ceph.mon.a.smithi069.stdout: Creating ceph-iscsi config... 2024-04-14T14:04:57.818 INFO:journalctl@ceph.mon.a.smithi069.stdout: Write file: /var/lib/ceph/dc326fc2-fa65-11ee-bc8e-c7b262605968/iscsi.foo.smithi069.yuumtr/iscsi-gateway.cfg 2024-04-14T14:04:57.818 INFO:journalctl@ceph.mon.a.smithi069.stdout: Write file: /var/lib/ceph/dc326fc2-fa65-11ee-bc8e-c7b262605968/iscsi.foo.smithi069.yuumtr/tcmu-runner-entrypoint.sh 2024-04-14T14:04:57.818 INFO:journalctl@ceph.mon.a.smithi069.stdout: Non-zero exit code 1 from systemctl restart ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968@iscsi.foo.smithi069.yuumtr 2024-04-14T14:04:57.818 INFO:journalctl@ceph.mon.a.smithi069.stdout: systemctl: stderr Job for ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968@iscsi.foo.smithi069.yuumtr.service failed because the control process exited with error code. 2024-04-14T14:04:57.818 INFO:journalctl@ceph.mon.a.smithi069.stdout: systemctl: stderr See "systemctl status ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968@iscsi.foo.smithi069.yuumtr.service" and "journalctl -xeu ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968@iscsi.foo.smithi069.yuumtr.service" for details. 2024-04-14T14:04:57.818 INFO:journalctl@ceph.mon.a.smithi069.stdout: Traceback (most recent call last): 2024-04-14T14:04:57.818 INFO:journalctl@ceph.mon.a.smithi069.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-04-14T14:04:57.818 INFO:journalctl@ceph.mon.a.smithi069.stdout: return _run_code(code, main_globals, None, 2024-04-14T14:04:57.818 INFO:journalctl@ceph.mon.a.smithi069.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-04-14T14:04:57.818 INFO:journalctl@ceph.mon.a.smithi069.stdout: exec(code, run_globals) 2024-04-14T14:04:57.819 INFO:journalctl@ceph.mon.a.smithi069.stdout: File "/var/lib/ceph/dc326fc2-fa65-11ee-bc8e-c7b262605968/cephadm.de2a8faf91349910526b51da93d8bd42d57efdd2e93c32cd09ae64ecc1748ae9/__main__.py", line 5608, in 2024-04-14T14:04:57.819 INFO:journalctl@ceph.mon.a.smithi069.stdout: File "/var/lib/ceph/dc326fc2-fa65-11ee-bc8e-c7b262605968/cephadm.de2a8faf91349910526b51da93d8bd42d57efdd2e93c32cd09ae64ecc1748ae9/__main__.py", line 5596, in main 2024-04-14T14:04:57.819 INFO:journalctl@ceph.mon.a.smithi069.stdout: File "/var/lib/ceph/dc326fc2-fa65-11ee-bc8e-c7b262605968/cephadm.de2a8faf91349910526b51da93d8bd42d57efdd2e93c32cd09ae64ecc1748ae9/__main__.py", line 3096, in command_deploy_from 2024-04-14T14:04:57.819 INFO:journalctl@ceph.mon.a.smithi069.stdout: File "/var/lib/ceph/dc326fc2-fa65-11ee-bc8e-c7b262605968/cephadm.de2a8faf91349910526b51da93d8bd42d57efdd2e93c32cd09ae64ecc1748ae9/__main__.py", line 3131, in _common_deploy 2024-04-14T14:04:57.819 INFO:journalctl@ceph.mon.a.smithi069.stdout: File "/var/lib/ceph/dc326fc2-fa65-11ee-bc8e-c7b262605968/cephadm.de2a8faf91349910526b51da93d8bd42d57efdd2e93c32cd09ae64ecc1748ae9/__main__.py", line 3151, in _deploy_daemon_container 2024-04-14T14:04:57.819 INFO:journalctl@ceph.mon.a.smithi069.stdout: File "/var/lib/ceph/dc326fc2-fa65-11ee-bc8e-c7b262605968/cephadm.de2a8faf91349910526b51da93d8bd42d57efdd2e93c32cd09ae64ecc1748ae9/__main__.py", line 1123, in deploy_daemon 2024-04-14T14:04:57.819 INFO:journalctl@ceph.mon.a.smithi069.stdout: File "/var/lib/ceph/dc326fc2-fa65-11ee-bc8e-c7b262605968/cephadm.de2a8faf91349910526b51da93d8bd42d57efdd2e93c32cd09ae64ecc1748ae9/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-04-14T14:04:57.819 INFO:journalctl@ceph.mon.a.smithi069.stdout: RuntimeError: Failed command: systemctl restart ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968@iscsi.foo.smithi069.yuumtr: Job for ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968@iscsi.foo.smithi069.yuumtr.service failed because the control process exited with error code. 2024-04-14T14:04:57.819 INFO:journalctl@ceph.mon.a.smithi069.stdout: See "systemctl status ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968@iscsi.foo.smithi069.yuumtr.service" and "journalctl -xeu ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968@iscsi.foo.smithi069.yuumtr.service" for details. 2024-04-14T14:04:57.819 INFO:journalctl@ceph.mon.a.smithi069.stdout: Traceback (most recent call last): 2024-04-14T14:04:57.819 INFO:journalctl@ceph.mon.a.smithi069.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1114, in _check_daemons 2024-04-14T14:04:57.819 INFO:journalctl@ceph.mon.a.smithi069.stdout: self.mgr._daemon_action(daemon_spec, action=action) 2024-04-14T14:04:57.819 INFO:journalctl@ceph.mon.a.smithi069.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 2424, in _daemon_action 2024-04-14T14:04:57.820 INFO:journalctl@ceph.mon.a.smithi069.stdout: CephadmServe(self)._create_daemon(daemon_spec, reconfig=(action == 'reconfig'))) 2024-04-14T14:04:57.820 INFO:journalctl@ceph.mon.a.smithi069.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 779, in wait_async 2024-04-14T14:04:57.820 INFO:journalctl@ceph.mon.a.smithi069.stdout: return self.event_loop.get_result(coro, timeout) 2024-04-14T14:04:57.820 INFO:journalctl@ceph.mon.a.smithi069.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 136, in get_result 2024-04-14T14:04:57.820 INFO:journalctl@ceph.mon.a.smithi069.stdout: return future.result(timeout) 2024-04-14T14:04:57.820 INFO:journalctl@ceph.mon.a.smithi069.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2024-04-14T14:04:57.820 INFO:journalctl@ceph.mon.a.smithi069.stdout: return self.__get_result() 2024-04-14T14:04:57.820 INFO:journalctl@ceph.mon.a.smithi069.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2024-04-14T14:04:57.820 INFO:journalctl@ceph.mon.a.smithi069.stdout: raise self._exception 2024-04-14T14:04:57.820 INFO:journalctl@ceph.mon.a.smithi069.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1374, in _create_daemon 2024-04-14T14:04:57.820 INFO:journalctl@ceph.mon.a.smithi069.stdout: config_blobs=daemon_spec.final_config, 2024-04-14T14:04:57.820 INFO:journalctl@ceph.mon.a.smithi069.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1685, in _run_cephadm 2024-04-14T14:04:57.820 INFO:journalctl@ceph.mon.a.smithi069.stdout: f'cephadm exited with an error code: {code}, stderr: {err}') 2024-04-14T14:04:57.820 INFO:journalctl@ceph.mon.a.smithi069.stdout: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi069.yuumtr ... 2024-04-14T14:04:57.820 INFO:journalctl@ceph.mon.a.smithi069.stdout: Creating ceph-iscsi config... 2024-04-14T14:04:57.821 INFO:journalctl@ceph.mon.a.smithi069.stdout: Write file: /var/lib/ceph/dc326fc2-fa65-11ee-bc8e-c7b262605968/iscsi.foo.smithi069.yuumtr/iscsi-gateway.cfg 2024-04-14T14:04:57.821 INFO:journalctl@ceph.mon.a.smithi069.stdout: Write file: /var/lib/ceph/dc326fc2-fa65-11ee-bc8e-c7b262605968/iscsi.foo.smithi069.yuumtr/tcmu-runner-entrypoint.sh 2024-04-14T14:04:57.821 INFO:journalctl@ceph.mon.a.smithi069.stdout: Non-zero exit code 1 from systemctl restart ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968@iscsi.foo.smithi069.yuumtr 2024-04-14T14:04:57.821 INFO:journalctl@ceph.mon.a.smithi069.stdout: systemctl: stderr Job for ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968@iscsi.foo.smithi069.yuumtr.service failed because the control process exited with error code. 2024-04-14T14:04:57.821 INFO:journalctl@ceph.mon.a.smithi069.stdout: systemctl: stderr See "systemctl status ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968@iscsi.foo.smithi069.yuumtr.service" and "journalctl -xeu ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968@iscsi.foo.smithi069.yuumtr.service" for details. 2024-04-14T14:04:57.821 INFO:journalctl@ceph.mon.a.smithi069.stdout: Traceback (most recent call last): 2024-04-14T14:04:57.821 INFO:journalctl@ceph.mon.a.smithi069.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-04-14T14:04:57.821 INFO:journalctl@ceph.mon.a.smithi069.stdout: return _run_code(code, main_globals, None, 2024-04-14T14:04:57.821 INFO:journalctl@ceph.mon.a.smithi069.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-04-14T14:04:57.821 INFO:journalctl@ceph.mon.a.smithi069.stdout: exec(code, run_globals) 2024-04-14T14:04:57.821 INFO:journalctl@ceph.mon.a.smithi069.stdout: File "/var/lib/ceph/dc326fc2-fa65-11ee-bc8e-c7b262605968/cephadm.de2a8faf91349910526b51da93d8bd42d57efdd2e93c32cd09ae64ecc1748ae9/__main__.py", line 5608, in 2024-04-14T14:04:57.821 INFO:journalctl@ceph.mon.a.smithi069.stdout: File "/var/lib/ceph/dc326fc2-fa65-11ee-bc8e-c7b262605968/cephadm.de2a8faf91349910526b51da93d8bd42d57efdd2e93c32cd09ae64ecc1748ae9/__main__.py", line 5596, in main 2024-04-14T14:04:57.821 INFO:journalctl@ceph.mon.a.smithi069.stdout: File "/var/lib/ceph/dc326fc2-fa65-11ee-bc8e-c7b262605968/cephadm.de2a8faf91349910526b51da93d8bd42d57efdd2e93c32cd09ae64ecc1748ae9/__main__.py", line 3096, in command_deploy_from 2024-04-14T14:04:57.821 INFO:journalctl@ceph.mon.a.smithi069.stdout: File "/var/lib/ceph/dc326fc2-fa65-11ee-bc8e-c7b262605968/cephadm.de2a8faf91349910526b51da93d8bd42d57efdd2e93c32cd09ae64ecc1748ae9/__main__.py", line 3131, in _common_deploy 2024-04-14T14:04:57.822 INFO:journalctl@ceph.mon.a.smithi069.stdout: File "/var/lib/ceph/dc326fc2-fa65-11ee-bc8e-c7b262605968/cephadm.de2a8faf91349910526b51da93d8bd42d57efdd2e93c32cd09ae64ecc1748ae9/__main__.py", line 3151, in _deploy_daemon_container 2024-04-14T14:04:57.822 INFO:journalctl@ceph.mon.a.smithi069.stdout: File "/var/lib/ceph/dc326fc2-fa65-11ee-bc8e-c7b262605968/cephadm.de2a8faf91349910526b51da93d8bd42d57efdd2e93c32cd09ae64ecc1748ae9/__main__.py", line 1123, in deploy_daemon 2024-04-14T14:04:57.822 INFO:journalctl@ceph.mon.a.smithi069.stdout: File "/var/lib/ceph/dc326fc2-fa65-11ee-bc8e-c7b262605968/cephadm.de2a8faf91349910526b51da93d8bd42d57efdd2e93c32cd09ae64ecc1748ae9/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-04-14T14:04:57.822 INFO:journalctl@ceph.mon.a.smithi069.stdout: RuntimeError: Failed command: systemctl restart ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968@iscsi.foo.smithi069.yuumtr: Job for ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968@iscsi.foo.smithi069.yuumtr.service failed because the control process exited with error code. 2024-04-14T14:04:57.822 INFO:journalctl@ceph.mon.a.smithi069.stdout: See "systemctl status ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968@iscsi.foo.smithi069.yuumtr.service" and "journalctl -xeu ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968@iscsi.foo.smithi069.yuumtr.service" for details. 2024-04-14T14:04:57.822 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:04:57 smithi069 ceph-mon[29472]: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi069.yuumtr ... 2024-04-14T14:04:57.822 INFO:journalctl@ceph.mon.c.smithi069.stdout: Creating ceph-iscsi config... 2024-04-14T14:04:57.822 INFO:journalctl@ceph.mon.c.smithi069.stdout: Write file: /var/lib/ceph/dc326fc2-fa65-11ee-bc8e-c7b262605968/iscsi.foo.smithi069.yuumtr/iscsi-gateway.cfg 2024-04-14T14:04:57.822 INFO:journalctl@ceph.mon.c.smithi069.stdout: Write file: /var/lib/ceph/dc326fc2-fa65-11ee-bc8e-c7b262605968/iscsi.foo.smithi069.yuumtr/tcmu-runner-entrypoint.sh 2024-04-14T14:04:57.822 INFO:journalctl@ceph.mon.c.smithi069.stdout: Non-zero exit code 1 from systemctl restart ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968@iscsi.foo.smithi069.yuumtr 2024-04-14T14:04:57.822 INFO:journalctl@ceph.mon.c.smithi069.stdout: systemctl: stderr Job for ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968@iscsi.foo.smithi069.yuumtr.service failed because the control process exited with error code. 2024-04-14T14:04:57.823 INFO:journalctl@ceph.mon.c.smithi069.stdout: systemctl: stderr See "systemctl status ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968@iscsi.foo.smithi069.yuumtr.service" and "journalctl -xeu ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968@iscsi.foo.smithi069.yuumtr.service" for details. 2024-04-14T14:04:57.823 INFO:journalctl@ceph.mon.c.smithi069.stdout: Traceback (most recent call last): 2024-04-14T14:04:57.823 INFO:journalctl@ceph.mon.c.smithi069.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-04-14T14:04:57.823 INFO:journalctl@ceph.mon.c.smithi069.stdout: return _run_code(code, main_globals, None, 2024-04-14T14:04:57.823 INFO:journalctl@ceph.mon.c.smithi069.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-04-14T14:04:57.823 INFO:journalctl@ceph.mon.c.smithi069.stdout: exec(code, run_globals) 2024-04-14T14:04:57.823 INFO:journalctl@ceph.mon.c.smithi069.stdout: File "/var/lib/ceph/dc326fc2-fa65-11ee-bc8e-c7b262605968/cephadm.de2a8faf91349910526b51da93d8bd42d57efdd2e93c32cd09ae64ecc1748ae9/__main__.py", line 5608, in 2024-04-14T14:04:57.823 INFO:journalctl@ceph.mon.c.smithi069.stdout: File "/var/lib/ceph/dc326fc2-fa65-11ee-bc8e-c7b262605968/cephadm.de2a8faf91349910526b51da93d8bd42d57efdd2e93c32cd09ae64ecc1748ae9/__main__.py", line 5596, in main 2024-04-14T14:04:57.823 INFO:journalctl@ceph.mon.c.smithi069.stdout: File "/var/lib/ceph/dc326fc2-fa65-11ee-bc8e-c7b262605968/cephadm.de2a8faf91349910526b51da93d8bd42d57efdd2e93c32cd09ae64ecc1748ae9/__main__.py", line 3096, in command_deploy_from 2024-04-14T14:04:57.823 INFO:journalctl@ceph.mon.c.smithi069.stdout: File "/var/lib/ceph/dc326fc2-fa65-11ee-bc8e-c7b262605968/cephadm.de2a8faf91349910526b51da93d8bd42d57efdd2e93c32cd09ae64ecc1748ae9/__main__.py", line 3131, in _common_deploy 2024-04-14T14:04:57.823 INFO:journalctl@ceph.mon.c.smithi069.stdout: File "/var/lib/ceph/dc326fc2-fa65-11ee-bc8e-c7b262605968/cephadm.de2a8faf91349910526b51da93d8bd42d57efdd2e93c32cd09ae64ecc1748ae9/__main__.py", line 3151, in _deploy_daemon_container 2024-04-14T14:04:57.823 INFO:journalctl@ceph.mon.c.smithi069.stdout: File "/var/lib/ceph/dc326fc2-fa65-11ee-bc8e-c7b262605968/cephadm.de2a8faf91349910526b51da93d8bd42d57efdd2e93c32cd09ae64ecc1748ae9/__main__.py", line 1123, in deploy_daemon 2024-04-14T14:04:57.823 INFO:journalctl@ceph.mon.c.smithi069.stdout: File "/var/lib/ceph/dc326fc2-fa65-11ee-bc8e-c7b262605968/cephadm.de2a8faf91349910526b51da93d8bd42d57efdd2e93c32cd09ae64ecc1748ae9/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-04-14T14:04:57.823 INFO:journalctl@ceph.mon.c.smithi069.stdout: RuntimeError: Failed command: systemctl restart ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968@iscsi.foo.smithi069.yuumtr: Job for ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968@iscsi.foo.smithi069.yuumtr.service failed because the control process exited with error code. 2024-04-14T14:04:57.824 INFO:journalctl@ceph.mon.c.smithi069.stdout: See "systemctl status ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968@iscsi.foo.smithi069.yuumtr.service" and "journalctl -xeu ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968@iscsi.foo.smithi069.yuumtr.service" for details. 2024-04-14T14:04:57.824 INFO:journalctl@ceph.mon.c.smithi069.stdout: Traceback (most recent call last): 2024-04-14T14:04:57.824 INFO:journalctl@ceph.mon.c.smithi069.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1114, in _check_daemons 2024-04-14T14:04:57.824 INFO:journalctl@ceph.mon.c.smithi069.stdout: self.mgr._daemon_action(daemon_spec, action=action) 2024-04-14T14:04:57.824 INFO:journalctl@ceph.mon.c.smithi069.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 2424, in _daemon_action 2024-04-14T14:04:57.824 INFO:journalctl@ceph.mon.c.smithi069.stdout: CephadmServe(self)._create_daemon(daemon_spec, reconfig=(action == 'reconfig'))) 2024-04-14T14:04:57.824 INFO:journalctl@ceph.mon.c.smithi069.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 779, in wait_async 2024-04-14T14:04:57.824 INFO:journalctl@ceph.mon.c.smithi069.stdout: return self.event_loop.get_result(coro, timeout) 2024-04-14T14:04:57.824 INFO:journalctl@ceph.mon.c.smithi069.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 136, in get_result 2024-04-14T14:04:57.824 INFO:journalctl@ceph.mon.c.smithi069.stdout: return future.result(timeout) 2024-04-14T14:04:57.824 INFO:journalctl@ceph.mon.c.smithi069.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2024-04-14T14:04:57.824 INFO:journalctl@ceph.mon.c.smithi069.stdout: return self.__get_result() 2024-04-14T14:04:57.824 INFO:journalctl@ceph.mon.c.smithi069.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2024-04-14T14:04:57.825 INFO:journalctl@ceph.mon.c.smithi069.stdout: raise self._exception 2024-04-14T14:04:57.825 INFO:journalctl@ceph.mon.c.smithi069.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1374, in _create_daemon 2024-04-14T14:04:57.825 INFO:journalctl@ceph.mon.c.smithi069.stdout: config_blobs=daemon_spec.final_config, 2024-04-14T14:04:57.825 INFO:journalctl@ceph.mon.c.smithi069.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1685, in _run_cephadm 2024-04-14T14:04:57.825 INFO:journalctl@ceph.mon.c.smithi069.stdout: f'cephadm exited with an error code: {code}, stderr: {err}') 2024-04-14T14:04:57.825 INFO:journalctl@ceph.mon.c.smithi069.stdout: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi069.yuumtr ... 2024-04-14T14:04:57.825 INFO:journalctl@ceph.mon.c.smithi069.stdout: Creating ceph-iscsi config... 2024-04-14T14:04:57.825 INFO:journalctl@ceph.mon.c.smithi069.stdout: Write file: /var/lib/ceph/dc326fc2-fa65-11ee-bc8e-c7b262605968/iscsi.foo.smithi069.yuumtr/iscsi-gateway.cfg 2024-04-14T14:04:57.825 INFO:journalctl@ceph.mon.c.smithi069.stdout: Write file: /var/lib/ceph/dc326fc2-fa65-11ee-bc8e-c7b262605968/iscsi.foo.smithi069.yuumtr/tcmu-runner-entrypoint.sh 2024-04-14T14:04:57.825 INFO:journalctl@ceph.mon.c.smithi069.stdout: Non-zero exit code 1 from systemctl restart ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968@iscsi.foo.smithi069.yuumtr 2024-04-14T14:04:57.825 INFO:journalctl@ceph.mon.c.smithi069.stdout: systemctl: stderr Job for ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968@iscsi.foo.smithi069.yuumtr.service failed because the control process exited with error code. 2024-04-14T14:04:57.825 INFO:journalctl@ceph.mon.c.smithi069.stdout: systemctl: stderr See "systemctl status ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968@iscsi.foo.smithi069.yuumtr.service" and "journalctl -xeu ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968@iscsi.foo.smithi069.yuumtr.service" for details. 2024-04-14T14:04:57.825 INFO:journalctl@ceph.mon.c.smithi069.stdout: Traceback (most recent call last): 2024-04-14T14:04:57.825 INFO:journalctl@ceph.mon.c.smithi069.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-04-14T14:04:57.826 INFO:journalctl@ceph.mon.c.smithi069.stdout: return _run_code(code, main_globals, None, 2024-04-14T14:04:57.826 INFO:journalctl@ceph.mon.c.smithi069.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-04-14T14:04:57.826 INFO:journalctl@ceph.mon.c.smithi069.stdout: exec(code, run_globals) 2024-04-14T14:04:57.826 INFO:journalctl@ceph.mon.c.smithi069.stdout: File "/var/lib/ceph/dc326fc2-fa65-11ee-bc8e-c7b262605968/cephadm.de2a8faf91349910526b51da93d8bd42d57efdd2e93c32cd09ae64ecc1748ae9/__main__.py", line 5608, in 2024-04-14T14:04:57.826 INFO:journalctl@ceph.mon.c.smithi069.stdout: File "/var/lib/ceph/dc326fc2-fa65-11ee-bc8e-c7b262605968/cephadm.de2a8faf91349910526b51da93d8bd42d57efdd2e93c32cd09ae64ecc1748ae9/__main__.py", line 5596, in main 2024-04-14T14:04:57.826 INFO:journalctl@ceph.mon.c.smithi069.stdout: File "/var/lib/ceph/dc326fc2-fa65-11ee-bc8e-c7b262605968/cephadm.de2a8faf91349910526b51da93d8bd42d57efdd2e93c32cd09ae64ecc1748ae9/__main__.py", line 3096, in command_deploy_from 2024-04-14T14:04:57.826 INFO:journalctl@ceph.mon.c.smithi069.stdout: File "/var/lib/ceph/dc326fc2-fa65-11ee-bc8e-c7b262605968/cephadm.de2a8faf91349910526b51da93d8bd42d57efdd2e93c32cd09ae64ecc1748ae9/__main__.py", line 3131, in _common_deploy 2024-04-14T14:04:57.826 INFO:journalctl@ceph.mon.c.smithi069.stdout: File "/var/lib/ceph/dc326fc2-fa65-11ee-bc8e-c7b262605968/cephadm.de2a8faf91349910526b51da93d8bd42d57efdd2e93c32cd09ae64ecc1748ae9/__main__.py", line 3151, in _deploy_daemon_container 2024-04-14T14:04:57.826 INFO:journalctl@ceph.mon.c.smithi069.stdout: File "/var/lib/ceph/dc326fc2-fa65-11ee-bc8e-c7b262605968/cephadm.de2a8faf91349910526b51da93d8bd42d57efdd2e93c32cd09ae64ecc1748ae9/__main__.py", line 1123, in deploy_daemon 2024-04-14T14:04:57.826 INFO:journalctl@ceph.mon.c.smithi069.stdout: File "/var/lib/ceph/dc326fc2-fa65-11ee-bc8e-c7b262605968/cephadm.de2a8faf91349910526b51da93d8bd42d57efdd2e93c32cd09ae64ecc1748ae9/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-04-14T14:04:57.826 INFO:journalctl@ceph.mon.c.smithi069.stdout: RuntimeError: Failed command: systemctl restart ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968@iscsi.foo.smithi069.yuumtr: Job for ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968@iscsi.foo.smithi069.yuumtr.service failed because the control process exited with error code. 2024-04-14T14:04:57.826 INFO:journalctl@ceph.mon.c.smithi069.stdout: See "systemctl status ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968@iscsi.foo.smithi069.yuumtr.service" and "journalctl -xeu ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968@iscsi.foo.smithi069.yuumtr.service" for details. 2024-04-14T14:04:57.844 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:04:57 smithi122 ceph-mon[25721]: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi069.yuumtr ... 2024-04-14T14:04:57.845 INFO:journalctl@ceph.mon.b.smithi122.stdout: Creating ceph-iscsi config... 2024-04-14T14:04:57.845 INFO:journalctl@ceph.mon.b.smithi122.stdout: Write file: /var/lib/ceph/dc326fc2-fa65-11ee-bc8e-c7b262605968/iscsi.foo.smithi069.yuumtr/iscsi-gateway.cfg 2024-04-14T14:04:57.845 INFO:journalctl@ceph.mon.b.smithi122.stdout: Write file: /var/lib/ceph/dc326fc2-fa65-11ee-bc8e-c7b262605968/iscsi.foo.smithi069.yuumtr/tcmu-runner-entrypoint.sh 2024-04-14T14:04:57.845 INFO:journalctl@ceph.mon.b.smithi122.stdout: Non-zero exit code 1 from systemctl restart ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968@iscsi.foo.smithi069.yuumtr 2024-04-14T14:04:57.845 INFO:journalctl@ceph.mon.b.smithi122.stdout: systemctl: stderr Job for ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968@iscsi.foo.smithi069.yuumtr.service failed because the control process exited with error code. 2024-04-14T14:04:57.845 INFO:journalctl@ceph.mon.b.smithi122.stdout: systemctl: stderr See "systemctl status ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968@iscsi.foo.smithi069.yuumtr.service" and "journalctl -xeu ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968@iscsi.foo.smithi069.yuumtr.service" for details. 2024-04-14T14:04:57.845 INFO:journalctl@ceph.mon.b.smithi122.stdout: Traceback (most recent call last): 2024-04-14T14:04:57.845 INFO:journalctl@ceph.mon.b.smithi122.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-04-14T14:04:57.845 INFO:journalctl@ceph.mon.b.smithi122.stdout: return _run_code(code, main_globals, None, 2024-04-14T14:04:57.845 INFO:journalctl@ceph.mon.b.smithi122.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-04-14T14:04:57.845 INFO:journalctl@ceph.mon.b.smithi122.stdout: exec(code, run_globals) 2024-04-14T14:04:57.845 INFO:journalctl@ceph.mon.b.smithi122.stdout: File "/var/lib/ceph/dc326fc2-fa65-11ee-bc8e-c7b262605968/cephadm.de2a8faf91349910526b51da93d8bd42d57efdd2e93c32cd09ae64ecc1748ae9/__main__.py", line 5608, in 2024-04-14T14:04:57.845 INFO:journalctl@ceph.mon.b.smithi122.stdout: File "/var/lib/ceph/dc326fc2-fa65-11ee-bc8e-c7b262605968/cephadm.de2a8faf91349910526b51da93d8bd42d57efdd2e93c32cd09ae64ecc1748ae9/__main__.py", line 5596, in main 2024-04-14T14:04:57.845 INFO:journalctl@ceph.mon.b.smithi122.stdout: File "/var/lib/ceph/dc326fc2-fa65-11ee-bc8e-c7b262605968/cephadm.de2a8faf91349910526b51da93d8bd42d57efdd2e93c32cd09ae64ecc1748ae9/__main__.py", line 3096, in command_deploy_from 2024-04-14T14:04:57.846 INFO:journalctl@ceph.mon.b.smithi122.stdout: File "/var/lib/ceph/dc326fc2-fa65-11ee-bc8e-c7b262605968/cephadm.de2a8faf91349910526b51da93d8bd42d57efdd2e93c32cd09ae64ecc1748ae9/__main__.py", line 3131, in _common_deploy 2024-04-14T14:04:57.846 INFO:journalctl@ceph.mon.b.smithi122.stdout: File "/var/lib/ceph/dc326fc2-fa65-11ee-bc8e-c7b262605968/cephadm.de2a8faf91349910526b51da93d8bd42d57efdd2e93c32cd09ae64ecc1748ae9/__main__.py", line 3151, in _deploy_daemon_container 2024-04-14T14:04:57.846 INFO:journalctl@ceph.mon.b.smithi122.stdout: File "/var/lib/ceph/dc326fc2-fa65-11ee-bc8e-c7b262605968/cephadm.de2a8faf91349910526b51da93d8bd42d57efdd2e93c32cd09ae64ecc1748ae9/__main__.py", line 1123, in deploy_daemon 2024-04-14T14:04:57.846 INFO:journalctl@ceph.mon.b.smithi122.stdout: File "/var/lib/ceph/dc326fc2-fa65-11ee-bc8e-c7b262605968/cephadm.de2a8faf91349910526b51da93d8bd42d57efdd2e93c32cd09ae64ecc1748ae9/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-04-14T14:04:57.846 INFO:journalctl@ceph.mon.b.smithi122.stdout: RuntimeError: Failed command: systemctl restart ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968@iscsi.foo.smithi069.yuumtr: Job for ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968@iscsi.foo.smithi069.yuumtr.service failed because the control process exited with error code. 2024-04-14T14:04:57.846 INFO:journalctl@ceph.mon.b.smithi122.stdout: See "systemctl status ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968@iscsi.foo.smithi069.yuumtr.service" and "journalctl -xeu ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968@iscsi.foo.smithi069.yuumtr.service" for details. 2024-04-14T14:04:57.846 INFO:journalctl@ceph.mon.b.smithi122.stdout: Traceback (most recent call last): 2024-04-14T14:04:57.846 INFO:journalctl@ceph.mon.b.smithi122.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1114, in _check_daemons 2024-04-14T14:04:57.846 INFO:journalctl@ceph.mon.b.smithi122.stdout: self.mgr._daemon_action(daemon_spec, action=action) 2024-04-14T14:04:57.846 INFO:journalctl@ceph.mon.b.smithi122.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 2424, in _daemon_action 2024-04-14T14:04:57.846 INFO:journalctl@ceph.mon.b.smithi122.stdout: CephadmServe(self)._create_daemon(daemon_spec, reconfig=(action == 'reconfig'))) 2024-04-14T14:04:57.846 INFO:journalctl@ceph.mon.b.smithi122.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 779, in wait_async 2024-04-14T14:04:57.846 INFO:journalctl@ceph.mon.b.smithi122.stdout: return self.event_loop.get_result(coro, timeout) 2024-04-14T14:04:57.846 INFO:journalctl@ceph.mon.b.smithi122.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 136, in get_result 2024-04-14T14:04:57.847 INFO:journalctl@ceph.mon.b.smithi122.stdout: return future.result(timeout) 2024-04-14T14:04:57.847 INFO:journalctl@ceph.mon.b.smithi122.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2024-04-14T14:04:57.847 INFO:journalctl@ceph.mon.b.smithi122.stdout: return self.__get_result() 2024-04-14T14:04:57.847 INFO:journalctl@ceph.mon.b.smithi122.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2024-04-14T14:04:57.847 INFO:journalctl@ceph.mon.b.smithi122.stdout: raise self._exception 2024-04-14T14:04:57.847 INFO:journalctl@ceph.mon.b.smithi122.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1374, in _create_daemon 2024-04-14T14:04:57.847 INFO:journalctl@ceph.mon.b.smithi122.stdout: config_blobs=daemon_spec.final_config, 2024-04-14T14:04:57.847 INFO:journalctl@ceph.mon.b.smithi122.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1685, in _run_cephadm 2024-04-14T14:04:57.847 INFO:journalctl@ceph.mon.b.smithi122.stdout: f'cephadm exited with an error code: {code}, stderr: {err}') 2024-04-14T14:04:57.847 INFO:journalctl@ceph.mon.b.smithi122.stdout: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi069.yuumtr ... 2024-04-14T14:04:57.847 INFO:journalctl@ceph.mon.b.smithi122.stdout: Creating ceph-iscsi config... 2024-04-14T14:04:57.847 INFO:journalctl@ceph.mon.b.smithi122.stdout: Write file: /var/lib/ceph/dc326fc2-fa65-11ee-bc8e-c7b262605968/iscsi.foo.smithi069.yuumtr/iscsi-gateway.cfg 2024-04-14T14:04:57.847 INFO:journalctl@ceph.mon.b.smithi122.stdout: Write file: /var/lib/ceph/dc326fc2-fa65-11ee-bc8e-c7b262605968/iscsi.foo.smithi069.yuumtr/tcmu-runner-entrypoint.sh 2024-04-14T14:04:57.847 INFO:journalctl@ceph.mon.b.smithi122.stdout: Non-zero exit code 1 from systemctl restart ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968@iscsi.foo.smithi069.yuumtr 2024-04-14T14:04:57.847 INFO:journalctl@ceph.mon.b.smithi122.stdout: systemctl: stderr Job for ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968@iscsi.foo.smithi069.yuumtr.service failed because the control process exited with error code. 2024-04-14T14:04:57.847 INFO:journalctl@ceph.mon.b.smithi122.stdout: systemctl: stderr See "systemctl status ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968@iscsi.foo.smithi069.yuumtr.service" and "journalctl -xeu ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968@iscsi.foo.smithi069.yuumtr.service" for details. 2024-04-14T14:04:57.848 INFO:journalctl@ceph.mon.b.smithi122.stdout: Traceback (most recent call last): 2024-04-14T14:04:57.848 INFO:journalctl@ceph.mon.b.smithi122.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-04-14T14:04:57.848 INFO:journalctl@ceph.mon.b.smithi122.stdout: return _run_code(code, main_globals, None, 2024-04-14T14:04:57.848 INFO:journalctl@ceph.mon.b.smithi122.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-04-14T14:04:57.848 INFO:journalctl@ceph.mon.b.smithi122.stdout: exec(code, run_globals) 2024-04-14T14:04:57.848 INFO:journalctl@ceph.mon.b.smithi122.stdout: File "/var/lib/ceph/dc326fc2-fa65-11ee-bc8e-c7b262605968/cephadm.de2a8faf91349910526b51da93d8bd42d57efdd2e93c32cd09ae64ecc1748ae9/__main__.py", line 5608, in 2024-04-14T14:04:57.848 INFO:journalctl@ceph.mon.b.smithi122.stdout: File "/var/lib/ceph/dc326fc2-fa65-11ee-bc8e-c7b262605968/cephadm.de2a8faf91349910526b51da93d8bd42d57efdd2e93c32cd09ae64ecc1748ae9/__main__.py", line 5596, in main 2024-04-14T14:04:57.848 INFO:journalctl@ceph.mon.b.smithi122.stdout: File "/var/lib/ceph/dc326fc2-fa65-11ee-bc8e-c7b262605968/cephadm.de2a8faf91349910526b51da93d8bd42d57efdd2e93c32cd09ae64ecc1748ae9/__main__.py", line 3096, in command_deploy_from 2024-04-14T14:04:57.848 INFO:journalctl@ceph.mon.b.smithi122.stdout: File "/var/lib/ceph/dc326fc2-fa65-11ee-bc8e-c7b262605968/cephadm.de2a8faf91349910526b51da93d8bd42d57efdd2e93c32cd09ae64ecc1748ae9/__main__.py", line 3131, in _common_deploy 2024-04-14T14:04:57.848 INFO:journalctl@ceph.mon.b.smithi122.stdout: File "/var/lib/ceph/dc326fc2-fa65-11ee-bc8e-c7b262605968/cephadm.de2a8faf91349910526b51da93d8bd42d57efdd2e93c32cd09ae64ecc1748ae9/__main__.py", line 3151, in _deploy_daemon_container 2024-04-14T14:04:57.848 INFO:journalctl@ceph.mon.b.smithi122.stdout: File "/var/lib/ceph/dc326fc2-fa65-11ee-bc8e-c7b262605968/cephadm.de2a8faf91349910526b51da93d8bd42d57efdd2e93c32cd09ae64ecc1748ae9/__main__.py", line 1123, in deploy_daemon 2024-04-14T14:04:57.848 INFO:journalctl@ceph.mon.b.smithi122.stdout: File "/var/lib/ceph/dc326fc2-fa65-11ee-bc8e-c7b262605968/cephadm.de2a8faf91349910526b51da93d8bd42d57efdd2e93c32cd09ae64ecc1748ae9/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-04-14T14:04:57.848 INFO:journalctl@ceph.mon.b.smithi122.stdout: RuntimeError: Failed command: systemctl restart ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968@iscsi.foo.smithi069.yuumtr: Job for ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968@iscsi.foo.smithi069.yuumtr.service failed because the control process exited with error code. 2024-04-14T14:04:57.849 INFO:journalctl@ceph.mon.b.smithi122.stdout: See "systemctl status ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968@iscsi.foo.smithi069.yuumtr.service" and "journalctl -xeu ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968@iscsi.foo.smithi069.yuumtr.service" for details. 2024-04-14T14:04:59.067 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:04:58 smithi069 ceph-mon[24818]: pgmap v40: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 868 B/s rd, 0 op/s 2024-04-14T14:04:59.067 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:04:58 smithi069 ceph-mon[24818]: from='mgr.24749 ' entity='mgr.x' 2024-04-14T14:04:59.067 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:04:58 smithi069 ceph-mon[24818]: from='mgr.24749 172.21.15.122:0/1892077634' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T14:04:59.067 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:04:58 smithi069 ceph-mon[29472]: pgmap v40: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 868 B/s rd, 0 op/s 2024-04-14T14:04:59.067 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:04:58 smithi069 ceph-mon[29472]: from='mgr.24749 ' entity='mgr.x' 2024-04-14T14:04:59.067 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:04:58 smithi069 ceph-mon[29472]: from='mgr.24749 172.21.15.122:0/1892077634' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T14:04:59.094 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:04:58 smithi122 ceph-mon[25721]: pgmap v40: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 868 B/s rd, 0 op/s 2024-04-14T14:04:59.094 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:04:58 smithi122 ceph-mon[25721]: from='mgr.24749 ' entity='mgr.x' 2024-04-14T14:04:59.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:04:58 smithi122 ceph-mon[25721]: from='mgr.24749 172.21.15.122:0/1892077634' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T14:04:59.095 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Apr 14 14:04:58 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-prometheus-a[49277]: ts=2024-04-14T14:04:58.659Z caller=manager.go:609 level=warn component="rule manager" group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on(ceph_daemon) group_left(hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.69\", device_class=\"ssd\", hostname=\"smithi069\", instance=\"172.21.15.122:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.69\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster=\"dc326fc2-fa65-11ee-bc8e-c7b262605968\", cluster_addr=\"172.21.15.69\", device_class=\"ssd\", hostname=\"smithi069\", instance=\"smithi122:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.69\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-04-14T14:05:00.352 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Apr 14 14:05:00 smithi122 systemd[1]: ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968@grafana.a.service: Scheduled restart job, restart counter is at 1. 2024-04-14T14:05:00.352 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Apr 14 14:05:00 smithi122 systemd[1]: Stopped Ceph grafana.a for dc326fc2-fa65-11ee-bc8e-c7b262605968. 2024-04-14T14:05:00.352 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Apr 14 14:05:00 smithi122 systemd[1]: Starting Ceph grafana.a for dc326fc2-fa65-11ee-bc8e-c7b262605968... 2024-04-14T14:05:00.352 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Apr 14 14:05:00 smithi122 podman[50587]: 2024-04-14 14:05:00.2422349 +0000 UTC m=+0.017328643 image pull dad864ee21e98e69f4029d1e417aa085001566be0d322fbc75bc6f29b0050c01 quay.io/ceph/ceph-grafana:8.3.5 2024-04-14T14:05:00.352 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Apr 14 14:05:00 smithi122 podman[50587]: 2024-04-14 14:05:00.350789224 +0000 UTC m=+0.125882953 container create 297569321a826d715a31c4eb8211e4decad7085e842f8f8d10788fde5aa9ddcc (image=quay.io/ceph/ceph-grafana:8.3.5, name=ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-grafana-a, io.openshift.expose-services=, com.redhat.component=ubi8-container, maintainer=Paul Cuzner , vendor=Red Hat, Inc., architecture=x86_64, version=8.5, summary=Grafana Container configured for Ceph mgr/dashboard integration, build-date=2022-03-28T10:36:18.413762, distribution-scope=public, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-236.1648460182, name=ubi8, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., description=Ceph Grafana Container, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, release=236.1648460182, vcs-type=git, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, io.k8s.display-name=Red Hat Universal Base Image 8, io.buildah.version=1.24.2, io.openshift.tags=base rhel8) 2024-04-14T14:05:00.644 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Apr 14 14:05:00 smithi122 podman[50587]: 2024-04-14 14:05:00.56141194 +0000 UTC m=+0.336505671 container init 297569321a826d715a31c4eb8211e4decad7085e842f8f8d10788fde5aa9ddcc (image=quay.io/ceph/ceph-grafana:8.3.5, name=ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-grafana-a, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., io.openshift.tags=base rhel8, io.openshift.expose-services=, vendor=Red Hat, Inc., name=ubi8, description=Ceph Grafana Container, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-236.1648460182, architecture=x86_64, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, version=8.5, io.k8s.display-name=Red Hat Universal Base Image 8, release=236.1648460182, summary=Grafana Container configured for Ceph mgr/dashboard integration, build-date=2022-03-28T10:36:18.413762, maintainer=Paul Cuzner , vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, com.redhat.component=ubi8-container, io.buildah.version=1.24.2, vcs-type=git, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, distribution-scope=public) 2024-04-14T14:05:00.644 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Apr 14 14:05:00 smithi122 podman[50587]: 2024-04-14 14:05:00.563866557 +0000 UTC m=+0.338960286 container start 297569321a826d715a31c4eb8211e4decad7085e842f8f8d10788fde5aa9ddcc (image=quay.io/ceph/ceph-grafana:8.3.5, name=ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-grafana-a, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, com.redhat.component=ubi8-container, io.k8s.display-name=Red Hat Universal Base Image 8, io.buildah.version=1.24.2, summary=Grafana Container configured for Ceph mgr/dashboard integration, io.openshift.tags=base rhel8, maintainer=Paul Cuzner , release=236.1648460182, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, vcs-type=git, build-date=2022-03-28T10:36:18.413762, distribution-scope=public, vendor=Red Hat, Inc., name=ubi8, architecture=x86_64, description=Ceph Grafana Container, version=8.5, io.openshift.expose-services=, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-236.1648460182) 2024-04-14T14:05:00.644 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Apr 14 14:05:00 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-grafana-a[50602]: Failed to start grafana. error: failed to create log directory "/usr/share/grafana/data/log": mkdir /usr/share/grafana/data: permission denied 2024-04-14T14:05:00.644 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Apr 14 14:05:00 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-grafana-a[50602]: failed to create log directory "/usr/share/grafana/data/log": mkdir /usr/share/grafana/data: permission denied 2024-04-14T14:05:00.915 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:05:00 smithi122 ceph-mon[25721]: pgmap v41: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-04-14T14:05:00.916 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Apr 14 14:05:00 smithi122 bash[50587]: 297569321a826d715a31c4eb8211e4decad7085e842f8f8d10788fde5aa9ddcc 2024-04-14T14:05:00.916 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Apr 14 14:05:00 smithi122 systemd[1]: Started Ceph grafana.a for dc326fc2-fa65-11ee-bc8e-c7b262605968. 2024-04-14T14:05:00.916 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Apr 14 14:05:00 smithi122 podman[50618]: 2024-04-14 14:05:00.652108621 +0000 UTC m=+0.052303926 container died 297569321a826d715a31c4eb8211e4decad7085e842f8f8d10788fde5aa9ddcc (image=quay.io/ceph/ceph-grafana:8.3.5, name=ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-grafana-a, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, io.openshift.tags=base rhel8, summary=Grafana Container configured for Ceph mgr/dashboard integration, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, maintainer=Paul Cuzner , vendor=Red Hat, Inc., io.k8s.display-name=Red Hat Universal Base Image 8, release=236.1648460182, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-236.1648460182, architecture=x86_64, com.redhat.component=ubi8-container, distribution-scope=public, io.buildah.version=1.24.2, name=ubi8, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, build-date=2022-03-28T10:36:18.413762, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., io.openshift.expose-services=, vcs-type=git, description=Ceph Grafana Container, version=8.5) 2024-04-14T14:05:01.066 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:05:00 smithi069 ceph-mon[24818]: pgmap v41: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-04-14T14:05:01.067 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:05:00 smithi069 ceph-mon[29472]: pgmap v41: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-04-14T14:05:01.595 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Apr 14 14:05:01 smithi122 podman[50618]: 2024-04-14 14:05:01.206817951 +0000 UTC m=+0.607013258 container remove 297569321a826d715a31c4eb8211e4decad7085e842f8f8d10788fde5aa9ddcc (image=quay.io/ceph/ceph-grafana:8.3.5, name=ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-grafana-a, io.openshift.tags=base rhel8, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., io.openshift.expose-services=, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, io.buildah.version=1.24.2, summary=Grafana Container configured for Ceph mgr/dashboard integration, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, architecture=x86_64, maintainer=Paul Cuzner , version=8.5, name=ubi8, description=Ceph Grafana Container, release=236.1648460182, vcs-type=git, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-236.1648460182, distribution-scope=public, build-date=2022-03-28T10:36:18.413762, com.redhat.component=ubi8-container, io.k8s.display-name=Red Hat Universal Base Image 8, vendor=Red Hat, Inc.) 2024-04-14T14:05:01.595 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Apr 14 14:05:01 smithi122 systemd[1]: ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968@grafana.a.service: Main process exited, code=exited, status=1/FAILURE 2024-04-14T14:05:01.595 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Apr 14 14:05:01 smithi122 systemd[1]: ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968@grafana.a.service: Failed with result 'exit-code'. 2024-04-14T14:05:03.066 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:05:02 smithi069 ceph-mon[24818]: pgmap v42: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 607 B/s rd, 0 op/s 2024-04-14T14:05:03.067 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:05:02 smithi069 ceph-mon[29472]: pgmap v42: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 607 B/s rd, 0 op/s 2024-04-14T14:05:03.094 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:05:02 smithi122 ceph-mon[25721]: pgmap v42: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 607 B/s rd, 0 op/s 2024-04-14T14:05:03.095 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Apr 14 14:05:02 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-x[45487]: ::ffff:172.21.15.122 - - [14/Apr/2024:14:05:02] "GET /metrics HTTP/1.1" 200 37964 "" "Prometheus/2.33.4" 2024-04-14T14:05:03.844 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Apr 14 14:05:03 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-prometheus-a[49277]: ts=2024-04-14T14:05:03.504Z caller=manager.go:609 level=warn component="rule manager" group=nodes msg="Evaluating rule failed" rule="alert: CephNodeDiskspaceWarning\nexpr: predict_linear(node_filesystem_free_bytes{device=~\"/.*\"}[2d], 3600 * 24 * 5)\n * on(instance) group_left(nodename) node_uname_info < 0\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.8.4\n severity: warning\n type: ceph_default\nannotations:\n description: Mountpoint {{ $labels.mountpoint }} on {{ $labels.nodename }} will\n be full in less than 5 days based on the 48 hour trailing fill rate.\n summary: Host filesystem free space is getting low\n" err="found duplicate series for the match group {instance=\"smithi069\"} on the right hand-side of the operation: [{__name__=\"node_uname_info\", domainname=\"(none)\", instance=\"smithi069\", job=\"node\", machine=\"x86_64\", nodename=\"smithi069\", release=\"5.14.0-435.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Fri Mar 29 18:46:37 UTC 2024\"}, {__name__=\"node_uname_info\", cluster=\"dc326fc2-fa65-11ee-bc8e-c7b262605968\", domainname=\"(none)\", instance=\"smithi069\", job=\"node\", machine=\"x86_64\", nodename=\"smithi069\", release=\"5.14.0-435.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Fri Mar 29 18:46:37 UTC 2024\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-04-14T14:05:05.066 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:05:04 smithi069 ceph-mon[24818]: pgmap v43: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 607 B/s rd, 0 op/s 2024-04-14T14:05:05.067 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:05:04 smithi069 ceph-mon[29472]: pgmap v43: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 607 B/s rd, 0 op/s 2024-04-14T14:05:05.094 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:05:04 smithi122 ceph-mon[25721]: pgmap v43: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 607 B/s rd, 0 op/s 2024-04-14T14:05:06.897 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:05:06 smithi069 ceph-mon[24818]: pgmap v44: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 597 B/s rd, 0 op/s 2024-04-14T14:05:06.897 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:05:06 smithi069 ceph-mon[29472]: pgmap v44: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 597 B/s rd, 0 op/s 2024-04-14T14:05:07.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:05:06 smithi122 ceph-mon[25721]: pgmap v44: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 597 B/s rd, 0 op/s 2024-04-14T14:05:08.817 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:05:08 smithi069 ceph-mon[24818]: pgmap v45: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-04-14T14:05:08.817 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:05:08 smithi069 ceph-mon[24818]: from='client.? 172.21.15.69:0/2359283352' entity='client.iscsi.foo.smithi069.yuumtr' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-04-14T14:05:08.817 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:05:08 smithi069 ceph-mon[29472]: pgmap v45: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-04-14T14:05:08.817 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:05:08 smithi069 ceph-mon[29472]: from='client.? 172.21.15.69:0/2359283352' entity='client.iscsi.foo.smithi069.yuumtr' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-04-14T14:05:09.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:05:08 smithi122 ceph-mon[25721]: pgmap v45: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-04-14T14:05:09.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:05:08 smithi122 ceph-mon[25721]: from='client.? 172.21.15.69:0/2359283352' entity='client.iscsi.foo.smithi069.yuumtr' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-04-14T14:05:09.095 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Apr 14 14:05:08 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-prometheus-a[49277]: ts=2024-04-14T14:05:08.659Z caller=manager.go:609 level=warn component="rule manager" group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on(ceph_daemon) group_left(hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.69\", device_class=\"ssd\", hostname=\"smithi069\", instance=\"172.21.15.122:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.69\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster=\"dc326fc2-fa65-11ee-bc8e-c7b262605968\", cluster_addr=\"172.21.15.69\", device_class=\"ssd\", hostname=\"smithi069\", instance=\"smithi122:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.69\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-04-14T14:05:11.066 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:05:10 smithi069 ceph-mon[24818]: pgmap v46: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 255 B/s rd, 0 op/s 2024-04-14T14:05:11.067 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:05:10 smithi069 ceph-mon[29472]: pgmap v46: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 255 B/s rd, 0 op/s 2024-04-14T14:05:11.094 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:05:10 smithi122 ceph-mon[25721]: pgmap v46: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 255 B/s rd, 0 op/s 2024-04-14T14:05:11.608 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Apr 14 14:05:11 smithi122 systemd[1]: ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968@grafana.a.service: Scheduled restart job, restart counter is at 2. 2024-04-14T14:05:11.608 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Apr 14 14:05:11 smithi122 systemd[1]: Stopped Ceph grafana.a for dc326fc2-fa65-11ee-bc8e-c7b262605968. 2024-04-14T14:05:11.609 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Apr 14 14:05:11 smithi122 systemd[1]: Starting Ceph grafana.a for dc326fc2-fa65-11ee-bc8e-c7b262605968... 2024-04-14T14:05:11.609 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Apr 14 14:05:11 smithi122 podman[50689]: 2024-04-14 14:05:11.480329032 +0000 UTC m=+0.017521521 image pull dad864ee21e98e69f4029d1e417aa085001566be0d322fbc75bc6f29b0050c01 quay.io/ceph/ceph-grafana:8.3.5 2024-04-14T14:05:11.903 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Apr 14 14:05:11 smithi122 podman[50689]: 2024-04-14 14:05:11.6076765 +0000 UTC m=+0.144868968 container create 1272434e3261df116e226bf60db1c7176056ae8ba7218f9848a88d5d61e57ce2 (image=quay.io/ceph/ceph-grafana:8.3.5, name=ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-grafana-a, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., summary=Grafana Container configured for Ceph mgr/dashboard integration, io.buildah.version=1.24.2, vcs-type=git, name=ubi8, release=236.1648460182, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, architecture=x86_64, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, maintainer=Paul Cuzner , version=8.5, description=Ceph Grafana Container, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-236.1648460182, build-date=2022-03-28T10:36:18.413762, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, io.openshift.tags=base rhel8, io.k8s.display-name=Red Hat Universal Base Image 8, com.redhat.component=ubi8-container, vendor=Red Hat, Inc., io.openshift.expose-services=, distribution-scope=public) 2024-04-14T14:05:12.248 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Apr 14 14:05:11 smithi122 podman[50689]: 2024-04-14 14:05:11.902376259 +0000 UTC m=+0.439568726 container init 1272434e3261df116e226bf60db1c7176056ae8ba7218f9848a88d5d61e57ce2 (image=quay.io/ceph/ceph-grafana:8.3.5, name=ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-grafana-a, distribution-scope=public, io.buildah.version=1.24.2, io.k8s.display-name=Red Hat Universal Base Image 8, architecture=x86_64, io.openshift.tags=base rhel8, release=236.1648460182, version=8.5, vendor=Red Hat, Inc., com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, description=Ceph Grafana Container, io.openshift.expose-services=, build-date=2022-03-28T10:36:18.413762, vcs-type=git, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, name=ubi8, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-236.1648460182, summary=Grafana Container configured for Ceph mgr/dashboard integration, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, com.redhat.component=ubi8-container, maintainer=Paul Cuzner ) 2024-04-14T14:05:12.249 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Apr 14 14:05:11 smithi122 podman[50689]: 2024-04-14 14:05:11.904983594 +0000 UTC m=+0.442176070 container start 1272434e3261df116e226bf60db1c7176056ae8ba7218f9848a88d5d61e57ce2 (image=quay.io/ceph/ceph-grafana:8.3.5, name=ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-grafana-a, description=Ceph Grafana Container, summary=Grafana Container configured for Ceph mgr/dashboard integration, io.buildah.version=1.24.2, vcs-type=git, name=ubi8, vendor=Red Hat, Inc., version=8.5, architecture=x86_64, io.openshift.expose-services=, io.openshift.tags=base rhel8, io.k8s.display-name=Red Hat Universal Base Image 8, com.redhat.component=ubi8-container, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, distribution-scope=public, build-date=2022-03-28T10:36:18.413762, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, maintainer=Paul Cuzner , release=236.1648460182, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-236.1648460182, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly.) 2024-04-14T14:05:12.249 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Apr 14 14:05:11 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-grafana-a[50704]: Failed to start grafana. error: failed to create log directory "/usr/share/grafana/data/log": mkdir /usr/share/grafana/data: permission denied 2024-04-14T14:05:12.249 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Apr 14 14:05:11 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-grafana-a[50704]: failed to create log directory "/usr/share/grafana/data/log": mkdir /usr/share/grafana/data: permission denied 2024-04-14T14:05:12.249 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Apr 14 14:05:12 smithi122 bash[50689]: 1272434e3261df116e226bf60db1c7176056ae8ba7218f9848a88d5d61e57ce2 2024-04-14T14:05:12.249 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Apr 14 14:05:12 smithi122 systemd[1]: Started Ceph grafana.a for dc326fc2-fa65-11ee-bc8e-c7b262605968. 2024-04-14T14:05:12.249 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Apr 14 14:05:12 smithi122 podman[50720]: 2024-04-14 14:05:12.035783276 +0000 UTC m=+0.097089556 container died 1272434e3261df116e226bf60db1c7176056ae8ba7218f9848a88d5d61e57ce2 (image=quay.io/ceph/ceph-grafana:8.3.5, name=ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-grafana-a, summary=Grafana Container configured for Ceph mgr/dashboard integration, io.buildah.version=1.24.2, io.k8s.display-name=Red Hat Universal Base Image 8, io.openshift.expose-services=, maintainer=Paul Cuzner , version=8.5, distribution-scope=public, name=ubi8, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-236.1648460182, vendor=Red Hat, Inc., com.redhat.component=ubi8-container, vcs-type=git, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, build-date=2022-03-28T10:36:18.413762, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, io.openshift.tags=base rhel8, architecture=x86_64, release=236.1648460182, description=Ceph Grafana Container, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly.) 2024-04-14T14:05:13.067 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:05:12 smithi069 ceph-mon[24818]: pgmap v47: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2024-04-14T14:05:13.067 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:05:12 smithi069 ceph-mon[24818]: from='mgr.24749 172.21.15.122:0/1892077634' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T14:05:13.067 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:05:12 smithi069 ceph-mon[29472]: pgmap v47: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2024-04-14T14:05:13.067 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:05:12 smithi069 ceph-mon[29472]: from='mgr.24749 172.21.15.122:0/1892077634' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T14:05:13.094 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:05:12 smithi122 ceph-mon[25721]: pgmap v47: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2024-04-14T14:05:13.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:05:12 smithi122 ceph-mon[25721]: from='mgr.24749 172.21.15.122:0/1892077634' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T14:05:13.095 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Apr 14 14:05:12 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-x[45487]: ::ffff:172.21.15.122 - - [14/Apr/2024:14:05:12] "GET /metrics HTTP/1.1" 200 37964 "" "Prometheus/2.33.4" 2024-04-14T14:05:13.095 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Apr 14 14:05:12 smithi122 podman[50720]: 2024-04-14 14:05:12.598294517 +0000 UTC m=+0.659600781 container remove 1272434e3261df116e226bf60db1c7176056ae8ba7218f9848a88d5d61e57ce2 (image=quay.io/ceph/ceph-grafana:8.3.5, name=ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-grafana-a, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., io.openshift.tags=base rhel8, version=8.5, name=ubi8, build-date=2022-03-28T10:36:18.413762, maintainer=Paul Cuzner , com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, vendor=Red Hat, Inc., io.k8s.display-name=Red Hat Universal Base Image 8, io.openshift.expose-services=, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, description=Ceph Grafana Container, summary=Grafana Container configured for Ceph mgr/dashboard integration, architecture=x86_64, vcs-type=git, distribution-scope=public, release=236.1648460182, com.redhat.component=ubi8-container, io.buildah.version=1.24.2, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-236.1648460182) 2024-04-14T14:05:13.095 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Apr 14 14:05:12 smithi122 systemd[1]: ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968@grafana.a.service: Main process exited, code=exited, status=1/FAILURE 2024-04-14T14:05:13.095 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Apr 14 14:05:12 smithi122 systemd[1]: ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968@grafana.a.service: Failed with result 'exit-code'. 2024-04-14T14:05:13.845 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Apr 14 14:05:13 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-prometheus-a[49277]: ts=2024-04-14T14:05:13.505Z caller=manager.go:609 level=warn component="rule manager" group=nodes msg="Evaluating rule failed" rule="alert: CephNodeDiskspaceWarning\nexpr: predict_linear(node_filesystem_free_bytes{device=~\"/.*\"}[2d], 3600 * 24 * 5)\n * on(instance) group_left(nodename) node_uname_info < 0\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.8.4\n severity: warning\n type: ceph_default\nannotations:\n description: Mountpoint {{ $labels.mountpoint }} on {{ $labels.nodename }} will\n be full in less than 5 days based on the 48 hour trailing fill rate.\n summary: Host filesystem free space is getting low\n" err="found duplicate series for the match group {instance=\"smithi069\"} on the right hand-side of the operation: [{__name__=\"node_uname_info\", domainname=\"(none)\", instance=\"smithi069\", job=\"node\", machine=\"x86_64\", nodename=\"smithi069\", release=\"5.14.0-435.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Fri Mar 29 18:46:37 UTC 2024\"}, {__name__=\"node_uname_info\", cluster=\"dc326fc2-fa65-11ee-bc8e-c7b262605968\", domainname=\"(none)\", instance=\"smithi069\", job=\"node\", machine=\"x86_64\", nodename=\"smithi069\", release=\"5.14.0-435.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Fri Mar 29 18:46:37 UTC 2024\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-04-14T14:05:15.066 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:05:14 smithi069 ceph-mon[24818]: pgmap v48: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2024-04-14T14:05:15.067 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:05:14 smithi069 ceph-mon[29472]: pgmap v48: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2024-04-14T14:05:15.094 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:05:14 smithi122 ceph-mon[25721]: pgmap v48: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2024-04-14T14:05:17.067 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:05:16 smithi069 ceph-mon[24818]: pgmap v49: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 511 B/s rd, 0 op/s 2024-04-14T14:05:17.067 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:05:16 smithi069 ceph-mon[29472]: pgmap v49: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 511 B/s rd, 0 op/s 2024-04-14T14:05:17.094 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:05:16 smithi122 ceph-mon[25721]: pgmap v49: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 511 B/s rd, 0 op/s 2024-04-14T14:05:19.067 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:05:18 smithi069 ceph-mon[24818]: pgmap v50: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 511 B/s rd, 0 op/s 2024-04-14T14:05:19.067 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:05:18 smithi069 ceph-mon[24818]: from='client.25048 -' entity='client.iscsi.foo.smithi069.yuumtr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T14:05:19.067 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:05:18 smithi069 ceph-mon[29472]: pgmap v50: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 511 B/s rd, 0 op/s 2024-04-14T14:05:19.067 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:05:18 smithi069 ceph-mon[29472]: from='client.25048 -' entity='client.iscsi.foo.smithi069.yuumtr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T14:05:19.094 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:05:18 smithi122 ceph-mon[25721]: pgmap v50: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 511 B/s rd, 0 op/s 2024-04-14T14:05:19.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:05:18 smithi122 ceph-mon[25721]: from='client.25048 -' entity='client.iscsi.foo.smithi069.yuumtr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T14:05:19.095 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Apr 14 14:05:18 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-prometheus-a[49277]: ts=2024-04-14T14:05:18.660Z caller=manager.go:609 level=warn component="rule manager" group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on(ceph_daemon) group_left(hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.69\", device_class=\"ssd\", hostname=\"smithi069\", instance=\"172.21.15.122:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.69\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster=\"dc326fc2-fa65-11ee-bc8e-c7b262605968\", cluster_addr=\"172.21.15.69\", device_class=\"ssd\", hostname=\"smithi069\", instance=\"smithi122:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.69\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-04-14T14:05:21.067 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:05:20 smithi069 ceph-mon[24818]: pgmap v51: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 938 B/s rd, 0 op/s 2024-04-14T14:05:21.067 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:05:20 smithi069 ceph-mon[29472]: pgmap v51: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 938 B/s rd, 0 op/s 2024-04-14T14:05:21.094 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:05:20 smithi122 ceph-mon[25721]: pgmap v51: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 938 B/s rd, 0 op/s 2024-04-14T14:05:22.918 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:05:22 smithi122 ceph-mon[25721]: pgmap v52: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:05:22.919 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Apr 14 14:05:22 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-x[45487]: ::ffff:172.21.15.122 - - [14/Apr/2024:14:05:22] "GET /metrics HTTP/1.1" 200 37964 "" "Prometheus/2.33.4" 2024-04-14T14:05:22.919 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Apr 14 14:05:22 smithi122 systemd[1]: ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968@grafana.a.service: Scheduled restart job, restart counter is at 3. 2024-04-14T14:05:22.919 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Apr 14 14:05:22 smithi122 systemd[1]: Stopped Ceph grafana.a for dc326fc2-fa65-11ee-bc8e-c7b262605968. 2024-04-14T14:05:22.919 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Apr 14 14:05:22 smithi122 systemd[1]: Starting Ceph grafana.a for dc326fc2-fa65-11ee-bc8e-c7b262605968... 2024-04-14T14:05:22.919 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Apr 14 14:05:22 smithi122 podman[50803]: 2024-04-14 14:05:22.790207305 +0000 UTC m=+0.017240590 image pull dad864ee21e98e69f4029d1e417aa085001566be0d322fbc75bc6f29b0050c01 quay.io/ceph/ceph-grafana:8.3.5 2024-04-14T14:05:22.919 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Apr 14 14:05:22 smithi122 podman[50803]: 2024-04-14 14:05:22.903078372 +0000 UTC m=+0.130111659 container create b313b2b330b97375b2e5d4b458f89f23b3f2d0314ba3bbff2d3e9dc9008d4295 (image=quay.io/ceph/ceph-grafana:8.3.5, name=ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-grafana-a, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, com.redhat.component=ubi8-container, vcs-type=git, architecture=x86_64, maintainer=Paul Cuzner , distribution-scope=public, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, description=Ceph Grafana Container, io.buildah.version=1.24.2, io.openshift.tags=base rhel8, version=8.5, summary=Grafana Container configured for Ceph mgr/dashboard integration, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., release=236.1648460182, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, io.k8s.display-name=Red Hat Universal Base Image 8, io.openshift.expose-services=, build-date=2022-03-28T10:36:18.413762, name=ubi8, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-236.1648460182, vendor=Red Hat, Inc.) 2024-04-14T14:05:23.067 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:05:22 smithi069 ceph-mon[24818]: pgmap v52: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:05:23.067 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:05:22 smithi069 ceph-mon[29472]: pgmap v52: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:05:23.595 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Apr 14 14:05:23 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-prometheus-a[49277]: ts=2024-04-14T14:05:23.504Z caller=manager.go:609 level=warn component="rule manager" group=nodes msg="Evaluating rule failed" rule="alert: CephNodeDiskspaceWarning\nexpr: predict_linear(node_filesystem_free_bytes{device=~\"/.*\"}[2d], 3600 * 24 * 5)\n * on(instance) group_left(nodename) node_uname_info < 0\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.8.4\n severity: warning\n type: ceph_default\nannotations:\n description: Mountpoint {{ $labels.mountpoint }} on {{ $labels.nodename }} will\n be full in less than 5 days based on the 48 hour trailing fill rate.\n summary: Host filesystem free space is getting low\n" err="found duplicate series for the match group {instance=\"smithi069\"} on the right hand-side of the operation: [{__name__=\"node_uname_info\", domainname=\"(none)\", instance=\"smithi069\", job=\"node\", machine=\"x86_64\", nodename=\"smithi069\", release=\"5.14.0-435.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Fri Mar 29 18:46:37 UTC 2024\"}, {__name__=\"node_uname_info\", cluster=\"dc326fc2-fa65-11ee-bc8e-c7b262605968\", domainname=\"(none)\", instance=\"smithi069\", job=\"node\", machine=\"x86_64\", nodename=\"smithi069\", release=\"5.14.0-435.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Fri Mar 29 18:46:37 UTC 2024\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-04-14T14:05:23.595 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Apr 14 14:05:23 smithi122 podman[50803]: 2024-04-14 14:05:23.330799596 +0000 UTC m=+0.557832881 container init b313b2b330b97375b2e5d4b458f89f23b3f2d0314ba3bbff2d3e9dc9008d4295 (image=quay.io/ceph/ceph-grafana:8.3.5, name=ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-grafana-a, io.k8s.display-name=Red Hat Universal Base Image 8, vcs-type=git, maintainer=Paul Cuzner , name=ubi8, com.redhat.component=ubi8-container, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, io.buildah.version=1.24.2, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., summary=Grafana Container configured for Ceph mgr/dashboard integration, architecture=x86_64, description=Ceph Grafana Container, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, io.openshift.tags=base rhel8, vendor=Red Hat, Inc., io.openshift.expose-services=, build-date=2022-03-28T10:36:18.413762, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-236.1648460182, distribution-scope=public, version=8.5, release=236.1648460182, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI) 2024-04-14T14:05:23.595 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Apr 14 14:05:23 smithi122 podman[50803]: 2024-04-14 14:05:23.333427822 +0000 UTC m=+0.560461109 container start b313b2b330b97375b2e5d4b458f89f23b3f2d0314ba3bbff2d3e9dc9008d4295 (image=quay.io/ceph/ceph-grafana:8.3.5, name=ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-grafana-a, com.redhat.component=ubi8-container, summary=Grafana Container configured for Ceph mgr/dashboard integration, vcs-type=git, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-236.1648460182, maintainer=Paul Cuzner , com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, vendor=Red Hat, Inc., io.k8s.display-name=Red Hat Universal Base Image 8, architecture=x86_64, distribution-scope=public, build-date=2022-03-28T10:36:18.413762, io.openshift.tags=base rhel8, version=8.5, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, release=236.1648460182, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, io.buildah.version=1.24.2, description=Ceph Grafana Container, io.openshift.expose-services=, name=ubi8) 2024-04-14T14:05:23.595 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Apr 14 14:05:23 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-grafana-a[50821]: Failed to start grafana. error: failed to create log directory "/usr/share/grafana/data/log": mkdir /usr/share/grafana/data: permission denied 2024-04-14T14:05:23.595 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Apr 14 14:05:23 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-grafana-a[50821]: failed to create log directory "/usr/share/grafana/data/log": mkdir /usr/share/grafana/data: permission denied 2024-04-14T14:05:23.595 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Apr 14 14:05:23 smithi122 bash[50803]: b313b2b330b97375b2e5d4b458f89f23b3f2d0314ba3bbff2d3e9dc9008d4295 2024-04-14T14:05:23.595 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Apr 14 14:05:23 smithi122 systemd[1]: Started Ceph grafana.a for dc326fc2-fa65-11ee-bc8e-c7b262605968. 2024-04-14T14:05:23.595 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Apr 14 14:05:23 smithi122 podman[50837]: 2024-04-14 14:05:23.44562614 +0000 UTC m=+0.078593822 container died b313b2b330b97375b2e5d4b458f89f23b3f2d0314ba3bbff2d3e9dc9008d4295 (image=quay.io/ceph/ceph-grafana:8.3.5, name=ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-grafana-a, com.redhat.component=ubi8-container, description=Ceph Grafana Container, io.openshift.expose-services=, io.openshift.tags=base rhel8, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-236.1648460182, architecture=x86_64, distribution-scope=public, vendor=Red Hat, Inc., vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, summary=Grafana Container configured for Ceph mgr/dashboard integration, io.buildah.version=1.24.2, maintainer=Paul Cuzner , com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., name=ubi8, vcs-type=git, version=8.5, io.k8s.display-name=Red Hat Universal Base Image 8, release=236.1648460182, build-date=2022-03-28T10:36:18.413762) 2024-04-14T14:05:24.345 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Apr 14 14:05:24 smithi122 podman[50837]: 2024-04-14 14:05:24.04346071 +0000 UTC m=+0.676428420 container remove b313b2b330b97375b2e5d4b458f89f23b3f2d0314ba3bbff2d3e9dc9008d4295 (image=quay.io/ceph/ceph-grafana:8.3.5, name=ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-grafana-a, maintainer=Paul Cuzner , version=8.5, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-236.1648460182, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., io.openshift.expose-services=, vendor=Red Hat, Inc., architecture=x86_64, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, vcs-type=git, io.buildah.version=1.24.2, io.k8s.display-name=Red Hat Universal Base Image 8, summary=Grafana Container configured for Ceph mgr/dashboard integration, description=Ceph Grafana Container, release=236.1648460182, build-date=2022-03-28T10:36:18.413762, com.redhat.component=ubi8-container, name=ubi8, io.openshift.tags=base rhel8, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, distribution-scope=public) 2024-04-14T14:05:24.345 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Apr 14 14:05:24 smithi122 systemd[1]: ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968@grafana.a.service: Main process exited, code=exited, status=1/FAILURE 2024-04-14T14:05:24.345 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Apr 14 14:05:24 smithi122 systemd[1]: ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968@grafana.a.service: Failed with result 'exit-code'. 2024-04-14T14:05:25.067 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:05:24 smithi069 ceph-mon[24818]: pgmap v53: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:05:25.067 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:05:24 smithi069 ceph-mon[29472]: pgmap v53: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:05:25.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:05:24 smithi122 ceph-mon[25721]: pgmap v53: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:05:27.067 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:05:26 smithi069 ceph-mon[24818]: pgmap v54: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:05:27.067 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:05:26 smithi069 ceph-mon[29472]: pgmap v54: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:05:27.094 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:05:26 smithi122 ceph-mon[25721]: pgmap v54: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:05:28.067 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:05:27 smithi069 ceph-mon[24818]: from='mgr.24749 172.21.15.122:0/1892077634' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T14:05:28.067 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:05:27 smithi069 ceph-mon[29472]: from='mgr.24749 172.21.15.122:0/1892077634' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T14:05:28.094 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:05:27 smithi122 ceph-mon[25721]: from='mgr.24749 172.21.15.122:0/1892077634' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T14:05:29.067 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:05:28 smithi069 ceph-mon[24818]: pgmap v55: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:05:29.067 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:05:28 smithi069 ceph-mon[24818]: from='client.25048 -' entity='client.iscsi.foo.smithi069.yuumtr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T14:05:29.067 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:05:28 smithi069 ceph-mon[29472]: pgmap v55: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:05:29.068 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:05:28 smithi069 ceph-mon[29472]: from='client.25048 -' entity='client.iscsi.foo.smithi069.yuumtr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T14:05:29.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:05:28 smithi122 ceph-mon[25721]: pgmap v55: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:05:29.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:05:28 smithi122 ceph-mon[25721]: from='client.25048 -' entity='client.iscsi.foo.smithi069.yuumtr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T14:05:29.096 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Apr 14 14:05:28 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-prometheus-a[49277]: ts=2024-04-14T14:05:28.660Z caller=manager.go:609 level=warn component="rule manager" group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on(ceph_daemon) group_left(hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.69\", device_class=\"ssd\", hostname=\"smithi069\", instance=\"172.21.15.122:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.69\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster=\"dc326fc2-fa65-11ee-bc8e-c7b262605968\", cluster_addr=\"172.21.15.69\", device_class=\"ssd\", hostname=\"smithi069\", instance=\"smithi122:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.69\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-04-14T14:05:31.066 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:05:30 smithi069 ceph-mon[24818]: pgmap v56: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:05:31.067 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:05:30 smithi069 ceph-mon[29472]: pgmap v56: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:05:31.094 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:05:30 smithi122 ceph-mon[25721]: pgmap v56: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:05:33.067 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:05:32 smithi069 ceph-mon[24818]: pgmap v57: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:05:33.067 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:05:32 smithi069 ceph-mon[29472]: pgmap v57: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:05:33.094 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:05:32 smithi122 ceph-mon[25721]: pgmap v57: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:05:33.095 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Apr 14 14:05:32 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-x[45487]: ::ffff:172.21.15.122 - - [14/Apr/2024:14:05:32] "GET /metrics HTTP/1.1" 200 37964 "" "Prometheus/2.33.4" 2024-04-14T14:05:33.845 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Apr 14 14:05:33 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-prometheus-a[49277]: ts=2024-04-14T14:05:33.505Z caller=manager.go:609 level=warn component="rule manager" group=nodes msg="Evaluating rule failed" rule="alert: CephNodeDiskspaceWarning\nexpr: predict_linear(node_filesystem_free_bytes{device=~\"/.*\"}[2d], 3600 * 24 * 5)\n * on(instance) group_left(nodename) node_uname_info < 0\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.8.4\n severity: warning\n type: ceph_default\nannotations:\n description: Mountpoint {{ $labels.mountpoint }} on {{ $labels.nodename }} will\n be full in less than 5 days based on the 48 hour trailing fill rate.\n summary: Host filesystem free space is getting low\n" err="found duplicate series for the match group {instance=\"smithi069\"} on the right hand-side of the operation: [{__name__=\"node_uname_info\", domainname=\"(none)\", instance=\"smithi069\", job=\"node\", machine=\"x86_64\", nodename=\"smithi069\", release=\"5.14.0-435.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Fri Mar 29 18:46:37 UTC 2024\"}, {__name__=\"node_uname_info\", cluster=\"dc326fc2-fa65-11ee-bc8e-c7b262605968\", domainname=\"(none)\", instance=\"smithi069\", job=\"node\", machine=\"x86_64\", nodename=\"smithi069\", release=\"5.14.0-435.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Fri Mar 29 18:46:37 UTC 2024\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-04-14T14:05:34.366 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Apr 14 14:05:34 smithi122 systemd[1]: ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968@grafana.a.service: Scheduled restart job, restart counter is at 4. 2024-04-14T14:05:34.366 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Apr 14 14:05:34 smithi122 systemd[1]: Stopped Ceph grafana.a for dc326fc2-fa65-11ee-bc8e-c7b262605968. 2024-04-14T14:05:34.367 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Apr 14 14:05:34 smithi122 systemd[1]: Starting Ceph grafana.a for dc326fc2-fa65-11ee-bc8e-c7b262605968... 2024-04-14T14:05:34.367 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Apr 14 14:05:34 smithi122 podman[50905]: 2024-04-14 14:05:34.234665678 +0000 UTC m=+0.017464624 image pull dad864ee21e98e69f4029d1e417aa085001566be0d322fbc75bc6f29b0050c01 quay.io/ceph/ceph-grafana:8.3.5 2024-04-14T14:05:34.367 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Apr 14 14:05:34 smithi122 podman[50905]: 2024-04-14 14:05:34.365713161 +0000 UTC m=+0.148512086 container create 3181da4f7f029d8b9d923f7024ba5873c75e3ba07e7e0a24a128c8e3ee22d67c (image=quay.io/ceph/ceph-grafana:8.3.5, name=ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-grafana-a, description=Ceph Grafana Container, release=236.1648460182, build-date=2022-03-28T10:36:18.413762, io.buildah.version=1.24.2, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, version=8.5, maintainer=Paul Cuzner , vcs-type=git, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, io.openshift.expose-services=, summary=Grafana Container configured for Ceph mgr/dashboard integration, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, name=ubi8, io.k8s.display-name=Red Hat Universal Base Image 8, vendor=Red Hat, Inc., com.redhat.component=ubi8-container, architecture=x86_64, distribution-scope=public, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-236.1648460182, io.openshift.tags=base rhel8, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly.) 2024-04-14T14:05:35.067 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:05:34 smithi069 ceph-mon[24818]: pgmap v58: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:05:35.067 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:05:34 smithi069 ceph-mon[29472]: pgmap v58: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:05:35.094 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:05:34 smithi122 ceph-mon[25721]: pgmap v58: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:05:35.095 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Apr 14 14:05:34 smithi122 podman[50905]: 2024-04-14 14:05:34.619043424 +0000 UTC m=+0.401842350 container init 3181da4f7f029d8b9d923f7024ba5873c75e3ba07e7e0a24a128c8e3ee22d67c (image=quay.io/ceph/ceph-grafana:8.3.5, name=ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-grafana-a, maintainer=Paul Cuzner , build-date=2022-03-28T10:36:18.413762, io.openshift.expose-services=, version=8.5, vendor=Red Hat, Inc., io.openshift.tags=base rhel8, com.redhat.component=ubi8-container, architecture=x86_64, name=ubi8, io.k8s.display-name=Red Hat Universal Base Image 8, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-236.1648460182, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, distribution-scope=public, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, io.buildah.version=1.24.2, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., description=Ceph Grafana Container, summary=Grafana Container configured for Ceph mgr/dashboard integration, release=236.1648460182, vcs-type=git) 2024-04-14T14:05:35.095 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Apr 14 14:05:34 smithi122 podman[50905]: 2024-04-14 14:05:34.621832854 +0000 UTC m=+0.404631789 container start 3181da4f7f029d8b9d923f7024ba5873c75e3ba07e7e0a24a128c8e3ee22d67c (image=quay.io/ceph/ceph-grafana:8.3.5, name=ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-grafana-a, version=8.5, release=236.1648460182, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, distribution-scope=public, io.buildah.version=1.24.2, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, io.openshift.expose-services=, summary=Grafana Container configured for Ceph mgr/dashboard integration, com.redhat.component=ubi8-container, vcs-type=git, io.k8s.display-name=Red Hat Universal Base Image 8, architecture=x86_64, build-date=2022-03-28T10:36:18.413762, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-236.1648460182, io.openshift.tags=base rhel8, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., maintainer=Paul Cuzner , description=Ceph Grafana Container, name=ubi8, vendor=Red Hat, Inc.) 2024-04-14T14:05:35.095 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Apr 14 14:05:34 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-grafana-a[50919]: Failed to start grafana. error: failed to create log directory "/usr/share/grafana/data/log": mkdir /usr/share/grafana/data: permission denied 2024-04-14T14:05:35.095 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Apr 14 14:05:34 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-grafana-a[50919]: failed to create log directory "/usr/share/grafana/data/log": mkdir /usr/share/grafana/data: permission denied 2024-04-14T14:05:35.095 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Apr 14 14:05:34 smithi122 bash[50905]: 3181da4f7f029d8b9d923f7024ba5873c75e3ba07e7e0a24a128c8e3ee22d67c 2024-04-14T14:05:35.095 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Apr 14 14:05:34 smithi122 systemd[1]: Started Ceph grafana.a for dc326fc2-fa65-11ee-bc8e-c7b262605968. 2024-04-14T14:05:35.095 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Apr 14 14:05:34 smithi122 podman[50934]: 2024-04-14 14:05:34.726594829 +0000 UTC m=+0.072464698 container died 3181da4f7f029d8b9d923f7024ba5873c75e3ba07e7e0a24a128c8e3ee22d67c (image=quay.io/ceph/ceph-grafana:8.3.5, name=ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-grafana-a, distribution-scope=public, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-236.1648460182, summary=Grafana Container configured for Ceph mgr/dashboard integration, vcs-type=git, io.k8s.display-name=Red Hat Universal Base Image 8, version=8.5, com.redhat.component=ubi8-container, vendor=Red Hat, Inc., com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, build-date=2022-03-28T10:36:18.413762, io.buildah.version=1.24.2, io.openshift.expose-services=, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, maintainer=Paul Cuzner , name=ubi8, release=236.1648460182, io.openshift.tags=base rhel8, description=Ceph Grafana Container, architecture=x86_64) 2024-04-14T14:05:35.594 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Apr 14 14:05:35 smithi122 podman[50934]: 2024-04-14 14:05:35.247100822 +0000 UTC m=+0.592970757 container remove 3181da4f7f029d8b9d923f7024ba5873c75e3ba07e7e0a24a128c8e3ee22d67c (image=quay.io/ceph/ceph-grafana:8.3.5, name=ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-grafana-a, vcs-type=git, distribution-scope=public, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-236.1648460182, io.openshift.tags=base rhel8, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, release=236.1648460182, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, com.redhat.component=ubi8-container, vendor=Red Hat, Inc., build-date=2022-03-28T10:36:18.413762, io.buildah.version=1.24.2, name=ubi8, description=Ceph Grafana Container, maintainer=Paul Cuzner , summary=Grafana Container configured for Ceph mgr/dashboard integration, architecture=x86_64, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., io.k8s.display-name=Red Hat Universal Base Image 8, io.openshift.expose-services=, version=8.5) 2024-04-14T14:05:35.595 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Apr 14 14:05:35 smithi122 systemd[1]: ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968@grafana.a.service: Main process exited, code=exited, status=1/FAILURE 2024-04-14T14:05:35.595 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Apr 14 14:05:35 smithi122 systemd[1]: ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968@grafana.a.service: Failed with result 'exit-code'. 2024-04-14T14:05:37.066 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:05:36 smithi069 ceph-mon[24818]: pgmap v59: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:05:37.067 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:05:36 smithi069 ceph-mon[29472]: pgmap v59: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:05:37.094 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:05:36 smithi122 ceph-mon[25721]: pgmap v59: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:05:39.066 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:05:38 smithi069 ceph-mon[24818]: pgmap v60: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:05:39.067 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:05:38 smithi069 ceph-mon[24818]: from='client.25048 -' entity='client.iscsi.foo.smithi069.yuumtr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T14:05:39.067 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:05:38 smithi069 ceph-mon[29472]: pgmap v60: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:05:39.067 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:05:38 smithi069 ceph-mon[29472]: from='client.25048 -' entity='client.iscsi.foo.smithi069.yuumtr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T14:05:39.094 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:05:38 smithi122 ceph-mon[25721]: pgmap v60: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:05:39.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:05:38 smithi122 ceph-mon[25721]: from='client.25048 -' entity='client.iscsi.foo.smithi069.yuumtr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T14:05:39.095 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Apr 14 14:05:38 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-prometheus-a[49277]: ts=2024-04-14T14:05:38.660Z caller=manager.go:609 level=warn component="rule manager" group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on(ceph_daemon) group_left(hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.69\", device_class=\"ssd\", hostname=\"smithi069\", instance=\"172.21.15.122:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.69\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster=\"dc326fc2-fa65-11ee-bc8e-c7b262605968\", cluster_addr=\"172.21.15.69\", device_class=\"ssd\", hostname=\"smithi069\", instance=\"smithi122:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.69\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-04-14T14:05:41.066 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:05:40 smithi069 ceph-mon[24818]: pgmap v61: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:05:41.067 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:05:40 smithi069 ceph-mon[29472]: pgmap v61: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:05:41.094 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:05:40 smithi122 ceph-mon[25721]: pgmap v61: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:05:43.067 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:05:42 smithi069 ceph-mon[24818]: pgmap v62: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:05:43.067 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:05:42 smithi069 ceph-mon[24818]: from='mgr.24749 172.21.15.122:0/1892077634' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T14:05:43.067 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:05:42 smithi069 ceph-mon[29472]: pgmap v62: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:05:43.067 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:05:42 smithi069 ceph-mon[29472]: from='mgr.24749 172.21.15.122:0/1892077634' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T14:05:43.094 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:05:42 smithi122 ceph-mon[25721]: pgmap v62: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:05:43.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:05:42 smithi122 ceph-mon[25721]: from='mgr.24749 172.21.15.122:0/1892077634' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T14:05:43.095 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Apr 14 14:05:42 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-x[45487]: ::ffff:172.21.15.122 - - [14/Apr/2024:14:05:42] "GET /metrics HTTP/1.1" 200 37963 "" "Prometheus/2.33.4" 2024-04-14T14:05:43.845 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Apr 14 14:05:43 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-prometheus-a[49277]: ts=2024-04-14T14:05:43.504Z caller=manager.go:609 level=warn component="rule manager" group=nodes msg="Evaluating rule failed" rule="alert: CephNodeDiskspaceWarning\nexpr: predict_linear(node_filesystem_free_bytes{device=~\"/.*\"}[2d], 3600 * 24 * 5)\n * on(instance) group_left(nodename) node_uname_info < 0\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.8.4\n severity: warning\n type: ceph_default\nannotations:\n description: Mountpoint {{ $labels.mountpoint }} on {{ $labels.nodename }} will\n be full in less than 5 days based on the 48 hour trailing fill rate.\n summary: Host filesystem free space is getting low\n" err="found duplicate series for the match group {instance=\"smithi069\"} on the right hand-side of the operation: [{__name__=\"node_uname_info\", domainname=\"(none)\", instance=\"smithi069\", job=\"node\", machine=\"x86_64\", nodename=\"smithi069\", release=\"5.14.0-435.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Fri Mar 29 18:46:37 UTC 2024\"}, {__name__=\"node_uname_info\", cluster=\"dc326fc2-fa65-11ee-bc8e-c7b262605968\", domainname=\"(none)\", instance=\"smithi069\", job=\"node\", machine=\"x86_64\", nodename=\"smithi069\", release=\"5.14.0-435.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Fri Mar 29 18:46:37 UTC 2024\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-04-14T14:05:45.070 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:05:44 smithi069 ceph-mon[24818]: pgmap v63: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:05:45.070 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:05:44 smithi069 ceph-mon[29472]: pgmap v63: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:05:45.094 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:05:44 smithi122 ceph-mon[25721]: pgmap v63: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:05:45.845 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Apr 14 14:05:45 smithi122 systemd[1]: ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968@grafana.a.service: Scheduled restart job, restart counter is at 5. 2024-04-14T14:05:45.845 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Apr 14 14:05:45 smithi122 systemd[1]: Stopped Ceph grafana.a for dc326fc2-fa65-11ee-bc8e-c7b262605968. 2024-04-14T14:05:45.845 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Apr 14 14:05:45 smithi122 systemd[1]: ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968@grafana.a.service: Start request repeated too quickly. 2024-04-14T14:05:45.845 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Apr 14 14:05:45 smithi122 systemd[1]: ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968@grafana.a.service: Failed with result 'exit-code'. 2024-04-14T14:05:45.845 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Apr 14 14:05:45 smithi122 systemd[1]: Failed to start Ceph grafana.a for dc326fc2-fa65-11ee-bc8e-c7b262605968. 2024-04-14T14:05:47.066 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:05:46 smithi069 ceph-mon[24818]: pgmap v64: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:05:47.067 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:05:46 smithi069 ceph-mon[29472]: pgmap v64: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:05:47.094 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:05:46 smithi122 ceph-mon[25721]: pgmap v64: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:05:49.066 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:05:48 smithi069 ceph-mon[24818]: pgmap v65: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:05:49.067 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:05:48 smithi069 ceph-mon[24818]: from='client.25048 -' entity='client.iscsi.foo.smithi069.yuumtr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T14:05:49.067 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:05:48 smithi069 ceph-mon[29472]: pgmap v65: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:05:49.067 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:05:48 smithi069 ceph-mon[29472]: from='client.25048 -' entity='client.iscsi.foo.smithi069.yuumtr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T14:05:49.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:05:48 smithi122 ceph-mon[25721]: pgmap v65: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:05:49.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:05:48 smithi122 ceph-mon[25721]: from='client.25048 -' entity='client.iscsi.foo.smithi069.yuumtr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T14:05:49.095 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Apr 14 14:05:48 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-prometheus-a[49277]: ts=2024-04-14T14:05:48.659Z caller=manager.go:609 level=warn component="rule manager" group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on(ceph_daemon) group_left(hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.69\", device_class=\"ssd\", hostname=\"smithi069\", instance=\"172.21.15.122:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.69\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster=\"dc326fc2-fa65-11ee-bc8e-c7b262605968\", cluster_addr=\"172.21.15.69\", device_class=\"ssd\", hostname=\"smithi069\", instance=\"smithi122:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.69\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-04-14T14:05:51.067 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:05:50 smithi069 ceph-mon[24818]: pgmap v66: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:05:51.067 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:05:50 smithi069 ceph-mon[29472]: pgmap v66: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:05:51.094 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:05:50 smithi122 ceph-mon[25721]: pgmap v66: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:05:53.067 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:05:52 smithi069 ceph-mon[24818]: pgmap v67: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:05:53.067 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:05:52 smithi069 ceph-mon[29472]: pgmap v67: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:05:53.094 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:05:52 smithi122 ceph-mon[25721]: pgmap v67: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:05:53.095 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Apr 14 14:05:52 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-x[45487]: ::ffff:172.21.15.122 - - [14/Apr/2024:14:05:52] "GET /metrics HTTP/1.1" 200 37963 "" "Prometheus/2.33.4" 2024-04-14T14:05:53.845 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Apr 14 14:05:53 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-prometheus-a[49277]: ts=2024-04-14T14:05:53.504Z caller=manager.go:609 level=warn component="rule manager" group=nodes msg="Evaluating rule failed" rule="alert: CephNodeDiskspaceWarning\nexpr: predict_linear(node_filesystem_free_bytes{device=~\"/.*\"}[2d], 3600 * 24 * 5)\n * on(instance) group_left(nodename) node_uname_info < 0\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.8.4\n severity: warning\n type: ceph_default\nannotations:\n description: Mountpoint {{ $labels.mountpoint }} on {{ $labels.nodename }} will\n be full in less than 5 days based on the 48 hour trailing fill rate.\n summary: Host filesystem free space is getting low\n" err="found duplicate series for the match group {instance=\"smithi069\"} on the right hand-side of the operation: [{__name__=\"node_uname_info\", domainname=\"(none)\", instance=\"smithi069\", job=\"node\", machine=\"x86_64\", nodename=\"smithi069\", release=\"5.14.0-435.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Fri Mar 29 18:46:37 UTC 2024\"}, {__name__=\"node_uname_info\", cluster=\"dc326fc2-fa65-11ee-bc8e-c7b262605968\", domainname=\"(none)\", instance=\"smithi069\", job=\"node\", machine=\"x86_64\", nodename=\"smithi069\", release=\"5.14.0-435.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Fri Mar 29 18:46:37 UTC 2024\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-04-14T14:05:55.067 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:05:54 smithi069 ceph-mon[24818]: pgmap v68: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:05:55.067 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:05:54 smithi069 ceph-mon[29472]: pgmap v68: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:05:55.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:05:54 smithi122 ceph-mon[25721]: pgmap v68: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:05:57.066 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:05:56 smithi069 ceph-mon[24818]: pgmap v69: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:05:57.067 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:05:56 smithi069 ceph-mon[29472]: pgmap v69: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:05:57.094 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:05:56 smithi122 ceph-mon[25721]: pgmap v69: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:05:58.066 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:05:57 smithi069 ceph-mon[24818]: from='mgr.24749 172.21.15.122:0/1892077634' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T14:05:58.067 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:05:57 smithi069 ceph-mon[29472]: from='mgr.24749 172.21.15.122:0/1892077634' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T14:05:58.094 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:05:57 smithi122 ceph-mon[25721]: from='mgr.24749 172.21.15.122:0/1892077634' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T14:05:58.740 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:05:58 smithi069 ceph-mon[24818]: pgmap v70: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:05:58.740 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:05:58 smithi069 ceph-mon[24818]: from='client.25048 -' entity='client.iscsi.foo.smithi069.yuumtr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T14:05:58.740 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:05:58 smithi069 ceph-mon[29472]: pgmap v70: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:05:58.741 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:05:58 smithi069 ceph-mon[29472]: from='client.25048 -' entity='client.iscsi.foo.smithi069.yuumtr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T14:05:58.932 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:05:58 smithi122 ceph-mon[25721]: pgmap v70: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:05:58.932 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:05:58 smithi122 ceph-mon[25721]: from='client.25048 -' entity='client.iscsi.foo.smithi069.yuumtr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T14:05:58.932 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Apr 14 14:05:58 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-prometheus-a[49277]: ts=2024-04-14T14:05:58.660Z caller=manager.go:609 level=warn component="rule manager" group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on(ceph_daemon) group_left(hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.69\", device_class=\"ssd\", hostname=\"smithi069\", instance=\"172.21.15.122:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.69\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster=\"dc326fc2-fa65-11ee-bc8e-c7b262605968\", cluster_addr=\"172.21.15.69\", device_class=\"ssd\", hostname=\"smithi069\", instance=\"smithi122:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.69\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-04-14T14:06:01.067 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:06:00 smithi069 ceph-mon[24818]: pgmap v71: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:06:01.067 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:06:00 smithi069 ceph-mon[24818]: from='mgr.24749 ' entity='mgr.x' 2024-04-14T14:06:01.067 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:06:00 smithi069 ceph-mon[24818]: from='mgr.24749 ' entity='mgr.x' 2024-04-14T14:06:01.067 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:06:00 smithi069 ceph-mon[24818]: from='mgr.24749 ' entity='mgr.x' 2024-04-14T14:06:01.067 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:06:00 smithi069 ceph-mon[24818]: from='mgr.24749 ' entity='mgr.x' 2024-04-14T14:06:01.067 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:06:00 smithi069 ceph-mon[29472]: pgmap v71: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:06:01.067 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:06:00 smithi069 ceph-mon[29472]: from='mgr.24749 ' entity='mgr.x' 2024-04-14T14:06:01.067 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:06:00 smithi069 ceph-mon[29472]: from='mgr.24749 ' entity='mgr.x' 2024-04-14T14:06:01.067 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:06:00 smithi069 ceph-mon[29472]: from='mgr.24749 ' entity='mgr.x' 2024-04-14T14:06:01.067 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:06:00 smithi069 ceph-mon[29472]: from='mgr.24749 ' entity='mgr.x' 2024-04-14T14:06:01.094 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:06:00 smithi122 ceph-mon[25721]: pgmap v71: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:06:01.094 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:06:00 smithi122 ceph-mon[25721]: from='mgr.24749 ' entity='mgr.x' 2024-04-14T14:06:01.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:06:00 smithi122 ceph-mon[25721]: from='mgr.24749 ' entity='mgr.x' 2024-04-14T14:06:01.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:06:00 smithi122 ceph-mon[25721]: from='mgr.24749 ' entity='mgr.x' 2024-04-14T14:06:01.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:06:00 smithi122 ceph-mon[25721]: from='mgr.24749 ' entity='mgr.x' 2024-04-14T14:06:01.920 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:06:01 smithi069 ceph-mon[29472]: from='mgr.24749 172.21.15.122:0/1892077634' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T14:06:01.921 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:06:01 smithi069 ceph-mon[29472]: from='mgr.24749 172.21.15.122:0/1892077634' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-14T14:06:01.921 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:06:01 smithi069 ceph-mon[29472]: from='mgr.24749 ' entity='mgr.x' 2024-04-14T14:06:01.921 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:06:01 smithi069 ceph-mon[29472]: from='mgr.24749 172.21.15.122:0/1892077634' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi069.yuumtr", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-04-14T14:06:01.921 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:06:01 smithi069 ceph-mon[29472]: from='mgr.24749 ' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi069.yuumtr", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-04-14T14:06:01.921 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:06:01 smithi069 ceph-mon[29472]: from='mgr.24749 172.21.15.122:0/1892077634' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T14:06:01.921 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:06:01 smithi069 ceph-mon[24818]: from='mgr.24749 172.21.15.122:0/1892077634' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T14:06:01.921 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:06:01 smithi069 ceph-mon[24818]: from='mgr.24749 172.21.15.122:0/1892077634' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-14T14:06:01.921 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:06:01 smithi069 ceph-mon[24818]: from='mgr.24749 ' entity='mgr.x' 2024-04-14T14:06:01.921 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:06:01 smithi069 ceph-mon[24818]: from='mgr.24749 172.21.15.122:0/1892077634' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi069.yuumtr", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-04-14T14:06:01.921 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:06:01 smithi069 ceph-mon[24818]: from='mgr.24749 ' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi069.yuumtr", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-04-14T14:06:01.922 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:06:01 smithi069 ceph-mon[24818]: from='mgr.24749 172.21.15.122:0/1892077634' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T14:06:02.094 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:06:01 smithi122 ceph-mon[25721]: from='mgr.24749 172.21.15.122:0/1892077634' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T14:06:02.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:06:01 smithi122 ceph-mon[25721]: from='mgr.24749 172.21.15.122:0/1892077634' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-14T14:06:02.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:06:01 smithi122 ceph-mon[25721]: from='mgr.24749 ' entity='mgr.x' 2024-04-14T14:06:02.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:06:01 smithi122 ceph-mon[25721]: from='mgr.24749 172.21.15.122:0/1892077634' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi069.yuumtr", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-04-14T14:06:02.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:06:01 smithi122 ceph-mon[25721]: from='mgr.24749 ' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi069.yuumtr", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-04-14T14:06:02.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:06:01 smithi122 ceph-mon[25721]: from='mgr.24749 172.21.15.122:0/1892077634' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T14:06:02.886 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:06:02 smithi069 ceph-mon[29472]: pgmap v72: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 883 B/s rd, 0 op/s 2024-04-14T14:06:02.887 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:06:02 smithi069 ceph-mon[29472]: pgmap v73: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-04-14T14:06:02.887 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:06:02 smithi069 ceph-mon[29472]: Reconfiguring iscsi.foo.smithi069.yuumtr (dependencies changed)... 2024-04-14T14:06:02.887 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:06:02 smithi069 ceph-mon[29472]: Reconfiguring daemon iscsi.foo.smithi069.yuumtr on smithi069 2024-04-14T14:06:02.887 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:06:02 smithi069 ceph-mon[24818]: pgmap v72: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 883 B/s rd, 0 op/s 2024-04-14T14:06:02.887 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:06:02 smithi069 ceph-mon[24818]: pgmap v73: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-04-14T14:06:02.887 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:06:02 smithi069 ceph-mon[24818]: Reconfiguring iscsi.foo.smithi069.yuumtr (dependencies changed)... 2024-04-14T14:06:02.887 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:06:02 smithi069 ceph-mon[24818]: Reconfiguring daemon iscsi.foo.smithi069.yuumtr on smithi069 2024-04-14T14:06:03.094 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Apr 14 14:06:02 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-x[45487]: ::ffff:172.21.15.122 - - [14/Apr/2024:14:06:02] "GET /metrics HTTP/1.1" 200 37963 "" "Prometheus/2.33.4" 2024-04-14T14:06:03.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:06:02 smithi122 ceph-mon[25721]: pgmap v72: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 883 B/s rd, 0 op/s 2024-04-14T14:06:03.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:06:02 smithi122 ceph-mon[25721]: pgmap v73: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-04-14T14:06:03.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:06:02 smithi122 ceph-mon[25721]: Reconfiguring iscsi.foo.smithi069.yuumtr (dependencies changed)... 2024-04-14T14:06:03.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:06:02 smithi122 ceph-mon[25721]: Reconfiguring daemon iscsi.foo.smithi069.yuumtr on smithi069 2024-04-14T14:06:03.845 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Apr 14 14:06:03 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-prometheus-a[49277]: ts=2024-04-14T14:06:03.504Z caller=manager.go:609 level=warn component="rule manager" group=nodes msg="Evaluating rule failed" rule="alert: CephNodeDiskspaceWarning\nexpr: predict_linear(node_filesystem_free_bytes{device=~\"/.*\"}[2d], 3600 * 24 * 5)\n * on(instance) group_left(nodename) node_uname_info < 0\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.8.4\n severity: warning\n type: ceph_default\nannotations:\n description: Mountpoint {{ $labels.mountpoint }} on {{ $labels.nodename }} will\n be full in less than 5 days based on the 48 hour trailing fill rate.\n summary: Host filesystem free space is getting low\n" err="found duplicate series for the match group {instance=\"smithi069\"} on the right hand-side of the operation: [{__name__=\"node_uname_info\", domainname=\"(none)\", instance=\"smithi069\", job=\"node\", machine=\"x86_64\", nodename=\"smithi069\", release=\"5.14.0-435.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Fri Mar 29 18:46:37 UTC 2024\"}, {__name__=\"node_uname_info\", cluster=\"dc326fc2-fa65-11ee-bc8e-c7b262605968\", domainname=\"(none)\", instance=\"smithi069\", job=\"node\", machine=\"x86_64\", nodename=\"smithi069\", release=\"5.14.0-435.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Fri Mar 29 18:46:37 UTC 2024\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-04-14T14:06:04.881 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:06:04 smithi069 ceph-mon[24818]: pgmap v74: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-04-14T14:06:04.881 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:06:04 smithi069 ceph-mon[29472]: pgmap v74: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-04-14T14:06:05.094 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:06:04 smithi122 ceph-mon[25721]: pgmap v74: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-04-14T14:06:06.817 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:06:06 smithi069 ceph-mon[24818]: pgmap v75: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-04-14T14:06:06.817 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:06:06 smithi069 ceph-mon[24818]: from='mgr.24749 ' entity='mgr.x' 2024-04-14T14:06:06.817 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:06:06 smithi069 ceph-mon[24818]: from='mgr.24749 ' entity='mgr.x' 2024-04-14T14:06:06.817 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:06:06 smithi069 ceph-mon[24818]: from='mgr.24749 172.21.15.122:0/1892077634' entity='mgr.x' cmd=[{"prefix": "dashboard iscsi-gateway-list"}]: dispatch 2024-04-14T14:06:06.817 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:06:06 smithi069 ceph-mon[24818]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard iscsi-gateway-list"}]: dispatch 2024-04-14T14:06:06.817 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:06:06 smithi069 ceph-mon[24818]: from='mgr.24749 172.21.15.122:0/1892077634' entity='mgr.x' cmd=[{"prefix": "dashboard set-iscsi-api-ssl-verification", "value": "true"}]: dispatch 2024-04-14T14:06:06.817 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:06:06 smithi069 ceph-mon[24818]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-iscsi-api-ssl-verification", "value": "true"}]: dispatch 2024-04-14T14:06:06.817 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:06:06 smithi069 ceph-mon[24818]: from='client.? 172.21.15.69:0/450149334' entity='client.iscsi.foo.smithi069.yuumtr' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-04-14T14:06:06.817 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:06:06 smithi069 ceph-mon[29472]: pgmap v75: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-04-14T14:06:06.818 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:06:06 smithi069 ceph-mon[29472]: from='mgr.24749 ' entity='mgr.x' 2024-04-14T14:06:06.818 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:06:06 smithi069 ceph-mon[29472]: from='mgr.24749 ' entity='mgr.x' 2024-04-14T14:06:06.818 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:06:06 smithi069 ceph-mon[29472]: from='mgr.24749 172.21.15.122:0/1892077634' entity='mgr.x' cmd=[{"prefix": "dashboard iscsi-gateway-list"}]: dispatch 2024-04-14T14:06:06.818 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:06:06 smithi069 ceph-mon[29472]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard iscsi-gateway-list"}]: dispatch 2024-04-14T14:06:06.818 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:06:06 smithi069 ceph-mon[29472]: from='mgr.24749 172.21.15.122:0/1892077634' entity='mgr.x' cmd=[{"prefix": "dashboard set-iscsi-api-ssl-verification", "value": "true"}]: dispatch 2024-04-14T14:06:06.818 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:06:06 smithi069 ceph-mon[29472]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-iscsi-api-ssl-verification", "value": "true"}]: dispatch 2024-04-14T14:06:06.818 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:06:06 smithi069 ceph-mon[29472]: from='client.? 172.21.15.69:0/450149334' entity='client.iscsi.foo.smithi069.yuumtr' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-04-14T14:06:06.844 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:06:06 smithi122 ceph-mon[25721]: pgmap v75: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-04-14T14:06:06.844 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:06:06 smithi122 ceph-mon[25721]: from='mgr.24749 ' entity='mgr.x' 2024-04-14T14:06:06.844 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:06:06 smithi122 ceph-mon[25721]: from='mgr.24749 ' entity='mgr.x' 2024-04-14T14:06:06.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:06:06 smithi122 ceph-mon[25721]: from='mgr.24749 172.21.15.122:0/1892077634' entity='mgr.x' cmd=[{"prefix": "dashboard iscsi-gateway-list"}]: dispatch 2024-04-14T14:06:06.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:06:06 smithi122 ceph-mon[25721]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard iscsi-gateway-list"}]: dispatch 2024-04-14T14:06:06.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:06:06 smithi122 ceph-mon[25721]: from='mgr.24749 172.21.15.122:0/1892077634' entity='mgr.x' cmd=[{"prefix": "dashboard set-iscsi-api-ssl-verification", "value": "true"}]: dispatch 2024-04-14T14:06:06.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:06:06 smithi122 ceph-mon[25721]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-iscsi-api-ssl-verification", "value": "true"}]: dispatch 2024-04-14T14:06:06.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:06:06 smithi122 ceph-mon[25721]: from='client.? 172.21.15.69:0/450149334' entity='client.iscsi.foo.smithi069.yuumtr' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-04-14T14:06:08.566 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:06:08 smithi069 ceph-mon[24818]: pgmap v76: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-04-14T14:06:08.567 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:06:08 smithi069 ceph-mon[29472]: pgmap v76: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-04-14T14:06:08.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:06:08 smithi122 ceph-mon[25721]: pgmap v76: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-04-14T14:06:08.845 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Apr 14 14:06:08 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-prometheus-a[49277]: ts=2024-04-14T14:06:08.660Z caller=manager.go:609 level=warn component="rule manager" group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on(ceph_daemon) group_left(hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.69\", device_class=\"ssd\", hostname=\"smithi069\", instance=\"172.21.15.122:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.69\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster=\"dc326fc2-fa65-11ee-bc8e-c7b262605968\", cluster_addr=\"172.21.15.69\", device_class=\"ssd\", hostname=\"smithi069\", instance=\"smithi122:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.69\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-04-14T14:06:10.567 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:06:10 smithi069 ceph-mon[24818]: pgmap v77: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 960 B/s rd, 1 op/s 2024-04-14T14:06:10.567 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:06:10 smithi069 ceph-mon[24818]: from='mgr.24749 ' entity='mgr.x' 2024-04-14T14:06:10.567 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:06:10 smithi069 ceph-mon[24818]: from='mgr.24749 ' entity='mgr.x' 2024-04-14T14:06:10.567 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:06:10 smithi069 ceph-mon[24818]: from='mgr.24749 ' entity='mgr.x' 2024-04-14T14:06:10.567 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:06:10 smithi069 ceph-mon[24818]: from='mgr.24749 ' entity='mgr.x' 2024-04-14T14:06:10.567 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:06:10 smithi069 ceph-mon[24818]: from='mgr.24749 172.21.15.122:0/1892077634' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T14:06:10.567 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:06:10 smithi069 ceph-mon[24818]: from='mgr.24749 172.21.15.122:0/1892077634' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-14T14:06:10.567 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:06:10 smithi069 ceph-mon[24818]: pgmap v78: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-04-14T14:06:10.567 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:06:10 smithi069 ceph-mon[24818]: pgmap v79: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-04-14T14:06:10.567 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:06:10 smithi069 ceph-mon[24818]: from='mgr.24749 ' entity='mgr.x' 2024-04-14T14:06:10.568 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:06:10 smithi069 ceph-mon[29472]: pgmap v77: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 960 B/s rd, 1 op/s 2024-04-14T14:06:10.568 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:06:10 smithi069 ceph-mon[29472]: from='mgr.24749 ' entity='mgr.x' 2024-04-14T14:06:10.568 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:06:10 smithi069 ceph-mon[29472]: from='mgr.24749 ' entity='mgr.x' 2024-04-14T14:06:10.568 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:06:10 smithi069 ceph-mon[29472]: from='mgr.24749 ' entity='mgr.x' 2024-04-14T14:06:10.568 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:06:10 smithi069 ceph-mon[29472]: from='mgr.24749 ' entity='mgr.x' 2024-04-14T14:06:10.568 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:06:10 smithi069 ceph-mon[29472]: from='mgr.24749 172.21.15.122:0/1892077634' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T14:06:10.568 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:06:10 smithi069 ceph-mon[29472]: from='mgr.24749 172.21.15.122:0/1892077634' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-14T14:06:10.568 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:06:10 smithi069 ceph-mon[29472]: pgmap v78: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-04-14T14:06:10.568 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:06:10 smithi069 ceph-mon[29472]: pgmap v79: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-04-14T14:06:10.568 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:06:10 smithi069 ceph-mon[29472]: from='mgr.24749 ' entity='mgr.x' 2024-04-14T14:06:10.595 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:06:10 smithi122 ceph-mon[25721]: pgmap v77: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 960 B/s rd, 1 op/s 2024-04-14T14:06:10.595 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:06:10 smithi122 ceph-mon[25721]: from='mgr.24749 ' entity='mgr.x' 2024-04-14T14:06:10.595 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:06:10 smithi122 ceph-mon[25721]: from='mgr.24749 ' entity='mgr.x' 2024-04-14T14:06:10.595 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:06:10 smithi122 ceph-mon[25721]: from='mgr.24749 ' entity='mgr.x' 2024-04-14T14:06:10.595 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:06:10 smithi122 ceph-mon[25721]: from='mgr.24749 ' entity='mgr.x' 2024-04-14T14:06:10.595 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:06:10 smithi122 ceph-mon[25721]: from='mgr.24749 172.21.15.122:0/1892077634' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T14:06:10.596 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:06:10 smithi122 ceph-mon[25721]: from='mgr.24749 172.21.15.122:0/1892077634' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-14T14:06:10.596 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:06:10 smithi122 ceph-mon[25721]: pgmap v78: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-04-14T14:06:10.596 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:06:10 smithi122 ceph-mon[25721]: pgmap v79: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-04-14T14:06:10.596 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:06:10 smithi122 ceph-mon[25721]: from='mgr.24749 ' entity='mgr.x' 2024-04-14T14:06:12.802 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:06:12 smithi122 ceph-mon[25721]: pgmap v80: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-04-14T14:06:12.816 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:06:12 smithi069 ceph-mon[24818]: pgmap v80: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-04-14T14:06:12.817 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:06:12 smithi069 ceph-mon[29472]: pgmap v80: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-04-14T14:06:13.095 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Apr 14 14:06:12 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-x[45487]: ::ffff:172.21.15.122 - - [14/Apr/2024:14:06:12] "GET /metrics HTTP/1.1" 200 37977 "" "Prometheus/2.33.4" 2024-04-14T14:06:13.817 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:06:13 smithi069 ceph-mon[24818]: from='mgr.24749 172.21.15.122:0/1892077634' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T14:06:13.817 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:06:13 smithi069 ceph-mon[29472]: from='mgr.24749 172.21.15.122:0/1892077634' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T14:06:13.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:06:13 smithi122 ceph-mon[25721]: from='mgr.24749 172.21.15.122:0/1892077634' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T14:06:13.845 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Apr 14 14:06:13 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-prometheus-a[49277]: ts=2024-04-14T14:06:13.504Z caller=manager.go:609 level=warn component="rule manager" group=nodes msg="Evaluating rule failed" rule="alert: CephNodeDiskspaceWarning\nexpr: predict_linear(node_filesystem_free_bytes{device=~\"/.*\"}[2d], 3600 * 24 * 5)\n * on(instance) group_left(nodename) node_uname_info < 0\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.8.4\n severity: warning\n type: ceph_default\nannotations:\n description: Mountpoint {{ $labels.mountpoint }} on {{ $labels.nodename }} will\n be full in less than 5 days based on the 48 hour trailing fill rate.\n summary: Host filesystem free space is getting low\n" err="found duplicate series for the match group {instance=\"smithi069\"} on the right hand-side of the operation: [{__name__=\"node_uname_info\", domainname=\"(none)\", instance=\"smithi069\", job=\"node\", machine=\"x86_64\", nodename=\"smithi069\", release=\"5.14.0-435.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Fri Mar 29 18:46:37 UTC 2024\"}, {__name__=\"node_uname_info\", cluster=\"dc326fc2-fa65-11ee-bc8e-c7b262605968\", domainname=\"(none)\", instance=\"smithi069\", job=\"node\", machine=\"x86_64\", nodename=\"smithi069\", release=\"5.14.0-435.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Fri Mar 29 18:46:37 UTC 2024\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-04-14T14:06:14.718 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:06:14 smithi069 ceph-mon[24818]: pgmap v81: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 368 B/s rd, 0 op/s 2024-04-14T14:06:14.718 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:06:14 smithi069 ceph-mon[29472]: pgmap v81: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 368 B/s rd, 0 op/s 2024-04-14T14:06:14.844 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:06:14 smithi122 ceph-mon[25721]: pgmap v81: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 368 B/s rd, 0 op/s 2024-04-14T14:06:16.817 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:06:16 smithi069 ceph-mon[24818]: pgmap v82: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 983 B/s rd, 1 op/s 2024-04-14T14:06:16.817 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:06:16 smithi069 ceph-mon[24818]: from='client.15177 -' entity='client.iscsi.foo.smithi069.yuumtr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T14:06:16.817 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:06:16 smithi069 ceph-mon[29472]: pgmap v82: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 983 B/s rd, 1 op/s 2024-04-14T14:06:16.817 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:06:16 smithi069 ceph-mon[29472]: from='client.15177 -' entity='client.iscsi.foo.smithi069.yuumtr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T14:06:16.844 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:06:16 smithi122 ceph-mon[25721]: pgmap v82: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 983 B/s rd, 1 op/s 2024-04-14T14:06:16.844 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:06:16 smithi122 ceph-mon[25721]: from='client.15177 -' entity='client.iscsi.foo.smithi069.yuumtr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T14:06:18.817 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:06:18 smithi069 ceph-mon[24818]: pgmap v83: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 614 B/s rd, 0 op/s 2024-04-14T14:06:18.817 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:06:18 smithi069 ceph-mon[29472]: pgmap v83: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 614 B/s rd, 0 op/s 2024-04-14T14:06:18.844 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:06:18 smithi122 ceph-mon[25721]: pgmap v83: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 614 B/s rd, 0 op/s 2024-04-14T14:06:18.845 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Apr 14 14:06:18 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-prometheus-a[49277]: ts=2024-04-14T14:06:18.660Z caller=manager.go:609 level=warn component="rule manager" group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on(ceph_daemon) group_left(hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.69\", device_class=\"ssd\", hostname=\"smithi069\", instance=\"172.21.15.122:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.69\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster=\"dc326fc2-fa65-11ee-bc8e-c7b262605968\", cluster_addr=\"172.21.15.69\", device_class=\"ssd\", hostname=\"smithi069\", instance=\"smithi122:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.69\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-04-14T14:06:20.817 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:06:20 smithi069 ceph-mon[24818]: pgmap v84: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-14T14:06:20.817 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:06:20 smithi069 ceph-mon[29472]: pgmap v84: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-14T14:06:20.844 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:06:20 smithi122 ceph-mon[25721]: pgmap v84: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-14T14:06:22.802 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:06:22 smithi122 ceph-mon[25721]: pgmap v85: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:06:22.817 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:06:22 smithi069 ceph-mon[24818]: pgmap v85: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:06:22.817 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:06:22 smithi069 ceph-mon[29472]: pgmap v85: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:06:23.094 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Apr 14 14:06:22 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-x[45487]: ::ffff:172.21.15.122 - - [14/Apr/2024:14:06:22] "GET /metrics HTTP/1.1" 200 37977 "" "Prometheus/2.33.4" 2024-04-14T14:06:23.845 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Apr 14 14:06:23 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-prometheus-a[49277]: ts=2024-04-14T14:06:23.504Z caller=manager.go:609 level=warn component="rule manager" group=nodes msg="Evaluating rule failed" rule="alert: CephNodeDiskspaceWarning\nexpr: predict_linear(node_filesystem_free_bytes{device=~\"/.*\"}[2d], 3600 * 24 * 5)\n * on(instance) group_left(nodename) node_uname_info < 0\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.8.4\n severity: warning\n type: ceph_default\nannotations:\n description: Mountpoint {{ $labels.mountpoint }} on {{ $labels.nodename }} will\n be full in less than 5 days based on the 48 hour trailing fill rate.\n summary: Host filesystem free space is getting low\n" err="found duplicate series for the match group {instance=\"smithi069\"} on the right hand-side of the operation: [{__name__=\"node_uname_info\", domainname=\"(none)\", instance=\"smithi069\", job=\"node\", machine=\"x86_64\", nodename=\"smithi069\", release=\"5.14.0-435.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Fri Mar 29 18:46:37 UTC 2024\"}, {__name__=\"node_uname_info\", cluster=\"dc326fc2-fa65-11ee-bc8e-c7b262605968\", domainname=\"(none)\", instance=\"smithi069\", job=\"node\", machine=\"x86_64\", nodename=\"smithi069\", release=\"5.14.0-435.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Fri Mar 29 18:46:37 UTC 2024\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-04-14T14:06:24.768 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:06:24 smithi069 ceph-mon[24818]: pgmap v86: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:06:24.768 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:06:24 smithi069 ceph-mon[29472]: pgmap v86: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:06:24.800 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:06:24 smithi122 ceph-mon[25721]: pgmap v86: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:06:26.817 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:06:26 smithi069 ceph-mon[24818]: pgmap v87: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:06:26.817 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:06:26 smithi069 ceph-mon[24818]: from='client.15177 -' entity='client.iscsi.foo.smithi069.yuumtr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T14:06:26.817 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:06:26 smithi069 ceph-mon[29472]: pgmap v87: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:06:26.817 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:06:26 smithi069 ceph-mon[29472]: from='client.15177 -' entity='client.iscsi.foo.smithi069.yuumtr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T14:06:26.844 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:06:26 smithi122 ceph-mon[25721]: pgmap v87: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:06:26.844 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:06:26 smithi122 ceph-mon[25721]: from='client.15177 -' entity='client.iscsi.foo.smithi069.yuumtr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T14:06:28.817 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:06:28 smithi069 ceph-mon[24818]: pgmap v88: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:06:28.817 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:06:28 smithi069 ceph-mon[24818]: from='mgr.24749 172.21.15.122:0/1892077634' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T14:06:28.817 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:06:28 smithi069 ceph-mon[29472]: pgmap v88: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:06:28.817 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:06:28 smithi069 ceph-mon[29472]: from='mgr.24749 172.21.15.122:0/1892077634' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T14:06:28.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:06:28 smithi122 ceph-mon[25721]: pgmap v88: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:06:28.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:06:28 smithi122 ceph-mon[25721]: from='mgr.24749 172.21.15.122:0/1892077634' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T14:06:28.845 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Apr 14 14:06:28 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-prometheus-a[49277]: ts=2024-04-14T14:06:28.660Z caller=manager.go:609 level=warn component="rule manager" group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on(ceph_daemon) group_left(hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.69\", device_class=\"ssd\", hostname=\"smithi069\", instance=\"172.21.15.122:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.69\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster=\"dc326fc2-fa65-11ee-bc8e-c7b262605968\", cluster_addr=\"172.21.15.69\", device_class=\"ssd\", hostname=\"smithi069\", instance=\"smithi122:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.69\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-04-14T14:06:30.817 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:06:30 smithi069 ceph-mon[24818]: pgmap v89: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:06:30.817 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:06:30 smithi069 ceph-mon[29472]: pgmap v89: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:06:30.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:06:30 smithi122 ceph-mon[25721]: pgmap v89: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:06:32.802 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:06:32 smithi122 ceph-mon[25721]: pgmap v90: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:06:32.816 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:06:32 smithi069 ceph-mon[24818]: pgmap v90: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:06:32.817 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:06:32 smithi069 ceph-mon[29472]: pgmap v90: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:06:33.094 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Apr 14 14:06:32 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-x[45487]: ::ffff:172.21.15.122 - - [14/Apr/2024:14:06:32] "GET /metrics HTTP/1.1" 200 37973 "" "Prometheus/2.33.4" 2024-04-14T14:06:33.845 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Apr 14 14:06:33 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-prometheus-a[49277]: ts=2024-04-14T14:06:33.504Z caller=manager.go:609 level=warn component="rule manager" group=nodes msg="Evaluating rule failed" rule="alert: CephNodeDiskspaceWarning\nexpr: predict_linear(node_filesystem_free_bytes{device=~\"/.*\"}[2d], 3600 * 24 * 5)\n * on(instance) group_left(nodename) node_uname_info < 0\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.8.4\n severity: warning\n type: ceph_default\nannotations:\n description: Mountpoint {{ $labels.mountpoint }} on {{ $labels.nodename }} will\n be full in less than 5 days based on the 48 hour trailing fill rate.\n summary: Host filesystem free space is getting low\n" err="found duplicate series for the match group {instance=\"smithi069\"} on the right hand-side of the operation: [{__name__=\"node_uname_info\", domainname=\"(none)\", instance=\"smithi069\", job=\"node\", machine=\"x86_64\", nodename=\"smithi069\", release=\"5.14.0-435.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Fri Mar 29 18:46:37 UTC 2024\"}, {__name__=\"node_uname_info\", cluster=\"dc326fc2-fa65-11ee-bc8e-c7b262605968\", domainname=\"(none)\", instance=\"smithi069\", job=\"node\", machine=\"x86_64\", nodename=\"smithi069\", release=\"5.14.0-435.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Fri Mar 29 18:46:37 UTC 2024\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-04-14T14:06:34.594 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:06:34 smithi122 ceph-mon[25721]: pgmap v91: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:06:34.816 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:06:34 smithi069 ceph-mon[24818]: pgmap v91: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:06:34.817 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:06:34 smithi069 ceph-mon[29472]: pgmap v91: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:06:36.817 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:06:36 smithi069 ceph-mon[24818]: pgmap v92: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:06:36.817 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:06:36 smithi069 ceph-mon[24818]: from='client.15177 -' entity='client.iscsi.foo.smithi069.yuumtr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T14:06:36.817 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:06:36 smithi069 ceph-mon[29472]: pgmap v92: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:06:36.817 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:06:36 smithi069 ceph-mon[29472]: from='client.15177 -' entity='client.iscsi.foo.smithi069.yuumtr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T14:06:36.844 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:06:36 smithi122 ceph-mon[25721]: pgmap v92: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:06:36.844 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:06:36 smithi122 ceph-mon[25721]: from='client.15177 -' entity='client.iscsi.foo.smithi069.yuumtr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T14:06:38.817 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:06:38 smithi069 ceph-mon[24818]: pgmap v93: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:06:38.817 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:06:38 smithi069 ceph-mon[29472]: pgmap v93: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:06:38.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:06:38 smithi122 ceph-mon[25721]: pgmap v93: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:06:38.845 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Apr 14 14:06:38 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-prometheus-a[49277]: ts=2024-04-14T14:06:38.660Z caller=manager.go:609 level=warn component="rule manager" group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on(ceph_daemon) group_left(hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.69\", device_class=\"ssd\", hostname=\"smithi069\", instance=\"172.21.15.122:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.69\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster=\"dc326fc2-fa65-11ee-bc8e-c7b262605968\", cluster_addr=\"172.21.15.69\", device_class=\"ssd\", hostname=\"smithi069\", instance=\"smithi122:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.69\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-04-14T14:06:40.817 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:06:40 smithi069 ceph-mon[24818]: pgmap v94: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:06:40.817 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:06:40 smithi069 ceph-mon[29472]: pgmap v94: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:06:40.844 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:06:40 smithi122 ceph-mon[25721]: pgmap v94: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:06:42.802 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:06:42 smithi122 ceph-mon[25721]: pgmap v95: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:06:42.817 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:06:42 smithi069 ceph-mon[24818]: pgmap v95: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:06:42.817 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:06:42 smithi069 ceph-mon[29472]: pgmap v95: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:06:43.094 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Apr 14 14:06:42 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-x[45487]: ::ffff:172.21.15.122 - - [14/Apr/2024:14:06:42] "GET /metrics HTTP/1.1" 200 37975 "" "Prometheus/2.33.4" 2024-04-14T14:06:43.817 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:06:43 smithi069 ceph-mon[24818]: from='mgr.24749 172.21.15.122:0/1892077634' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T14:06:43.817 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:06:43 smithi069 ceph-mon[29472]: from='mgr.24749 172.21.15.122:0/1892077634' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T14:06:43.844 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:06:43 smithi122 ceph-mon[25721]: from='mgr.24749 172.21.15.122:0/1892077634' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T14:06:43.845 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Apr 14 14:06:43 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-prometheus-a[49277]: ts=2024-04-14T14:06:43.504Z caller=manager.go:609 level=warn component="rule manager" group=nodes msg="Evaluating rule failed" rule="alert: CephNodeDiskspaceWarning\nexpr: predict_linear(node_filesystem_free_bytes{device=~\"/.*\"}[2d], 3600 * 24 * 5)\n * on(instance) group_left(nodename) node_uname_info < 0\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.8.4\n severity: warning\n type: ceph_default\nannotations:\n description: Mountpoint {{ $labels.mountpoint }} on {{ $labels.nodename }} will\n be full in less than 5 days based on the 48 hour trailing fill rate.\n summary: Host filesystem free space is getting low\n" err="found duplicate series for the match group {instance=\"smithi069\"} on the right hand-side of the operation: [{__name__=\"node_uname_info\", domainname=\"(none)\", instance=\"smithi069\", job=\"node\", machine=\"x86_64\", nodename=\"smithi069\", release=\"5.14.0-435.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Fri Mar 29 18:46:37 UTC 2024\"}, {__name__=\"node_uname_info\", cluster=\"dc326fc2-fa65-11ee-bc8e-c7b262605968\", domainname=\"(none)\", instance=\"smithi069\", job=\"node\", machine=\"x86_64\", nodename=\"smithi069\", release=\"5.14.0-435.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Fri Mar 29 18:46:37 UTC 2024\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-04-14T14:06:44.817 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:06:44 smithi069 ceph-mon[24818]: pgmap v96: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:06:44.817 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:06:44 smithi069 ceph-mon[29472]: pgmap v96: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:06:44.844 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:06:44 smithi122 ceph-mon[25721]: pgmap v96: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:06:46.817 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:06:46 smithi069 ceph-mon[24818]: pgmap v97: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:06:46.817 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:06:46 smithi069 ceph-mon[24818]: from='client.15177 -' entity='client.iscsi.foo.smithi069.yuumtr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T14:06:46.817 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:06:46 smithi069 ceph-mon[29472]: pgmap v97: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:06:46.817 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:06:46 smithi069 ceph-mon[29472]: from='client.15177 -' entity='client.iscsi.foo.smithi069.yuumtr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T14:06:46.844 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:06:46 smithi122 ceph-mon[25721]: pgmap v97: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:06:46.844 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:06:46 smithi122 ceph-mon[25721]: from='client.15177 -' entity='client.iscsi.foo.smithi069.yuumtr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T14:06:48.817 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:06:48 smithi069 ceph-mon[24818]: pgmap v98: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:06:48.817 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:06:48 smithi069 ceph-mon[29472]: pgmap v98: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:06:48.844 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:06:48 smithi122 ceph-mon[25721]: pgmap v98: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:06:48.845 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Apr 14 14:06:48 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-prometheus-a[49277]: ts=2024-04-14T14:06:48.660Z caller=manager.go:609 level=warn component="rule manager" group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on(ceph_daemon) group_left(hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.69\", device_class=\"ssd\", hostname=\"smithi069\", instance=\"172.21.15.122:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.69\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster=\"dc326fc2-fa65-11ee-bc8e-c7b262605968\", cluster_addr=\"172.21.15.69\", device_class=\"ssd\", hostname=\"smithi069\", instance=\"smithi122:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.69\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-04-14T14:06:50.817 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:06:50 smithi069 ceph-mon[24818]: pgmap v99: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:06:50.817 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:06:50 smithi069 ceph-mon[29472]: pgmap v99: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:06:50.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:06:50 smithi122 ceph-mon[25721]: pgmap v99: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:06:52.801 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:06:52 smithi122 ceph-mon[25721]: pgmap v100: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:06:52.817 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:06:52 smithi069 ceph-mon[24818]: pgmap v100: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:06:52.817 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:06:52 smithi069 ceph-mon[29472]: pgmap v100: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:06:53.094 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Apr 14 14:06:52 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-x[45487]: ::ffff:172.21.15.122 - - [14/Apr/2024:14:06:52] "GET /metrics HTTP/1.1" 200 37975 "" "Prometheus/2.33.4" 2024-04-14T14:06:53.845 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Apr 14 14:06:53 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-prometheus-a[49277]: ts=2024-04-14T14:06:53.505Z caller=manager.go:609 level=warn component="rule manager" group=nodes msg="Evaluating rule failed" rule="alert: CephNodeDiskspaceWarning\nexpr: predict_linear(node_filesystem_free_bytes{device=~\"/.*\"}[2d], 3600 * 24 * 5)\n * on(instance) group_left(nodename) node_uname_info < 0\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.8.4\n severity: warning\n type: ceph_default\nannotations:\n description: Mountpoint {{ $labels.mountpoint }} on {{ $labels.nodename }} will\n be full in less than 5 days based on the 48 hour trailing fill rate.\n summary: Host filesystem free space is getting low\n" err="found duplicate series for the match group {instance=\"smithi069\"} on the right hand-side of the operation: [{__name__=\"node_uname_info\", domainname=\"(none)\", instance=\"smithi069\", job=\"node\", machine=\"x86_64\", nodename=\"smithi069\", release=\"5.14.0-435.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Fri Mar 29 18:46:37 UTC 2024\"}, {__name__=\"node_uname_info\", cluster=\"dc326fc2-fa65-11ee-bc8e-c7b262605968\", domainname=\"(none)\", instance=\"smithi069\", job=\"node\", machine=\"x86_64\", nodename=\"smithi069\", release=\"5.14.0-435.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Fri Mar 29 18:46:37 UTC 2024\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-04-14T14:06:54.817 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:06:54 smithi069 ceph-mon[24818]: pgmap v101: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:06:54.817 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:06:54 smithi069 ceph-mon[29472]: pgmap v101: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:06:54.844 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:06:54 smithi122 ceph-mon[25721]: pgmap v101: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:06:56.817 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:06:56 smithi069 ceph-mon[24818]: pgmap v102: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:06:56.817 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:06:56 smithi069 ceph-mon[24818]: from='client.15177 -' entity='client.iscsi.foo.smithi069.yuumtr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T14:06:56.817 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:06:56 smithi069 ceph-mon[29472]: pgmap v102: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:06:56.817 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:06:56 smithi069 ceph-mon[29472]: from='client.15177 -' entity='client.iscsi.foo.smithi069.yuumtr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T14:06:56.844 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:06:56 smithi122 ceph-mon[25721]: pgmap v102: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:06:56.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:06:56 smithi122 ceph-mon[25721]: from='client.15177 -' entity='client.iscsi.foo.smithi069.yuumtr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T14:06:58.817 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:06:58 smithi069 ceph-mon[24818]: pgmap v103: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:06:58.817 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:06:58 smithi069 ceph-mon[24818]: from='mgr.24749 172.21.15.122:0/1892077634' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T14:06:58.817 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:06:58 smithi069 ceph-mon[29472]: pgmap v103: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:06:58.817 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:06:58 smithi069 ceph-mon[29472]: from='mgr.24749 172.21.15.122:0/1892077634' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T14:06:58.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:06:58 smithi122 ceph-mon[25721]: pgmap v103: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:06:58.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:06:58 smithi122 ceph-mon[25721]: from='mgr.24749 172.21.15.122:0/1892077634' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T14:06:58.845 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Apr 14 14:06:58 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-prometheus-a[49277]: ts=2024-04-14T14:06:58.660Z caller=manager.go:609 level=warn component="rule manager" group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on(ceph_daemon) group_left(hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.69\", device_class=\"ssd\", hostname=\"smithi069\", instance=\"172.21.15.122:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.69\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster=\"dc326fc2-fa65-11ee-bc8e-c7b262605968\", cluster_addr=\"172.21.15.69\", device_class=\"ssd\", hostname=\"smithi069\", instance=\"smithi122:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.69\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-04-14T14:07:00.396 DEBUG:teuthology.orchestra.run.smithi069:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid dc326fc2-fa65-11ee-bc8e-c7b262605968 -e sha1=a971fd5f5719038c70b817bafacd6d5400b1d888 -- bash -c 'ceph orch daemon redeploy "mgr.$(ceph mgr dump -f json | jq .standbys | jq .[] | jq -r .name)" --image quay.ceph.io/ceph-ci/ceph:$sha1' 2024-04-14T14:07:00.708 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:07:00 smithi069 ceph-mon[24818]: pgmap v104: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:07:00.708 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:07:00 smithi069 ceph-mon[29472]: pgmap v104: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:07:00.844 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:07:00 smithi122 ceph-mon[25721]: pgmap v104: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:07:02.738 INFO:teuthology.orchestra.run.smithi069.stdout:Scheduled to redeploy mgr.y on host 'smithi069' 2024-04-14T14:07:02.802 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:07:02 smithi122 ceph-mon[25721]: pgmap v105: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:07:02.802 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:07:02 smithi122 ceph-mon[25721]: from='client.? 172.21.15.69:0/1829621160' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-04-14T14:07:02.817 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:07:02 smithi069 ceph-mon[24818]: pgmap v105: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:07:02.817 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:07:02 smithi069 ceph-mon[24818]: from='client.? 172.21.15.69:0/1829621160' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-04-14T14:07:02.817 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:07:02 smithi069 ceph-mon[29472]: pgmap v105: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:07:02.818 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:07:02 smithi069 ceph-mon[29472]: from='client.? 172.21.15.69:0/1829621160' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-04-14T14:07:03.094 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Apr 14 14:07:02 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-x[45487]: ::ffff:172.21.15.122 - - [14/Apr/2024:14:07:02] "GET /metrics HTTP/1.1" 200 37971 "" "Prometheus/2.33.4" 2024-04-14T14:07:03.429 DEBUG:teuthology.orchestra.run.smithi069:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid dc326fc2-fa65-11ee-bc8e-c7b262605968 -e sha1=a971fd5f5719038c70b817bafacd6d5400b1d888 -- bash -c 'ceph orch ps --refresh' 2024-04-14T14:07:03.845 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Apr 14 14:07:03 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-prometheus-a[49277]: ts=2024-04-14T14:07:03.504Z caller=manager.go:609 level=warn component="rule manager" group=nodes msg="Evaluating rule failed" rule="alert: CephNodeDiskspaceWarning\nexpr: predict_linear(node_filesystem_free_bytes{device=~\"/.*\"}[2d], 3600 * 24 * 5)\n * on(instance) group_left(nodename) node_uname_info < 0\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.8.4\n severity: warning\n type: ceph_default\nannotations:\n description: Mountpoint {{ $labels.mountpoint }} on {{ $labels.nodename }} will\n be full in less than 5 days based on the 48 hour trailing fill rate.\n summary: Host filesystem free space is getting low\n" err="found duplicate series for the match group {instance=\"smithi069\"} on the right hand-side of the operation: [{__name__=\"node_uname_info\", domainname=\"(none)\", instance=\"smithi069\", job=\"node\", machine=\"x86_64\", nodename=\"smithi069\", release=\"5.14.0-435.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Fri Mar 29 18:46:37 UTC 2024\"}, {__name__=\"node_uname_info\", cluster=\"dc326fc2-fa65-11ee-bc8e-c7b262605968\", domainname=\"(none)\", instance=\"smithi069\", job=\"node\", machine=\"x86_64\", nodename=\"smithi069\", release=\"5.14.0-435.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Fri Mar 29 18:46:37 UTC 2024\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-04-14T14:07:03.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:07:03 smithi122 ceph-mon[25721]: from='client.25093 -' entity='client.admin' cmd=[{"prefix": "orch daemon redeploy", "name": "mgr.y", "image": "quay.ceph.io/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T14:07:03.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:07:03 smithi122 ceph-mon[25721]: from='mgr.24749 ' entity='mgr.x' 2024-04-14T14:07:03.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:07:03 smithi122 ceph-mon[25721]: Schedule redeploy daemon mgr.y 2024-04-14T14:07:03.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:07:03 smithi122 ceph-mon[25721]: from='mgr.24749 ' entity='mgr.x' 2024-04-14T14:07:03.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:07:03 smithi122 ceph-mon[25721]: from='mgr.24749 ' entity='mgr.x' 2024-04-14T14:07:03.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:07:03 smithi122 ceph-mon[25721]: from='mgr.24749 172.21.15.122:0/1892077634' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T14:07:03.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:07:03 smithi122 ceph-mon[25721]: from='mgr.24749 172.21.15.122:0/1892077634' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-14T14:07:03.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:07:03 smithi122 ceph-mon[25721]: from='mgr.24749 ' entity='mgr.x' 2024-04-14T14:07:04.026 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:07:03 smithi069 ceph-mon[29472]: from='client.25093 -' entity='client.admin' cmd=[{"prefix": "orch daemon redeploy", "name": "mgr.y", "image": "quay.ceph.io/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T14:07:04.026 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:07:03 smithi069 ceph-mon[29472]: from='mgr.24749 ' entity='mgr.x' 2024-04-14T14:07:04.026 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:07:03 smithi069 ceph-mon[29472]: Schedule redeploy daemon mgr.y 2024-04-14T14:07:04.026 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:07:03 smithi069 ceph-mon[29472]: from='mgr.24749 ' entity='mgr.x' 2024-04-14T14:07:04.026 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:07:03 smithi069 ceph-mon[29472]: from='mgr.24749 ' entity='mgr.x' 2024-04-14T14:07:04.026 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:07:03 smithi069 ceph-mon[29472]: from='mgr.24749 172.21.15.122:0/1892077634' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T14:07:04.026 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:07:03 smithi069 ceph-mon[29472]: from='mgr.24749 172.21.15.122:0/1892077634' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-14T14:07:04.026 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:07:03 smithi069 ceph-mon[29472]: from='mgr.24749 ' entity='mgr.x' 2024-04-14T14:07:04.027 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:07:03 smithi069 ceph-mon[24818]: from='client.25093 -' entity='client.admin' cmd=[{"prefix": "orch daemon redeploy", "name": "mgr.y", "image": "quay.ceph.io/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T14:07:04.027 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:07:03 smithi069 ceph-mon[24818]: from='mgr.24749 ' entity='mgr.x' 2024-04-14T14:07:04.027 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:07:03 smithi069 ceph-mon[24818]: Schedule redeploy daemon mgr.y 2024-04-14T14:07:04.027 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:07:03 smithi069 ceph-mon[24818]: from='mgr.24749 ' entity='mgr.x' 2024-04-14T14:07:04.027 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:07:03 smithi069 ceph-mon[24818]: from='mgr.24749 ' entity='mgr.x' 2024-04-14T14:07:04.027 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:07:03 smithi069 ceph-mon[24818]: from='mgr.24749 172.21.15.122:0/1892077634' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T14:07:04.027 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:07:03 smithi069 ceph-mon[24818]: from='mgr.24749 172.21.15.122:0/1892077634' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-14T14:07:04.027 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:07:03 smithi069 ceph-mon[24818]: from='mgr.24749 ' entity='mgr.x' 2024-04-14T14:07:04.974 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:07:04 smithi069 ceph-mon[24818]: pgmap v106: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:07:04.974 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:07:04 smithi069 ceph-mon[24818]: pgmap v107: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-04-14T14:07:04.974 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:07:04 smithi069 ceph-mon[24818]: pgmap v108: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:07:04.974 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:07:04 smithi069 ceph-mon[24818]: from='mgr.24749 172.21.15.122:0/1892077634' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-14T14:07:04.974 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:07:04 smithi069 ceph-mon[24818]: from='mgr.24749 ' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-14T14:07:04.974 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:07:04 smithi069 ceph-mon[24818]: from='mgr.24749 172.21.15.122:0/1892077634' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-14T14:07:04.974 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:07:04 smithi069 ceph-mon[24818]: from='mgr.24749 172.21.15.122:0/1892077634' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T14:07:04.974 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:07:04 smithi069 ceph-mon[24818]: Deploying daemon mgr.y on smithi069 2024-04-14T14:07:04.975 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:07:04 smithi069 ceph-mon[29472]: pgmap v106: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:07:04.975 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:07:04 smithi069 ceph-mon[29472]: pgmap v107: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-04-14T14:07:04.975 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:07:04 smithi069 ceph-mon[29472]: pgmap v108: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:07:04.975 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:07:04 smithi069 ceph-mon[29472]: from='mgr.24749 172.21.15.122:0/1892077634' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-14T14:07:04.975 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:07:04 smithi069 ceph-mon[29472]: from='mgr.24749 ' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-14T14:07:04.975 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:07:04 smithi069 ceph-mon[29472]: from='mgr.24749 172.21.15.122:0/1892077634' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-14T14:07:04.975 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:07:04 smithi069 ceph-mon[29472]: from='mgr.24749 172.21.15.122:0/1892077634' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T14:07:04.975 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:07:04 smithi069 ceph-mon[29472]: Deploying daemon mgr.y on smithi069 2024-04-14T14:07:05.094 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:07:04 smithi122 ceph-mon[25721]: pgmap v106: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:07:05.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:07:04 smithi122 ceph-mon[25721]: pgmap v107: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-04-14T14:07:05.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:07:04 smithi122 ceph-mon[25721]: pgmap v108: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:07:05.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:07:04 smithi122 ceph-mon[25721]: from='mgr.24749 172.21.15.122:0/1892077634' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-14T14:07:05.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:07:04 smithi122 ceph-mon[25721]: from='mgr.24749 ' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-14T14:07:05.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:07:04 smithi122 ceph-mon[25721]: from='mgr.24749 172.21.15.122:0/1892077634' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-14T14:07:05.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:07:04 smithi122 ceph-mon[25721]: from='mgr.24749 172.21.15.122:0/1892077634' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T14:07:05.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:07:04 smithi122 ceph-mon[25721]: Deploying daemon mgr.y on smithi069 2024-04-14T14:07:05.653 INFO:teuthology.orchestra.run.smithi069.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-14T14:07:05.653 INFO:teuthology.orchestra.run.smithi069.stdout:alertmanager.a smithi069 *:9093,9094 running (2m) 56s ago 10m 16.8M - 0.23.0 ba2b418f427c e300fc36e997 2024-04-14T14:07:05.653 INFO:teuthology.orchestra.run.smithi069.stdout:grafana.a smithi122 *:3000 error 56s ago 10m - - 2024-04-14T14:07:05.653 INFO:teuthology.orchestra.run.smithi069.stdout:iscsi.foo.smithi069.yuumtr smithi069 running (60s) 56s ago 9m 43.7M - 3.5 e1d6a67b021e ce79fedc7dfd 2024-04-14T14:07:05.653 INFO:teuthology.orchestra.run.smithi069.stdout:mgr.x smithi122 *:8443,9283 running (5m) 56s ago 14m 512M - 19.0.0-2778-ga971fd5f b96c509eef84 813744c2b615 2024-04-14T14:07:05.653 INFO:teuthology.orchestra.run.smithi069.stdout:mgr.y smithi069 *:9283 running (16m) 56s ago 16m 411M - 17.2.0 e1d6a67b021e 52e1363a8f98 2024-04-14T14:07:05.653 INFO:teuthology.orchestra.run.smithi069.stdout:mon.a smithi069 running (16m) 56s ago 16m 56.7M 2048M 17.2.0 e1d6a67b021e d3c553f91913 2024-04-14T14:07:05.654 INFO:teuthology.orchestra.run.smithi069.stdout:mon.b smithi122 running (15m) 56s ago 15m 48.5M 2048M 17.2.0 e1d6a67b021e 3b766bfce4be 2024-04-14T14:07:05.654 INFO:teuthology.orchestra.run.smithi069.stdout:mon.c smithi069 running (15m) 56s ago 15m 44.1M 2048M 17.2.0 e1d6a67b021e 23c078b9d1e5 2024-04-14T14:07:05.654 INFO:teuthology.orchestra.run.smithi069.stdout:node-exporter.a smithi069 *:9100 running (2m) 56s ago 11m 17.2M - 1.3.1 1dbe0e931976 ca074142ab73 2024-04-14T14:07:05.654 INFO:teuthology.orchestra.run.smithi069.stdout:node-exporter.b smithi122 *:9100 running (2m) 56s ago 11m 8724k - 1.3.1 1dbe0e931976 996115353a0b 2024-04-14T14:07:05.654 INFO:teuthology.orchestra.run.smithi069.stdout:osd.0 smithi069 running (14m) 56s ago 14m 54.4M 2992M 17.2.0 e1d6a67b021e 46654654b6b6 2024-04-14T14:07:05.654 INFO:teuthology.orchestra.run.smithi069.stdout:osd.1 smithi069 running (14m) 56s ago 14m 51.6M 2992M 17.2.0 e1d6a67b021e cdba54a59609 2024-04-14T14:07:05.654 INFO:teuthology.orchestra.run.smithi069.stdout:osd.2 smithi069 running (13m) 56s ago 13m 48.0M 2992M 17.2.0 e1d6a67b021e bc37a1a9cba0 2024-04-14T14:07:05.654 INFO:teuthology.orchestra.run.smithi069.stdout:osd.3 smithi069 running (13m) 56s ago 13m 53.0M 2992M 17.2.0 e1d6a67b021e 0925ba3da801 2024-04-14T14:07:05.654 INFO:teuthology.orchestra.run.smithi069.stdout:osd.4 smithi122 running (13m) 56s ago 13m 51.6M 3248M 17.2.0 e1d6a67b021e c00975caf771 2024-04-14T14:07:05.654 INFO:teuthology.orchestra.run.smithi069.stdout:osd.5 smithi122 running (12m) 56s ago 12m 52.5M 3248M 17.2.0 e1d6a67b021e 5f52d217ceb8 2024-04-14T14:07:05.654 INFO:teuthology.orchestra.run.smithi069.stdout:osd.6 smithi122 running (12m) 56s ago 12m 47.6M 3248M 17.2.0 e1d6a67b021e a2acfec61475 2024-04-14T14:07:05.654 INFO:teuthology.orchestra.run.smithi069.stdout:osd.7 smithi122 running (12m) 56s ago 12m 53.3M 3248M 17.2.0 e1d6a67b021e 62411cd06801 2024-04-14T14:07:05.654 INFO:teuthology.orchestra.run.smithi069.stdout:prometheus.a smithi122 *:9095 running (2m) 56s ago 11m 67.0M - 2.33.4 514e6a882f6e 2bdf6dae6e1e 2024-04-14T14:07:05.654 INFO:teuthology.orchestra.run.smithi069.stdout:rgw.foo.smithi069.wesedz smithi069 *:8000 running (9m) 56s ago 9m 74.4M - 17.2.0 e1d6a67b021e 16de491440f3 2024-04-14T14:07:05.655 INFO:teuthology.orchestra.run.smithi069.stdout:rgw.foo.smithi122.jiobyv smithi122 *:8000 running (9m) 56s ago 9m 71.2M - 17.2.0 e1d6a67b021e 6d16de445dba 2024-04-14T14:07:06.379 DEBUG:teuthology.orchestra.run.smithi069:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid dc326fc2-fa65-11ee-bc8e-c7b262605968 -e sha1=a971fd5f5719038c70b817bafacd6d5400b1d888 -- bash -c 'sleep 180' 2024-04-14T14:07:06.706 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:07:06 smithi069 ceph-mon[24818]: pgmap v109: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:07:06.706 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:07:06 smithi069 ceph-mon[24818]: from='client.15204 -' entity='client.admin' cmd=[{"prefix": "orch ps", "refresh": true, "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T14:07:06.706 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:07:06 smithi069 ceph-mon[24818]: from='client.15177 -' entity='client.iscsi.foo.smithi069.yuumtr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T14:07:06.707 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:07:06 smithi069 ceph-mon[29472]: pgmap v109: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:07:06.707 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:07:06 smithi069 ceph-mon[29472]: from='client.15204 -' entity='client.admin' cmd=[{"prefix": "orch ps", "refresh": true, "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T14:07:06.707 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:07:06 smithi069 ceph-mon[29472]: from='client.15177 -' entity='client.iscsi.foo.smithi069.yuumtr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T14:07:06.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:07:06 smithi122 ceph-mon[25721]: pgmap v109: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:07:06.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:07:06 smithi122 ceph-mon[25721]: from='client.15204 -' entity='client.admin' cmd=[{"prefix": "orch ps", "refresh": true, "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T14:07:06.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:07:06 smithi122 ceph-mon[25721]: from='client.15177 -' entity='client.iscsi.foo.smithi069.yuumtr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T14:07:09.067 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:07:08 smithi069 ceph-mon[24818]: pgmap v110: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 615 B/s rd, 0 op/s 2024-04-14T14:07:09.067 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:07:08 smithi069 ceph-mon[29472]: pgmap v110: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 615 B/s rd, 0 op/s 2024-04-14T14:07:09.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:07:08 smithi122 ceph-mon[25721]: pgmap v110: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 615 B/s rd, 0 op/s 2024-04-14T14:07:09.095 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Apr 14 14:07:08 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-prometheus-a[49277]: ts=2024-04-14T14:07:08.659Z caller=manager.go:609 level=warn component="rule manager" group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on(ceph_daemon) group_left(hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.69\", device_class=\"ssd\", hostname=\"smithi069\", instance=\"172.21.15.122:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.69\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster=\"dc326fc2-fa65-11ee-bc8e-c7b262605968\", cluster_addr=\"172.21.15.69\", device_class=\"ssd\", hostname=\"smithi069\", instance=\"smithi122:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.69\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-04-14T14:07:11.067 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:07:10 smithi069 ceph-mon[24818]: pgmap v111: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:07:11.067 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:07:10 smithi069 ceph-mon[29472]: pgmap v111: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:07:11.094 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:07:10 smithi122 ceph-mon[25721]: pgmap v111: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:07:13.067 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:07:12 smithi069 ceph-mon[24818]: pgmap v112: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:07:13.067 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:07:12 smithi069 ceph-mon[24818]: from='mgr.24749 172.21.15.122:0/1892077634' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T14:07:13.067 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:07:12 smithi069 ceph-mon[29472]: pgmap v112: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:07:13.067 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:07:12 smithi069 ceph-mon[29472]: from='mgr.24749 172.21.15.122:0/1892077634' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T14:07:13.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:07:12 smithi122 ceph-mon[25721]: pgmap v112: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:07:13.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:07:12 smithi122 ceph-mon[25721]: from='mgr.24749 172.21.15.122:0/1892077634' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T14:07:13.095 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Apr 14 14:07:12 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-x[45487]: ::ffff:172.21.15.122 - - [14/Apr/2024:14:07:12] "GET /metrics HTTP/1.1" 200 37974 "" "Prometheus/2.33.4" 2024-04-14T14:07:13.844 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Apr 14 14:07:13 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-prometheus-a[49277]: ts=2024-04-14T14:07:13.504Z caller=manager.go:609 level=warn component="rule manager" group=nodes msg="Evaluating rule failed" rule="alert: CephNodeDiskspaceWarning\nexpr: predict_linear(node_filesystem_free_bytes{device=~\"/.*\"}[2d], 3600 * 24 * 5)\n * on(instance) group_left(nodename) node_uname_info < 0\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.8.4\n severity: warning\n type: ceph_default\nannotations:\n description: Mountpoint {{ $labels.mountpoint }} on {{ $labels.nodename }} will\n be full in less than 5 days based on the 48 hour trailing fill rate.\n summary: Host filesystem free space is getting low\n" err="found duplicate series for the match group {instance=\"smithi069\"} on the right hand-side of the operation: [{__name__=\"node_uname_info\", domainname=\"(none)\", instance=\"smithi069\", job=\"node\", machine=\"x86_64\", nodename=\"smithi069\", release=\"5.14.0-435.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Fri Mar 29 18:46:37 UTC 2024\"}, {__name__=\"node_uname_info\", cluster=\"dc326fc2-fa65-11ee-bc8e-c7b262605968\", domainname=\"(none)\", instance=\"smithi069\", job=\"node\", machine=\"x86_64\", nodename=\"smithi069\", release=\"5.14.0-435.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Fri Mar 29 18:46:37 UTC 2024\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-04-14T14:07:15.067 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:07:14 smithi069 ceph-mon[24818]: pgmap v113: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-14T14:07:15.067 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:07:14 smithi069 ceph-mon[29472]: pgmap v113: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-14T14:07:15.094 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:07:14 smithi122 ceph-mon[25721]: pgmap v113: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-14T14:07:16.817 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:07:16 smithi069 ceph-mon[24818]: pgmap v114: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:07:16.817 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:07:16 smithi069 ceph-mon[24818]: from='client.15177 -' entity='client.iscsi.foo.smithi069.yuumtr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T14:07:16.817 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:07:16 smithi069 ceph-mon[29472]: pgmap v114: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:07:16.817 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:07:16 smithi069 ceph-mon[29472]: from='client.15177 -' entity='client.iscsi.foo.smithi069.yuumtr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T14:07:16.844 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:07:16 smithi122 ceph-mon[25721]: pgmap v114: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:07:16.844 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:07:16 smithi122 ceph-mon[25721]: from='client.15177 -' entity='client.iscsi.foo.smithi069.yuumtr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T14:07:19.066 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:07:18 smithi069 ceph-mon[24818]: pgmap v115: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:07:19.067 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:07:18 smithi069 ceph-mon[29472]: pgmap v115: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:07:19.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:07:18 smithi122 ceph-mon[25721]: pgmap v115: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:07:19.095 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Apr 14 14:07:18 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-prometheus-a[49277]: ts=2024-04-14T14:07:18.659Z caller=manager.go:609 level=warn component="rule manager" group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on(ceph_daemon) group_left(hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.69\", device_class=\"ssd\", hostname=\"smithi069\", instance=\"172.21.15.122:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.69\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster=\"dc326fc2-fa65-11ee-bc8e-c7b262605968\", cluster_addr=\"172.21.15.69\", device_class=\"ssd\", hostname=\"smithi069\", instance=\"smithi122:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.69\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-04-14T14:07:21.067 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:07:20 smithi069 ceph-mon[24818]: pgmap v116: 161 pgs: 161 active+clean; 585 KiB data, 73 MiB used, 715 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2024-04-14T14:07:21.067 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:07:20 smithi069 ceph-mon[29472]: pgmap v116: 161 pgs: 161 active+clean; 585 KiB data, 73 MiB used, 715 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2024-04-14T14:07:21.094 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:07:20 smithi122 ceph-mon[25721]: pgmap v116: 161 pgs: 161 active+clean; 585 KiB data, 73 MiB used, 715 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2024-04-14T14:07:23.066 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:07:22 smithi069 ceph-mon[24818]: pgmap v117: 161 pgs: 161 active+clean; 585 KiB data, 73 MiB used, 715 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2024-04-14T14:07:23.067 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:07:22 smithi069 ceph-mon[29472]: pgmap v117: 161 pgs: 161 active+clean; 585 KiB data, 73 MiB used, 715 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2024-04-14T14:07:23.094 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:07:22 smithi122 ceph-mon[25721]: pgmap v117: 161 pgs: 161 active+clean; 585 KiB data, 73 MiB used, 715 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2024-04-14T14:07:23.095 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Apr 14 14:07:22 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-x[45487]: ::ffff:172.21.15.122 - - [14/Apr/2024:14:07:22] "GET /metrics HTTP/1.1" 200 37974 "" "Prometheus/2.33.4" 2024-04-14T14:07:23.845 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Apr 14 14:07:23 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-prometheus-a[49277]: ts=2024-04-14T14:07:23.504Z caller=manager.go:609 level=warn component="rule manager" group=nodes msg="Evaluating rule failed" rule="alert: CephNodeDiskspaceWarning\nexpr: predict_linear(node_filesystem_free_bytes{device=~\"/.*\"}[2d], 3600 * 24 * 5)\n * on(instance) group_left(nodename) node_uname_info < 0\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.8.4\n severity: warning\n type: ceph_default\nannotations:\n description: Mountpoint {{ $labels.mountpoint }} on {{ $labels.nodename }} will\n be full in less than 5 days based on the 48 hour trailing fill rate.\n summary: Host filesystem free space is getting low\n" err="found duplicate series for the match group {instance=\"smithi069\"} on the right hand-side of the operation: [{__name__=\"node_uname_info\", domainname=\"(none)\", instance=\"smithi069\", job=\"node\", machine=\"x86_64\", nodename=\"smithi069\", release=\"5.14.0-435.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Fri Mar 29 18:46:37 UTC 2024\"}, {__name__=\"node_uname_info\", cluster=\"dc326fc2-fa65-11ee-bc8e-c7b262605968\", domainname=\"(none)\", instance=\"smithi069\", job=\"node\", machine=\"x86_64\", nodename=\"smithi069\", release=\"5.14.0-435.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Fri Mar 29 18:46:37 UTC 2024\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-04-14T14:07:25.066 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:07:24 smithi069 ceph-mon[24818]: pgmap v118: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 73 KiB/s rd, 0 B/s wr, 120 op/s 2024-04-14T14:07:25.067 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:07:24 smithi069 ceph-mon[29472]: pgmap v118: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 73 KiB/s rd, 0 B/s wr, 120 op/s 2024-04-14T14:07:25.094 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:07:24 smithi122 ceph-mon[25721]: pgmap v118: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 73 KiB/s rd, 0 B/s wr, 120 op/s 2024-04-14T14:07:26.817 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:07:26 smithi069 ceph-mon[24818]: pgmap v119: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 72 KiB/s rd, 0 B/s wr, 119 op/s 2024-04-14T14:07:26.817 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:07:26 smithi069 ceph-mon[24818]: from='client.15177 -' entity='client.iscsi.foo.smithi069.yuumtr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T14:07:26.817 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:07:26 smithi069 ceph-mon[29472]: pgmap v119: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 72 KiB/s rd, 0 B/s wr, 119 op/s 2024-04-14T14:07:26.817 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:07:26 smithi069 ceph-mon[29472]: from='client.15177 -' entity='client.iscsi.foo.smithi069.yuumtr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T14:07:26.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:07:26 smithi122 ceph-mon[25721]: pgmap v119: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 72 KiB/s rd, 0 B/s wr, 119 op/s 2024-04-14T14:07:26.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:07:26 smithi122 ceph-mon[25721]: from='client.15177 -' entity='client.iscsi.foo.smithi069.yuumtr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T14:07:28.066 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:07:27 smithi069 ceph-mon[24818]: from='mgr.24749 172.21.15.122:0/1892077634' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T14:07:28.067 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:07:27 smithi069 ceph-mon[29472]: from='mgr.24749 172.21.15.122:0/1892077634' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T14:07:28.094 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:07:27 smithi122 ceph-mon[25721]: from='mgr.24749 172.21.15.122:0/1892077634' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T14:07:29.067 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:07:28 smithi069 ceph-mon[24818]: pgmap v120: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 72 KiB/s rd, 0 B/s wr, 119 op/s 2024-04-14T14:07:29.067 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:07:28 smithi069 ceph-mon[29472]: pgmap v120: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 72 KiB/s rd, 0 B/s wr, 119 op/s 2024-04-14T14:07:29.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:07:28 smithi122 ceph-mon[25721]: pgmap v120: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 72 KiB/s rd, 0 B/s wr, 119 op/s 2024-04-14T14:07:29.095 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Apr 14 14:07:28 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-prometheus-a[49277]: ts=2024-04-14T14:07:28.659Z caller=manager.go:609 level=warn component="rule manager" group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on(ceph_daemon) group_left(hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.69\", device_class=\"ssd\", hostname=\"smithi069\", instance=\"172.21.15.122:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.69\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster=\"dc326fc2-fa65-11ee-bc8e-c7b262605968\", cluster_addr=\"172.21.15.69\", device_class=\"ssd\", hostname=\"smithi069\", instance=\"smithi122:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.69\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-04-14T14:07:31.067 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:07:30 smithi069 ceph-mon[24818]: pgmap v121: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 73 KiB/s rd, 0 B/s wr, 120 op/s 2024-04-14T14:07:31.067 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:07:30 smithi069 ceph-mon[29472]: pgmap v121: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 73 KiB/s rd, 0 B/s wr, 120 op/s 2024-04-14T14:07:31.094 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:07:30 smithi122 ceph-mon[25721]: pgmap v121: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 73 KiB/s rd, 0 B/s wr, 120 op/s 2024-04-14T14:07:33.067 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:07:32 smithi069 ceph-mon[24818]: pgmap v122: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2024-04-14T14:07:33.067 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:07:32 smithi069 ceph-mon[29472]: pgmap v122: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2024-04-14T14:07:33.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:07:32 smithi122 ceph-mon[25721]: pgmap v122: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2024-04-14T14:07:33.095 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Apr 14 14:07:32 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-x[45487]: ::ffff:172.21.15.122 - - [14/Apr/2024:14:07:32] "GET /metrics HTTP/1.1" 200 37968 "" "Prometheus/2.33.4" 2024-04-14T14:07:33.845 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Apr 14 14:07:33 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-prometheus-a[49277]: ts=2024-04-14T14:07:33.504Z caller=manager.go:609 level=warn component="rule manager" group=nodes msg="Evaluating rule failed" rule="alert: CephNodeDiskspaceWarning\nexpr: predict_linear(node_filesystem_free_bytes{device=~\"/.*\"}[2d], 3600 * 24 * 5)\n * on(instance) group_left(nodename) node_uname_info < 0\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.8.4\n severity: warning\n type: ceph_default\nannotations:\n description: Mountpoint {{ $labels.mountpoint }} on {{ $labels.nodename }} will\n be full in less than 5 days based on the 48 hour trailing fill rate.\n summary: Host filesystem free space is getting low\n" err="found duplicate series for the match group {instance=\"smithi069\"} on the right hand-side of the operation: [{__name__=\"node_uname_info\", domainname=\"(none)\", instance=\"smithi069\", job=\"node\", machine=\"x86_64\", nodename=\"smithi069\", release=\"5.14.0-435.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Fri Mar 29 18:46:37 UTC 2024\"}, {__name__=\"node_uname_info\", cluster=\"dc326fc2-fa65-11ee-bc8e-c7b262605968\", domainname=\"(none)\", instance=\"smithi069\", job=\"node\", machine=\"x86_64\", nodename=\"smithi069\", release=\"5.14.0-435.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Fri Mar 29 18:46:37 UTC 2024\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-04-14T14:07:34.797 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:07:34 smithi069 ceph-mon[24818]: pgmap v123: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2024-04-14T14:07:34.797 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:07:34 smithi069 ceph-mon[29472]: pgmap v123: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2024-04-14T14:07:35.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:07:34 smithi122 ceph-mon[25721]: pgmap v123: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2024-04-14T14:07:36.317 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 14 14:07:35 smithi069 systemd[1]: Stopping Ceph mgr.y for dc326fc2-fa65-11ee-bc8e-c7b262605968... 2024-04-14T14:07:36.317 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 14 14:07:36 smithi069 podman[62882]: 2024-04-14 14:07:36.095227254 +0000 UTC m=+0.106810853 container died 52e1363a8f98693029d56b4c0560dff745fb1b1bb8ed2d0f2dec9a4a49e3d02e (image=quay.io/ceph/ceph:v17.2.0, name=ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-y, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, io.openshift.tags=base centos centos-stream, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., release=754, GIT_BRANCH=HEAD, io.openshift.expose-services=, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, build-date=2022-05-03T08:36:31.336870, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, vendor=Red Hat, Inc., description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, GIT_CLEAN=True, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, io.k8s.display-name=CentOS Stream 8, CEPH_POINT_RELEASE=-17.2.0, name=centos-stream, RELEASE=HEAD, GIT_REPO=https://github.com/ceph/ceph-container.git, maintainer=Guillaume Abrioux , distribution-scope=public, architecture=x86_64, vcs-type=git, com.redhat.component=centos-stream-container, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, version=8, io.buildah.version=1.19.8, ceph=True) 2024-04-14T14:07:36.642 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:07:36 smithi069 ceph-mon[24818]: pgmap v124: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:07:36.642 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:07:36 smithi069 ceph-mon[24818]: from='client.15177 -' entity='client.iscsi.foo.smithi069.yuumtr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T14:07:36.642 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 14 14:07:36 smithi069 podman[62882]: 2024-04-14 14:07:36.430659303 +0000 UTC m=+0.442242891 container cleanup 52e1363a8f98693029d56b4c0560dff745fb1b1bb8ed2d0f2dec9a4a49e3d02e (image=quay.io/ceph/ceph:v17.2.0, name=ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-y, version=8, ceph=True, CEPH_POINT_RELEASE=-17.2.0, build-date=2022-05-03T08:36:31.336870, io.openshift.expose-services=, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, io.openshift.tags=base centos centos-stream, release=754, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, distribution-scope=public, io.k8s.display-name=CentOS Stream 8, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, GIT_REPO=https://github.com/ceph/ceph-container.git, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, RELEASE=HEAD, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, maintainer=Guillaume Abrioux , io.buildah.version=1.19.8, name=centos-stream, GIT_BRANCH=HEAD, vcs-type=git, com.redhat.component=centos-stream-container, vendor=Red Hat, Inc., GIT_CLEAN=True, architecture=x86_64) 2024-04-14T14:07:36.642 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 14 14:07:36 smithi069 bash[62882]: ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-y 2024-04-14T14:07:36.642 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:07:36 smithi069 ceph-mon[29472]: pgmap v124: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:07:36.642 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:07:36 smithi069 ceph-mon[29472]: from='client.15177 -' entity='client.iscsi.foo.smithi069.yuumtr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T14:07:36.844 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:07:36 smithi122 ceph-mon[25721]: pgmap v124: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:07:36.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:07:36 smithi122 ceph-mon[25721]: from='client.15177 -' entity='client.iscsi.foo.smithi069.yuumtr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T14:07:37.051 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 14 14:07:36 smithi069 podman[62896]: 2024-04-14 14:07:36.640301224 +0000 UTC m=+0.548949019 container remove 52e1363a8f98693029d56b4c0560dff745fb1b1bb8ed2d0f2dec9a4a49e3d02e (image=quay.io/ceph/ceph:v17.2.0, name=ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-y, build-date=2022-05-03T08:36:31.336870, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., GIT_BRANCH=HEAD, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, io.k8s.display-name=CentOS Stream 8, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, vcs-type=git, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, name=centos-stream, vendor=Red Hat, Inc., distribution-scope=public, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, GIT_CLEAN=True, GIT_REPO=https://github.com/ceph/ceph-container.git, RELEASE=HEAD, com.redhat.component=centos-stream-container, maintainer=Guillaume Abrioux , ceph=True, version=8, architecture=x86_64, io.openshift.tags=base centos centos-stream, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, io.buildah.version=1.19.8, io.openshift.expose-services=, release=754, CEPH_POINT_RELEASE=-17.2.0) 2024-04-14T14:07:37.051 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 14 14:07:36 smithi069 systemd[1]: ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968@mgr.y.service: Main process exited, code=exited, status=143/n/a 2024-04-14T14:07:37.051 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 14 14:07:36 smithi069 systemd[1]: ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968@mgr.y.service: Failed with result 'exit-code'. 2024-04-14T14:07:37.051 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 14 14:07:36 smithi069 systemd[1]: Stopped Ceph mgr.y for dc326fc2-fa65-11ee-bc8e-c7b262605968. 2024-04-14T14:07:37.052 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 14 14:07:36 smithi069 systemd[1]: ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968@mgr.y.service: Consumed 59.061s CPU time. 2024-04-14T14:07:37.318 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 14 14:07:37 smithi069 systemd[1]: Starting Ceph mgr.y for dc326fc2-fa65-11ee-bc8e-c7b262605968... 2024-04-14T14:07:37.318 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 14 14:07:37 smithi069 podman[63023]: 2024-04-14 14:07:37.167224805 +0000 UTC m=+0.017482209 image pull b96c509eef841460b3271f5fdd331860da87b004524f4ed59414aa3680183b53 quay.ceph.io/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888 2024-04-14T14:07:37.318 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 14 14:07:37 smithi069 podman[63023]: 2024-04-14 14:07:37.29960529 +0000 UTC m=+0.149862680 container create 7af3d29d0e5fda9c03e65cecd2e5360e8fdf5b95688bc0c3f2871d3c90b65f16 (image=quay.ceph.io/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888, name=ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-y, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240403, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_CLEAN=True, RELEASE=main-a971fd5, maintainer=Guillaume Abrioux ) 2024-04-14T14:07:39.067 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:07:38 smithi069 ceph-mon[24818]: pgmap v125: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:07:39.067 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:07:38 smithi069 ceph-mon[29472]: pgmap v125: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:07:39.067 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 14 14:07:38 smithi069 podman[63023]: 2024-04-14 14:07:38.798802362 +0000 UTC m=+1.649059752 container init 7af3d29d0e5fda9c03e65cecd2e5360e8fdf5b95688bc0c3f2871d3c90b65f16 (image=quay.ceph.io/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888, name=ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-y, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, ceph=True, org.label-schema.build-date=20240403, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, RELEASE=main-a971fd5, org.label-schema.license=GPLv2, GIT_CLEAN=True, CEPH_POINT_RELEASE=) 2024-04-14T14:07:39.067 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 14 14:07:38 smithi069 podman[63023]: 2024-04-14 14:07:38.802368509 +0000 UTC m=+1.652625907 container start 7af3d29d0e5fda9c03e65cecd2e5360e8fdf5b95688bc0c3f2871d3c90b65f16 (image=quay.ceph.io/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888, name=ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-y, RELEASE=main-a971fd5, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, ceph=True, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240403, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0) 2024-04-14T14:07:39.067 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 14 14:07:38 smithi069 ceph-mgr[63042]: -- 172.21.15.69:0/849352687 <== mon.1 v2:172.21.15.69:3301/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55d0f61905a0 con 0x55d0f6164c00 2024-04-14T14:07:39.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:07:38 smithi122 ceph-mon[25721]: pgmap v125: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:07:39.095 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Apr 14 14:07:38 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-prometheus-a[49277]: ts=2024-04-14T14:07:38.660Z caller=manager.go:609 level=warn component="rule manager" group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on(ceph_daemon) group_left(hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.69\", device_class=\"ssd\", hostname=\"smithi069\", instance=\"172.21.15.122:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.69\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster=\"dc326fc2-fa65-11ee-bc8e-c7b262605968\", cluster_addr=\"172.21.15.69\", device_class=\"ssd\", hostname=\"smithi069\", instance=\"smithi122:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.69\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-04-14T14:07:39.567 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 14 14:07:39 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-y[63038]: 2024-04-14T14:07:39.365+0000 7fc794cd8200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-14T14:07:39.567 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 14 14:07:39 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-y[63038]: 2024-04-14T14:07:39.466+0000 7fc794cd8200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-14T14:07:40.066 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 14 14:07:39 smithi069 bash[63023]: 7af3d29d0e5fda9c03e65cecd2e5360e8fdf5b95688bc0c3f2871d3c90b65f16 2024-04-14T14:07:40.067 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 14 14:07:39 smithi069 systemd[1]: Started Ceph mgr.y for dc326fc2-fa65-11ee-bc8e-c7b262605968. 2024-04-14T14:07:40.455 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 14 14:07:40 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-y[63038]: 2024-04-14T14:07:40.171+0000 7fc794cd8200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-14T14:07:40.455 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 14 14:07:40 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-y[63038]: 2024-04-14T14:07:40.283+0000 7fc794cd8200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-14T14:07:40.817 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:07:40 smithi069 ceph-mon[24818]: pgmap v126: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:07:40.817 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:07:40 smithi069 ceph-mon[24818]: from='mgr.24749 ' entity='mgr.x' 2024-04-14T14:07:40.817 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:07:40 smithi069 ceph-mon[24818]: from='mgr.24749 ' entity='mgr.x' 2024-04-14T14:07:40.817 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:07:40 smithi069 ceph-mon[24818]: from='mgr.24749 ' entity='mgr.x' 2024-04-14T14:07:40.817 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:07:40 smithi069 ceph-mon[24818]: from='mgr.24749 ' entity='mgr.x' 2024-04-14T14:07:40.817 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 14 14:07:40 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-y[63038]: 2024-04-14T14:07:40.447+0000 7fc794cd8200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-14T14:07:40.817 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 14 14:07:40 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-y[63038]: 2024-04-14T14:07:40.545+0000 7fc794cd8200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-14T14:07:40.818 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 14 14:07:40 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-y[63038]: 2024-04-14T14:07:40.628+0000 7fc794cd8200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-14T14:07:40.818 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:07:40 smithi069 ceph-mon[29472]: pgmap v126: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:07:40.818 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:07:40 smithi069 ceph-mon[29472]: from='mgr.24749 ' entity='mgr.x' 2024-04-14T14:07:40.818 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:07:40 smithi069 ceph-mon[29472]: from='mgr.24749 ' entity='mgr.x' 2024-04-14T14:07:40.818 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:07:40 smithi069 ceph-mon[29472]: from='mgr.24749 ' entity='mgr.x' 2024-04-14T14:07:40.818 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:07:40 smithi069 ceph-mon[29472]: from='mgr.24749 ' entity='mgr.x' 2024-04-14T14:07:40.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:07:40 smithi122 ceph-mon[25721]: pgmap v126: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:07:40.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:07:40 smithi122 ceph-mon[25721]: from='mgr.24749 ' entity='mgr.x' 2024-04-14T14:07:40.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:07:40 smithi122 ceph-mon[25721]: from='mgr.24749 ' entity='mgr.x' 2024-04-14T14:07:40.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:07:40 smithi122 ceph-mon[25721]: from='mgr.24749 ' entity='mgr.x' 2024-04-14T14:07:40.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:07:40 smithi122 ceph-mon[25721]: from='mgr.24749 ' entity='mgr.x' 2024-04-14T14:07:41.241 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 14 14:07:40 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-y[63038]: 2024-04-14T14:07:40.900+0000 7fc794cd8200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-14T14:07:41.241 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 14 14:07:41 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-y[63038]: 2024-04-14T14:07:41.010+0000 7fc794cd8200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-14T14:07:41.241 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 14 14:07:41 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-y[63038]: 2024-04-14T14:07:41.097+0000 7fc794cd8200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-14T14:07:41.566 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 14 14:07:41 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-y[63038]: 2024-04-14T14:07:41.191+0000 7fc794cd8200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-14T14:07:41.567 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 14 14:07:41 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-y[63038]: 2024-04-14T14:07:41.369+0000 7fc794cd8200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-14T14:07:42.616 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 14 14:07:42 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-y[63038]: 2024-04-14T14:07:42.352+0000 7fc794cd8200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-14T14:07:42.617 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 14 14:07:42 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-y[63038]: 2024-04-14T14:07:42.486+0000 7fc794cd8200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-14T14:07:42.930 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:07:42 smithi069 ceph-mon[24818]: pgmap v127: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:07:42.930 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:07:42 smithi069 ceph-mon[24818]: from='mgr.24749 172.21.15.122:0/1892077634' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T14:07:42.930 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 14 14:07:42 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-y[63038]: 2024-04-14T14:07:42.835+0000 7fc794cd8200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-14T14:07:42.930 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:07:42 smithi069 ceph-mon[29472]: pgmap v127: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:07:42.930 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:07:42 smithi069 ceph-mon[29472]: from='mgr.24749 172.21.15.122:0/1892077634' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T14:07:42.969 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:07:42 smithi122 ceph-mon[25721]: pgmap v127: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:07:42.970 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:07:42 smithi122 ceph-mon[25721]: from='mgr.24749 172.21.15.122:0/1892077634' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T14:07:42.970 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Apr 14 14:07:42 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-x[45487]: ::ffff:172.21.15.122 - - [14/Apr/2024:14:07:42] "GET /metrics HTTP/1.1" 200 37972 "" "Prometheus/2.33.4" 2024-04-14T14:07:43.318 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 14 14:07:42 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-y[63038]: 2024-04-14T14:07:42.928+0000 7fc794cd8200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-14T14:07:43.318 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 14 14:07:43 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-y[63038]: 2024-04-14T14:07:43.132+0000 7fc794cd8200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-14T14:07:43.628 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 14 14:07:43 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-y[63038]: 2024-04-14T14:07:43.325+0000 7fc794cd8200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-14T14:07:43.845 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Apr 14 14:07:43 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-prometheus-a[49277]: ts=2024-04-14T14:07:43.503Z caller=manager.go:609 level=warn component="rule manager" group=nodes msg="Evaluating rule failed" rule="alert: CephNodeDiskspaceWarning\nexpr: predict_linear(node_filesystem_free_bytes{device=~\"/.*\"}[2d], 3600 * 24 * 5)\n * on(instance) group_left(nodename) node_uname_info < 0\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.8.4\n severity: warning\n type: ceph_default\nannotations:\n description: Mountpoint {{ $labels.mountpoint }} on {{ $labels.nodename }} will\n be full in less than 5 days based on the 48 hour trailing fill rate.\n summary: Host filesystem free space is getting low\n" err="found duplicate series for the match group {instance=\"smithi069\"} on the right hand-side of the operation: [{__name__=\"node_uname_info\", domainname=\"(none)\", instance=\"smithi069\", job=\"node\", machine=\"x86_64\", nodename=\"smithi069\", release=\"5.14.0-435.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Fri Mar 29 18:46:37 UTC 2024\"}, {__name__=\"node_uname_info\", cluster=\"dc326fc2-fa65-11ee-bc8e-c7b262605968\", domainname=\"(none)\", instance=\"smithi069\", job=\"node\", machine=\"x86_64\", nodename=\"smithi069\", release=\"5.14.0-435.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Fri Mar 29 18:46:37 UTC 2024\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-04-14T14:07:43.891 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 14 14:07:43 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-y[63038]: 2024-04-14T14:07:43.626+0000 7fc794cd8200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-14T14:07:44.316 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 14 14:07:43 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-y[63038]: 2024-04-14T14:07:43.889+0000 7fc794cd8200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-14T14:07:44.317 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 14 14:07:44 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-y[63038]: 2024-04-14T14:07:44.134+0000 7fc794cd8200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-14T14:07:44.817 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:07:44 smithi069 ceph-mon[24818]: from='mgr.24749 ' entity='mgr.x' 2024-04-14T14:07:44.817 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:07:44 smithi069 ceph-mon[24818]: from='mgr.24749 ' entity='mgr.x' 2024-04-14T14:07:44.817 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:07:44 smithi069 ceph-mon[24818]: pgmap v128: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:07:44.817 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:07:44 smithi069 ceph-mon[29472]: from='mgr.24749 ' entity='mgr.x' 2024-04-14T14:07:44.817 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:07:44 smithi069 ceph-mon[29472]: from='mgr.24749 ' entity='mgr.x' 2024-04-14T14:07:44.817 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:07:44 smithi069 ceph-mon[29472]: pgmap v128: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:07:44.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:07:44 smithi122 ceph-mon[25721]: from='mgr.24749 ' entity='mgr.x' 2024-04-14T14:07:44.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:07:44 smithi122 ceph-mon[25721]: from='mgr.24749 ' entity='mgr.x' 2024-04-14T14:07:44.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:07:44 smithi122 ceph-mon[25721]: pgmap v128: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:07:45.317 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 14 14:07:44 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-y[63038]: 2024-04-14T14:07:44.961+0000 7fc794cd8200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-14T14:07:45.817 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 14 14:07:45 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-y[63038]: 2024-04-14T14:07:45.540+0000 7fc794cd8200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-14T14:07:45.817 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 14 14:07:45 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-y[63038]: 2024-04-14T14:07:45.642+0000 7fc794cd8200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-14T14:07:46.317 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 14 14:07:45 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-y[63038]: 2024-04-14T14:07:45.866+0000 7fc794cd8200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-14T14:07:46.317 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 14 14:07:45 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-y[63038]: [14/Apr/2024:14:07:45] ENGINE Bus STARTING 2024-04-14T14:07:46.317 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 14 14:07:45 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-y[63038]: CherryPy Checker: 2024-04-14T14:07:46.317 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 14 14:07:45 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-y[63038]: The Application mounted at '' has an empty config. 2024-04-14T14:07:46.317 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 14 14:07:45 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-y[63038]: 2024-04-14T14:07:46.317 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 14 14:07:45 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-y[63038]: [14/Apr/2024:14:07:45] ENGINE Serving on http://:::9283 2024-04-14T14:07:46.317 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 14 14:07:45 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-y[63038]: [14/Apr/2024:14:07:45] ENGINE Bus STARTED 2024-04-14T14:07:46.817 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:07:46 smithi069 ceph-mon[29472]: pgmap v129: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:07:46.817 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:07:46 smithi069 ceph-mon[29472]: from='client.15177 -' entity='client.iscsi.foo.smithi069.yuumtr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T14:07:46.817 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:07:46 smithi069 ceph-mon[29472]: Standby manager daemon y restarted 2024-04-14T14:07:46.817 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:07:46 smithi069 ceph-mon[29472]: Standby manager daemon y started 2024-04-14T14:07:46.817 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:07:46 smithi069 ceph-mon[29472]: from='mgr.? 172.21.15.69:0/4213893117' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-04-14T14:07:46.817 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:07:46 smithi069 ceph-mon[29472]: from='mgr.? 172.21.15.69:0/4213893117' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-14T14:07:46.817 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:07:46 smithi069 ceph-mon[29472]: from='mgr.? 172.21.15.69:0/4213893117' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-04-14T14:07:46.818 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:07:46 smithi069 ceph-mon[29472]: from='mgr.? 172.21.15.69:0/4213893117' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-14T14:07:46.818 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:07:46 smithi069 ceph-mon[24818]: pgmap v129: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:07:46.818 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:07:46 smithi069 ceph-mon[24818]: from='client.15177 -' entity='client.iscsi.foo.smithi069.yuumtr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T14:07:46.818 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:07:46 smithi069 ceph-mon[24818]: Standby manager daemon y restarted 2024-04-14T14:07:46.818 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:07:46 smithi069 ceph-mon[24818]: Standby manager daemon y started 2024-04-14T14:07:46.818 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:07:46 smithi069 ceph-mon[24818]: from='mgr.? 172.21.15.69:0/4213893117' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-04-14T14:07:46.818 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:07:46 smithi069 ceph-mon[24818]: from='mgr.? 172.21.15.69:0/4213893117' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-14T14:07:46.818 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:07:46 smithi069 ceph-mon[24818]: from='mgr.? 172.21.15.69:0/4213893117' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-04-14T14:07:46.818 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:07:46 smithi069 ceph-mon[24818]: from='mgr.? 172.21.15.69:0/4213893117' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-14T14:07:46.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:07:46 smithi122 ceph-mon[25721]: pgmap v129: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:07:46.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:07:46 smithi122 ceph-mon[25721]: from='client.15177 -' entity='client.iscsi.foo.smithi069.yuumtr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T14:07:46.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:07:46 smithi122 ceph-mon[25721]: Standby manager daemon y restarted 2024-04-14T14:07:46.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:07:46 smithi122 ceph-mon[25721]: Standby manager daemon y started 2024-04-14T14:07:46.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:07:46 smithi122 ceph-mon[25721]: from='mgr.? 172.21.15.69:0/4213893117' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-04-14T14:07:46.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:07:46 smithi122 ceph-mon[25721]: from='mgr.? 172.21.15.69:0/4213893117' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-14T14:07:46.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:07:46 smithi122 ceph-mon[25721]: from='mgr.? 172.21.15.69:0/4213893117' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-04-14T14:07:46.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:07:46 smithi122 ceph-mon[25721]: from='mgr.? 172.21.15.69:0/4213893117' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-14T14:07:47.817 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:07:47 smithi069 ceph-mon[24818]: mgrmap e26: x(active, since 3m), standbys: y 2024-04-14T14:07:47.817 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:07:47 smithi069 ceph-mon[29472]: mgrmap e26: x(active, since 3m), standbys: y 2024-04-14T14:07:47.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:07:47 smithi122 ceph-mon[25721]: mgrmap e26: x(active, since 3m), standbys: y 2024-04-14T14:07:48.817 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:07:48 smithi069 ceph-mon[24818]: pgmap v130: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:07:48.817 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:07:48 smithi069 ceph-mon[29472]: pgmap v130: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:07:48.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:07:48 smithi122 ceph-mon[25721]: pgmap v130: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:07:48.845 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Apr 14 14:07:48 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-prometheus-a[49277]: ts=2024-04-14T14:07:48.659Z caller=manager.go:609 level=warn component="rule manager" group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on(ceph_daemon) group_left(hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.69\", device_class=\"ssd\", hostname=\"smithi069\", instance=\"172.21.15.122:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.69\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster=\"dc326fc2-fa65-11ee-bc8e-c7b262605968\", cluster_addr=\"172.21.15.69\", device_class=\"ssd\", hostname=\"smithi069\", instance=\"smithi122:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.69\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-04-14T14:07:50.792 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:07:50 smithi069 ceph-mon[29472]: pgmap v131: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:07:50.792 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:07:50 smithi069 ceph-mon[24818]: pgmap v131: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:07:51.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:07:50 smithi122 ceph-mon[25721]: pgmap v131: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:07:52.567 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:07:52 smithi069 ceph-mon[24818]: from='mgr.24749 ' entity='mgr.x' 2024-04-14T14:07:52.567 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:07:52 smithi069 ceph-mon[24818]: from='mgr.24749 ' entity='mgr.x' 2024-04-14T14:07:52.567 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:07:52 smithi069 ceph-mon[24818]: from='mgr.24749 172.21.15.122:0/1892077634' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T14:07:52.567 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:07:52 smithi069 ceph-mon[24818]: from='mgr.24749 172.21.15.122:0/1892077634' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-14T14:07:52.567 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:07:52 smithi069 ceph-mon[24818]: pgmap v132: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 891 B/s rd, 0 op/s 2024-04-14T14:07:52.567 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:07:52 smithi069 ceph-mon[24818]: pgmap v133: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-04-14T14:07:52.567 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:07:52 smithi069 ceph-mon[24818]: from='mgr.24749 ' entity='mgr.x' 2024-04-14T14:07:52.568 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:07:52 smithi069 ceph-mon[29472]: from='mgr.24749 ' entity='mgr.x' 2024-04-14T14:07:52.568 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:07:52 smithi069 ceph-mon[29472]: from='mgr.24749 ' entity='mgr.x' 2024-04-14T14:07:52.568 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:07:52 smithi069 ceph-mon[29472]: from='mgr.24749 172.21.15.122:0/1892077634' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T14:07:52.568 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:07:52 smithi069 ceph-mon[29472]: from='mgr.24749 172.21.15.122:0/1892077634' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-14T14:07:52.568 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:07:52 smithi069 ceph-mon[29472]: pgmap v132: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 891 B/s rd, 0 op/s 2024-04-14T14:07:52.568 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:07:52 smithi069 ceph-mon[29472]: pgmap v133: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-04-14T14:07:52.568 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:07:52 smithi069 ceph-mon[29472]: from='mgr.24749 ' entity='mgr.x' 2024-04-14T14:07:52.595 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:07:52 smithi122 ceph-mon[25721]: from='mgr.24749 ' entity='mgr.x' 2024-04-14T14:07:52.595 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:07:52 smithi122 ceph-mon[25721]: from='mgr.24749 ' entity='mgr.x' 2024-04-14T14:07:52.595 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:07:52 smithi122 ceph-mon[25721]: from='mgr.24749 172.21.15.122:0/1892077634' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T14:07:52.595 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:07:52 smithi122 ceph-mon[25721]: from='mgr.24749 172.21.15.122:0/1892077634' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-14T14:07:52.595 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:07:52 smithi122 ceph-mon[25721]: pgmap v132: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 891 B/s rd, 0 op/s 2024-04-14T14:07:52.595 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:07:52 smithi122 ceph-mon[25721]: pgmap v133: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-04-14T14:07:52.595 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:07:52 smithi122 ceph-mon[25721]: from='mgr.24749 ' entity='mgr.x' 2024-04-14T14:07:53.005 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Apr 14 14:07:52 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-x[45487]: ::ffff:172.21.15.122 - - [14/Apr/2024:14:07:52] "GET /metrics HTTP/1.1" 200 37972 "" "Prometheus/2.33.4" 2024-04-14T14:07:53.845 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Apr 14 14:07:53 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-prometheus-a[49277]: ts=2024-04-14T14:07:53.504Z caller=manager.go:609 level=warn component="rule manager" group=nodes msg="Evaluating rule failed" rule="alert: CephNodeDiskspaceWarning\nexpr: predict_linear(node_filesystem_free_bytes{device=~\"/.*\"}[2d], 3600 * 24 * 5)\n * on(instance) group_left(nodename) node_uname_info < 0\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.8.4\n severity: warning\n type: ceph_default\nannotations:\n description: Mountpoint {{ $labels.mountpoint }} on {{ $labels.nodename }} will\n be full in less than 5 days based on the 48 hour trailing fill rate.\n summary: Host filesystem free space is getting low\n" err="found duplicate series for the match group {instance=\"smithi069\"} on the right hand-side of the operation: [{__name__=\"node_uname_info\", domainname=\"(none)\", instance=\"smithi069\", job=\"node\", machine=\"x86_64\", nodename=\"smithi069\", release=\"5.14.0-435.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Fri Mar 29 18:46:37 UTC 2024\"}, {__name__=\"node_uname_info\", cluster=\"dc326fc2-fa65-11ee-bc8e-c7b262605968\", domainname=\"(none)\", instance=\"smithi069\", job=\"node\", machine=\"x86_64\", nodename=\"smithi069\", release=\"5.14.0-435.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Fri Mar 29 18:46:37 UTC 2024\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-04-14T14:07:54.567 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:07:54 smithi069 ceph-mon[24818]: pgmap v134: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 539 B/s rd, 0 op/s 2024-04-14T14:07:54.567 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:07:54 smithi069 ceph-mon[29472]: pgmap v134: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 539 B/s rd, 0 op/s 2024-04-14T14:07:54.594 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:07:54 smithi122 ceph-mon[25721]: pgmap v134: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 539 B/s rd, 0 op/s 2024-04-14T14:07:56.817 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:07:56 smithi069 ceph-mon[24818]: pgmap v135: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-04-14T14:07:56.817 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:07:56 smithi069 ceph-mon[24818]: from='client.15177 -' entity='client.iscsi.foo.smithi069.yuumtr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T14:07:56.817 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:07:56 smithi069 ceph-mon[29472]: pgmap v135: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-04-14T14:07:56.817 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:07:56 smithi069 ceph-mon[29472]: from='client.15177 -' entity='client.iscsi.foo.smithi069.yuumtr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T14:07:56.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:07:56 smithi122 ceph-mon[25721]: pgmap v135: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-04-14T14:07:56.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:07:56 smithi122 ceph-mon[25721]: from='client.15177 -' entity='client.iscsi.foo.smithi069.yuumtr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T14:07:58.818 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:07:58 smithi069 ceph-mon[24818]: pgmap v136: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-04-14T14:07:58.818 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:07:58 smithi069 ceph-mon[24818]: from='mgr.24749 172.21.15.122:0/1892077634' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T14:07:58.819 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:07:58 smithi069 ceph-mon[29472]: pgmap v136: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-04-14T14:07:58.819 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:07:58 smithi069 ceph-mon[29472]: from='mgr.24749 172.21.15.122:0/1892077634' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T14:07:58.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:07:58 smithi122 ceph-mon[25721]: pgmap v136: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-04-14T14:07:58.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:07:58 smithi122 ceph-mon[25721]: from='mgr.24749 172.21.15.122:0/1892077634' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T14:07:58.845 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Apr 14 14:07:58 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-prometheus-a[49277]: ts=2024-04-14T14:07:58.660Z caller=manager.go:609 level=warn component="rule manager" group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on(ceph_daemon) group_left(hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.69\", device_class=\"ssd\", hostname=\"smithi069\", instance=\"172.21.15.122:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.69\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster=\"dc326fc2-fa65-11ee-bc8e-c7b262605968\", cluster_addr=\"172.21.15.69\", device_class=\"ssd\", hostname=\"smithi069\", instance=\"smithi122:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.69\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-04-14T14:08:00.817 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:08:00 smithi069 ceph-mon[24818]: pgmap v137: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-04-14T14:08:00.817 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:08:00 smithi069 ceph-mon[29472]: pgmap v137: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-04-14T14:08:00.844 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:08:00 smithi122 ceph-mon[25721]: pgmap v137: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-04-14T14:08:02.485 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:08:02 smithi122 ceph-mon[25721]: pgmap v138: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-14T14:08:02.817 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:08:02 smithi069 ceph-mon[24818]: pgmap v138: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-14T14:08:02.817 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:08:02 smithi069 ceph-mon[29472]: pgmap v138: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-14T14:08:03.095 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Apr 14 14:08:02 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-x[45487]: ::ffff:172.21.15.122 - - [14/Apr/2024:14:08:02] "GET /metrics HTTP/1.1" 200 37970 "" "Prometheus/2.33.4" 2024-04-14T14:08:03.807 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Apr 14 14:08:03 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-prometheus-a[49277]: ts=2024-04-14T14:08:03.504Z caller=manager.go:609 level=warn component="rule manager" group=nodes msg="Evaluating rule failed" rule="alert: CephNodeDiskspaceWarning\nexpr: predict_linear(node_filesystem_free_bytes{device=~\"/.*\"}[2d], 3600 * 24 * 5)\n * on(instance) group_left(nodename) node_uname_info < 0\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.8.4\n severity: warning\n type: ceph_default\nannotations:\n description: Mountpoint {{ $labels.mountpoint }} on {{ $labels.nodename }} will\n be full in less than 5 days based on the 48 hour trailing fill rate.\n summary: Host filesystem free space is getting low\n" err="found duplicate series for the match group {instance=\"smithi069\"} on the right hand-side of the operation: [{__name__=\"node_uname_info\", domainname=\"(none)\", instance=\"smithi069\", job=\"node\", machine=\"x86_64\", nodename=\"smithi069\", release=\"5.14.0-435.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Fri Mar 29 18:46:37 UTC 2024\"}, {__name__=\"node_uname_info\", cluster=\"dc326fc2-fa65-11ee-bc8e-c7b262605968\", domainname=\"(none)\", instance=\"smithi069\", job=\"node\", machine=\"x86_64\", nodename=\"smithi069\", release=\"5.14.0-435.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Fri Mar 29 18:46:37 UTC 2024\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-04-14T14:08:04.817 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:08:04 smithi069 ceph-mon[24818]: pgmap v139: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:08:04.817 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:08:04 smithi069 ceph-mon[29472]: pgmap v139: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:08:04.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:08:04 smithi122 ceph-mon[25721]: pgmap v139: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:08:06.817 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:08:06 smithi069 ceph-mon[24818]: pgmap v140: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:08:06.817 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:08:06 smithi069 ceph-mon[24818]: from='client.15177 -' entity='client.iscsi.foo.smithi069.yuumtr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T14:08:06.817 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:08:06 smithi069 ceph-mon[29472]: pgmap v140: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:08:06.817 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:08:06 smithi069 ceph-mon[29472]: from='client.15177 -' entity='client.iscsi.foo.smithi069.yuumtr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T14:08:06.844 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:08:06 smithi122 ceph-mon[25721]: pgmap v140: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:08:06.844 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:08:06 smithi122 ceph-mon[25721]: from='client.15177 -' entity='client.iscsi.foo.smithi069.yuumtr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T14:08:08.817 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:08:08 smithi069 ceph-mon[24818]: pgmap v141: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:08:08.817 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:08:08 smithi069 ceph-mon[29472]: pgmap v141: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:08:08.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:08:08 smithi122 ceph-mon[25721]: pgmap v141: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:08:08.845 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Apr 14 14:08:08 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-prometheus-a[49277]: ts=2024-04-14T14:08:08.660Z caller=manager.go:609 level=warn component="rule manager" group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on(ceph_daemon) group_left(hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.69\", device_class=\"ssd\", hostname=\"smithi069\", instance=\"172.21.15.122:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.69\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster=\"dc326fc2-fa65-11ee-bc8e-c7b262605968\", cluster_addr=\"172.21.15.69\", device_class=\"ssd\", hostname=\"smithi069\", instance=\"smithi122:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.69\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-04-14T14:08:10.817 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:08:10 smithi069 ceph-mon[24818]: pgmap v142: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:08:10.817 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:08:10 smithi069 ceph-mon[29472]: pgmap v142: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:08:10.844 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:08:10 smithi122 ceph-mon[25721]: pgmap v142: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:08:12.802 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:08:12 smithi122 ceph-mon[25721]: pgmap v143: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:08:12.816 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:08:12 smithi069 ceph-mon[24818]: pgmap v143: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:08:12.817 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:08:12 smithi069 ceph-mon[29472]: pgmap v143: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:08:13.095 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Apr 14 14:08:12 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-x[45487]: ::ffff:172.21.15.122 - - [14/Apr/2024:14:08:12] "GET /metrics HTTP/1.1" 200 37973 "" "Prometheus/2.33.4" 2024-04-14T14:08:13.817 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:08:13 smithi069 ceph-mon[24818]: from='mgr.24749 172.21.15.122:0/1892077634' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T14:08:13.817 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:08:13 smithi069 ceph-mon[29472]: from='mgr.24749 172.21.15.122:0/1892077634' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T14:08:13.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:08:13 smithi122 ceph-mon[25721]: from='mgr.24749 172.21.15.122:0/1892077634' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T14:08:13.845 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Apr 14 14:08:13 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-prometheus-a[49277]: ts=2024-04-14T14:08:13.504Z caller=manager.go:609 level=warn component="rule manager" group=nodes msg="Evaluating rule failed" rule="alert: CephNodeDiskspaceWarning\nexpr: predict_linear(node_filesystem_free_bytes{device=~\"/.*\"}[2d], 3600 * 24 * 5)\n * on(instance) group_left(nodename) node_uname_info < 0\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.8.4\n severity: warning\n type: ceph_default\nannotations:\n description: Mountpoint {{ $labels.mountpoint }} on {{ $labels.nodename }} will\n be full in less than 5 days based on the 48 hour trailing fill rate.\n summary: Host filesystem free space is getting low\n" err="found duplicate series for the match group {instance=\"smithi069\"} on the right hand-side of the operation: [{__name__=\"node_uname_info\", domainname=\"(none)\", instance=\"smithi069\", job=\"node\", machine=\"x86_64\", nodename=\"smithi069\", release=\"5.14.0-435.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Fri Mar 29 18:46:37 UTC 2024\"}, {__name__=\"node_uname_info\", cluster=\"dc326fc2-fa65-11ee-bc8e-c7b262605968\", domainname=\"(none)\", instance=\"smithi069\", job=\"node\", machine=\"x86_64\", nodename=\"smithi069\", release=\"5.14.0-435.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Fri Mar 29 18:46:37 UTC 2024\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-04-14T14:08:14.738 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:08:14 smithi069 ceph-mon[24818]: pgmap v144: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:08:14.738 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:08:14 smithi069 ceph-mon[29472]: pgmap v144: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:08:14.844 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:08:14 smithi122 ceph-mon[25721]: pgmap v144: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:08:16.818 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:08:16 smithi069 ceph-mon[24818]: pgmap v145: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:08:16.818 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:08:16 smithi069 ceph-mon[24818]: from='client.15177 -' entity='client.iscsi.foo.smithi069.yuumtr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T14:08:16.819 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:08:16 smithi069 ceph-mon[29472]: pgmap v145: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:08:16.819 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:08:16 smithi069 ceph-mon[29472]: from='client.15177 -' entity='client.iscsi.foo.smithi069.yuumtr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T14:08:16.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:08:16 smithi122 ceph-mon[25721]: pgmap v145: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:08:16.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:08:16 smithi122 ceph-mon[25721]: from='client.15177 -' entity='client.iscsi.foo.smithi069.yuumtr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T14:08:18.817 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:08:18 smithi069 ceph-mon[24818]: pgmap v146: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:08:18.817 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:08:18 smithi069 ceph-mon[29472]: pgmap v146: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:08:18.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:08:18 smithi122 ceph-mon[25721]: pgmap v146: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:08:18.845 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Apr 14 14:08:18 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-prometheus-a[49277]: ts=2024-04-14T14:08:18.659Z caller=manager.go:609 level=warn component="rule manager" group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on(ceph_daemon) group_left(hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.69\", device_class=\"ssd\", hostname=\"smithi069\", instance=\"172.21.15.122:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.69\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster=\"dc326fc2-fa65-11ee-bc8e-c7b262605968\", cluster_addr=\"172.21.15.69\", device_class=\"ssd\", hostname=\"smithi069\", instance=\"smithi122:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.69\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-04-14T14:08:20.817 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:08:20 smithi069 ceph-mon[24818]: pgmap v147: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:08:20.817 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:08:20 smithi069 ceph-mon[29472]: pgmap v147: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:08:20.844 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:08:20 smithi122 ceph-mon[25721]: pgmap v147: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:08:22.802 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:08:22 smithi122 ceph-mon[25721]: pgmap v148: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:08:22.816 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:08:22 smithi069 ceph-mon[24818]: pgmap v148: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:08:22.817 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:08:22 smithi069 ceph-mon[29472]: pgmap v148: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:08:23.095 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Apr 14 14:08:22 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-x[45487]: ::ffff:172.21.15.122 - - [14/Apr/2024:14:08:22] "GET /metrics HTTP/1.1" 200 37973 "" "Prometheus/2.33.4" 2024-04-14T14:08:23.845 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Apr 14 14:08:23 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-prometheus-a[49277]: ts=2024-04-14T14:08:23.504Z caller=manager.go:609 level=warn component="rule manager" group=nodes msg="Evaluating rule failed" rule="alert: CephNodeDiskspaceWarning\nexpr: predict_linear(node_filesystem_free_bytes{device=~\"/.*\"}[2d], 3600 * 24 * 5)\n * on(instance) group_left(nodename) node_uname_info < 0\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.8.4\n severity: warning\n type: ceph_default\nannotations:\n description: Mountpoint {{ $labels.mountpoint }} on {{ $labels.nodename }} will\n be full in less than 5 days based on the 48 hour trailing fill rate.\n summary: Host filesystem free space is getting low\n" err="found duplicate series for the match group {instance=\"smithi069\"} on the right hand-side of the operation: [{__name__=\"node_uname_info\", domainname=\"(none)\", instance=\"smithi069\", job=\"node\", machine=\"x86_64\", nodename=\"smithi069\", release=\"5.14.0-435.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Fri Mar 29 18:46:37 UTC 2024\"}, {__name__=\"node_uname_info\", cluster=\"dc326fc2-fa65-11ee-bc8e-c7b262605968\", domainname=\"(none)\", instance=\"smithi069\", job=\"node\", machine=\"x86_64\", nodename=\"smithi069\", release=\"5.14.0-435.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Fri Mar 29 18:46:37 UTC 2024\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-04-14T14:08:24.817 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:08:24 smithi069 ceph-mon[24818]: pgmap v149: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:08:24.817 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:08:24 smithi069 ceph-mon[29472]: pgmap v149: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:08:24.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:08:24 smithi122 ceph-mon[25721]: pgmap v149: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:08:26.817 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:08:26 smithi069 ceph-mon[29472]: pgmap v150: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:08:26.817 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:08:26 smithi069 ceph-mon[29472]: from='client.15177 -' entity='client.iscsi.foo.smithi069.yuumtr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T14:08:26.818 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:08:26 smithi069 ceph-mon[24818]: pgmap v150: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:08:26.818 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:08:26 smithi069 ceph-mon[24818]: from='client.15177 -' entity='client.iscsi.foo.smithi069.yuumtr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T14:08:26.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:08:26 smithi122 ceph-mon[25721]: pgmap v150: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:08:26.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:08:26 smithi122 ceph-mon[25721]: from='client.15177 -' entity='client.iscsi.foo.smithi069.yuumtr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T14:08:28.817 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:08:28 smithi069 ceph-mon[24818]: pgmap v151: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:08:28.817 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:08:28 smithi069 ceph-mon[24818]: from='mgr.24749 172.21.15.122:0/1892077634' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T14:08:28.817 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:08:28 smithi069 ceph-mon[29472]: pgmap v151: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:08:28.817 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:08:28 smithi069 ceph-mon[29472]: from='mgr.24749 172.21.15.122:0/1892077634' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T14:08:28.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:08:28 smithi122 ceph-mon[25721]: pgmap v151: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:08:28.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:08:28 smithi122 ceph-mon[25721]: from='mgr.24749 172.21.15.122:0/1892077634' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T14:08:28.845 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Apr 14 14:08:28 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-prometheus-a[49277]: ts=2024-04-14T14:08:28.660Z caller=manager.go:609 level=warn component="rule manager" group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on(ceph_daemon) group_left(hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.69\", device_class=\"ssd\", hostname=\"smithi069\", instance=\"172.21.15.122:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.69\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster=\"dc326fc2-fa65-11ee-bc8e-c7b262605968\", cluster_addr=\"172.21.15.69\", device_class=\"ssd\", hostname=\"smithi069\", instance=\"smithi122:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.69\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-04-14T14:08:30.817 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:08:30 smithi069 ceph-mon[24818]: pgmap v152: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:08:30.817 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:08:30 smithi069 ceph-mon[29472]: pgmap v152: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:08:30.844 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:08:30 smithi122 ceph-mon[25721]: pgmap v152: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:08:32.802 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:08:32 smithi122 ceph-mon[25721]: pgmap v153: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:08:32.817 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:08:32 smithi069 ceph-mon[24818]: pgmap v153: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:08:32.817 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:08:32 smithi069 ceph-mon[29472]: pgmap v153: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:08:33.095 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Apr 14 14:08:32 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-x[45487]: ::ffff:172.21.15.122 - - [14/Apr/2024:14:08:32] "GET /metrics HTTP/1.1" 200 37971 "" "Prometheus/2.33.4" 2024-04-14T14:08:33.845 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Apr 14 14:08:33 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-prometheus-a[49277]: ts=2024-04-14T14:08:33.504Z caller=manager.go:609 level=warn component="rule manager" group=nodes msg="Evaluating rule failed" rule="alert: CephNodeDiskspaceWarning\nexpr: predict_linear(node_filesystem_free_bytes{device=~\"/.*\"}[2d], 3600 * 24 * 5)\n * on(instance) group_left(nodename) node_uname_info < 0\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.8.4\n severity: warning\n type: ceph_default\nannotations:\n description: Mountpoint {{ $labels.mountpoint }} on {{ $labels.nodename }} will\n be full in less than 5 days based on the 48 hour trailing fill rate.\n summary: Host filesystem free space is getting low\n" err="found duplicate series for the match group {instance=\"smithi069\"} on the right hand-side of the operation: [{__name__=\"node_uname_info\", domainname=\"(none)\", instance=\"smithi069\", job=\"node\", machine=\"x86_64\", nodename=\"smithi069\", release=\"5.14.0-435.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Fri Mar 29 18:46:37 UTC 2024\"}, {__name__=\"node_uname_info\", cluster=\"dc326fc2-fa65-11ee-bc8e-c7b262605968\", domainname=\"(none)\", instance=\"smithi069\", job=\"node\", machine=\"x86_64\", nodename=\"smithi069\", release=\"5.14.0-435.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Fri Mar 29 18:46:37 UTC 2024\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-04-14T14:08:34.817 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:08:34 smithi069 ceph-mon[24818]: pgmap v154: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:08:34.817 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:08:34 smithi069 ceph-mon[29472]: pgmap v154: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:08:34.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:08:34 smithi122 ceph-mon[25721]: pgmap v154: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:08:36.817 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:08:36 smithi069 ceph-mon[24818]: pgmap v155: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:08:36.817 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:08:36 smithi069 ceph-mon[24818]: from='client.15177 -' entity='client.iscsi.foo.smithi069.yuumtr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T14:08:36.818 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:08:36 smithi069 ceph-mon[29472]: pgmap v155: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:08:36.818 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:08:36 smithi069 ceph-mon[29472]: from='client.15177 -' entity='client.iscsi.foo.smithi069.yuumtr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T14:08:36.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:08:36 smithi122 ceph-mon[25721]: pgmap v155: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:08:36.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:08:36 smithi122 ceph-mon[25721]: from='client.15177 -' entity='client.iscsi.foo.smithi069.yuumtr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T14:08:38.817 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:08:38 smithi069 ceph-mon[24818]: pgmap v156: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:08:38.817 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:08:38 smithi069 ceph-mon[29472]: pgmap v156: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:08:38.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:08:38 smithi122 ceph-mon[25721]: pgmap v156: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:08:38.846 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Apr 14 14:08:38 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-prometheus-a[49277]: ts=2024-04-14T14:08:38.660Z caller=manager.go:609 level=warn component="rule manager" group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on(ceph_daemon) group_left(hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.69\", device_class=\"ssd\", hostname=\"smithi069\", instance=\"172.21.15.122:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.69\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster=\"dc326fc2-fa65-11ee-bc8e-c7b262605968\", cluster_addr=\"172.21.15.69\", device_class=\"ssd\", hostname=\"smithi069\", instance=\"smithi122:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.69\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-04-14T14:08:40.817 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:08:40 smithi069 ceph-mon[24818]: pgmap v157: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:08:40.817 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:08:40 smithi069 ceph-mon[29472]: pgmap v157: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:08:40.844 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:08:40 smithi122 ceph-mon[25721]: pgmap v157: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:08:42.802 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:08:42 smithi122 ceph-mon[25721]: pgmap v158: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:08:42.817 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:08:42 smithi069 ceph-mon[24818]: pgmap v158: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:08:42.817 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:08:42 smithi069 ceph-mon[29472]: pgmap v158: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:08:43.095 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Apr 14 14:08:42 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-x[45487]: ::ffff:172.21.15.122 - - [14/Apr/2024:14:08:42] "GET /metrics HTTP/1.1" 200 37971 "" "Prometheus/2.33.4" 2024-04-14T14:08:43.817 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:08:43 smithi069 ceph-mon[24818]: from='mgr.24749 172.21.15.122:0/1892077634' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T14:08:43.817 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:08:43 smithi069 ceph-mon[29472]: from='mgr.24749 172.21.15.122:0/1892077634' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T14:08:43.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:08:43 smithi122 ceph-mon[25721]: from='mgr.24749 172.21.15.122:0/1892077634' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T14:08:43.845 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Apr 14 14:08:43 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-prometheus-a[49277]: ts=2024-04-14T14:08:43.504Z caller=manager.go:609 level=warn component="rule manager" group=nodes msg="Evaluating rule failed" rule="alert: CephNodeDiskspaceWarning\nexpr: predict_linear(node_filesystem_free_bytes{device=~\"/.*\"}[2d], 3600 * 24 * 5)\n * on(instance) group_left(nodename) node_uname_info < 0\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.8.4\n severity: warning\n type: ceph_default\nannotations:\n description: Mountpoint {{ $labels.mountpoint }} on {{ $labels.nodename }} will\n be full in less than 5 days based on the 48 hour trailing fill rate.\n summary: Host filesystem free space is getting low\n" err="found duplicate series for the match group {instance=\"smithi069\"} on the right hand-side of the operation: [{__name__=\"node_uname_info\", domainname=\"(none)\", instance=\"smithi069\", job=\"node\", machine=\"x86_64\", nodename=\"smithi069\", release=\"5.14.0-435.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Fri Mar 29 18:46:37 UTC 2024\"}, {__name__=\"node_uname_info\", cluster=\"dc326fc2-fa65-11ee-bc8e-c7b262605968\", domainname=\"(none)\", instance=\"smithi069\", job=\"node\", machine=\"x86_64\", nodename=\"smithi069\", release=\"5.14.0-435.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Fri Mar 29 18:46:37 UTC 2024\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-04-14T14:08:44.817 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:08:44 smithi069 ceph-mon[24818]: pgmap v159: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:08:44.817 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:08:44 smithi069 ceph-mon[29472]: pgmap v159: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:08:44.844 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:08:44 smithi122 ceph-mon[25721]: pgmap v159: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:08:46.817 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:08:46 smithi069 ceph-mon[24818]: pgmap v160: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:08:46.817 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:08:46 smithi069 ceph-mon[24818]: from='client.15177 -' entity='client.iscsi.foo.smithi069.yuumtr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T14:08:46.817 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:08:46 smithi069 ceph-mon[29472]: pgmap v160: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:08:46.817 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:08:46 smithi069 ceph-mon[29472]: from='client.15177 -' entity='client.iscsi.foo.smithi069.yuumtr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T14:08:46.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:08:46 smithi122 ceph-mon[25721]: pgmap v160: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:08:46.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:08:46 smithi122 ceph-mon[25721]: from='client.15177 -' entity='client.iscsi.foo.smithi069.yuumtr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T14:08:48.817 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:08:48 smithi069 ceph-mon[24818]: pgmap v161: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:08:48.817 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:08:48 smithi069 ceph-mon[29472]: pgmap v161: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:08:48.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:08:48 smithi122 ceph-mon[25721]: pgmap v161: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:08:48.845 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Apr 14 14:08:48 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-prometheus-a[49277]: ts=2024-04-14T14:08:48.659Z caller=manager.go:609 level=warn component="rule manager" group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on(ceph_daemon) group_left(hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.69\", device_class=\"ssd\", hostname=\"smithi069\", instance=\"172.21.15.122:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.69\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster=\"dc326fc2-fa65-11ee-bc8e-c7b262605968\", cluster_addr=\"172.21.15.69\", device_class=\"ssd\", hostname=\"smithi069\", instance=\"smithi122:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.69\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-04-14T14:08:50.817 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:08:50 smithi069 ceph-mon[24818]: pgmap v162: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:08:50.817 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:08:50 smithi069 ceph-mon[29472]: pgmap v162: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:08:50.844 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:08:50 smithi122 ceph-mon[25721]: pgmap v162: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:08:52.802 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:08:52 smithi122 ceph-mon[25721]: pgmap v163: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:08:52.817 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:08:52 smithi069 ceph-mon[24818]: pgmap v163: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:08:52.817 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:08:52 smithi069 ceph-mon[29472]: pgmap v163: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:08:53.095 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Apr 14 14:08:52 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-x[45487]: ::ffff:172.21.15.122 - - [14/Apr/2024:14:08:52] "GET /metrics HTTP/1.1" 200 37971 "" "Prometheus/2.33.4" 2024-04-14T14:08:53.845 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Apr 14 14:08:53 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-prometheus-a[49277]: ts=2024-04-14T14:08:53.504Z caller=manager.go:609 level=warn component="rule manager" group=nodes msg="Evaluating rule failed" rule="alert: CephNodeDiskspaceWarning\nexpr: predict_linear(node_filesystem_free_bytes{device=~\"/.*\"}[2d], 3600 * 24 * 5)\n * on(instance) group_left(nodename) node_uname_info < 0\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.8.4\n severity: warning\n type: ceph_default\nannotations:\n description: Mountpoint {{ $labels.mountpoint }} on {{ $labels.nodename }} will\n be full in less than 5 days based on the 48 hour trailing fill rate.\n summary: Host filesystem free space is getting low\n" err="found duplicate series for the match group {instance=\"smithi069\"} on the right hand-side of the operation: [{__name__=\"node_uname_info\", domainname=\"(none)\", instance=\"smithi069\", job=\"node\", machine=\"x86_64\", nodename=\"smithi069\", release=\"5.14.0-435.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Fri Mar 29 18:46:37 UTC 2024\"}, {__name__=\"node_uname_info\", cluster=\"dc326fc2-fa65-11ee-bc8e-c7b262605968\", domainname=\"(none)\", instance=\"smithi069\", job=\"node\", machine=\"x86_64\", nodename=\"smithi069\", release=\"5.14.0-435.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Fri Mar 29 18:46:37 UTC 2024\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-04-14T14:08:54.817 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:08:54 smithi069 ceph-mon[24818]: pgmap v164: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:08:54.817 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:08:54 smithi069 ceph-mon[29472]: pgmap v164: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:08:54.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:08:54 smithi122 ceph-mon[25721]: pgmap v164: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:08:56.817 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:08:56 smithi069 ceph-mon[29472]: pgmap v165: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:08:56.817 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:08:56 smithi069 ceph-mon[29472]: from='mgr.24749 ' entity='mgr.x' 2024-04-14T14:08:56.817 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:08:56 smithi069 ceph-mon[29472]: from='mgr.24749 ' entity='mgr.x' 2024-04-14T14:08:56.817 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:08:56 smithi069 ceph-mon[29472]: from='client.15177 -' entity='client.iscsi.foo.smithi069.yuumtr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T14:08:56.817 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:08:56 smithi069 ceph-mon[29472]: from='mgr.24749 172.21.15.122:0/1892077634' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T14:08:56.818 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:08:56 smithi069 ceph-mon[29472]: from='mgr.24749 172.21.15.122:0/1892077634' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-14T14:08:56.818 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:08:56 smithi069 ceph-mon[29472]: from='mgr.24749 ' entity='mgr.x' 2024-04-14T14:08:56.818 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:08:56 smithi069 ceph-mon[24818]: pgmap v165: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:08:56.818 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:08:56 smithi069 ceph-mon[24818]: from='mgr.24749 ' entity='mgr.x' 2024-04-14T14:08:56.818 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:08:56 smithi069 ceph-mon[24818]: from='mgr.24749 ' entity='mgr.x' 2024-04-14T14:08:56.818 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:08:56 smithi069 ceph-mon[24818]: from='client.15177 -' entity='client.iscsi.foo.smithi069.yuumtr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T14:08:56.818 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:08:56 smithi069 ceph-mon[24818]: from='mgr.24749 172.21.15.122:0/1892077634' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T14:08:56.818 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:08:56 smithi069 ceph-mon[24818]: from='mgr.24749 172.21.15.122:0/1892077634' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-14T14:08:56.818 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:08:56 smithi069 ceph-mon[24818]: from='mgr.24749 ' entity='mgr.x' 2024-04-14T14:08:56.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:08:56 smithi122 ceph-mon[25721]: pgmap v165: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:08:56.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:08:56 smithi122 ceph-mon[25721]: from='mgr.24749 ' entity='mgr.x' 2024-04-14T14:08:56.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:08:56 smithi122 ceph-mon[25721]: from='mgr.24749 ' entity='mgr.x' 2024-04-14T14:08:56.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:08:56 smithi122 ceph-mon[25721]: from='client.15177 -' entity='client.iscsi.foo.smithi069.yuumtr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T14:08:56.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:08:56 smithi122 ceph-mon[25721]: from='mgr.24749 172.21.15.122:0/1892077634' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T14:08:56.846 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:08:56 smithi122 ceph-mon[25721]: from='mgr.24749 172.21.15.122:0/1892077634' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-14T14:08:56.846 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:08:56 smithi122 ceph-mon[25721]: from='mgr.24749 ' entity='mgr.x' 2024-04-14T14:08:57.817 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:08:57 smithi069 ceph-mon[24818]: pgmap v166: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 932 B/s rd, 0 op/s 2024-04-14T14:08:57.817 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:08:57 smithi069 ceph-mon[24818]: pgmap v167: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-04-14T14:08:57.817 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:08:57 smithi069 ceph-mon[29472]: pgmap v166: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 932 B/s rd, 0 op/s 2024-04-14T14:08:57.817 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:08:57 smithi069 ceph-mon[29472]: pgmap v167: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-04-14T14:08:57.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:08:57 smithi122 ceph-mon[25721]: pgmap v166: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 932 B/s rd, 0 op/s 2024-04-14T14:08:57.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:08:57 smithi122 ceph-mon[25721]: pgmap v167: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-04-14T14:08:58.817 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:08:58 smithi069 ceph-mon[24818]: from='mgr.24749 172.21.15.122:0/1892077634' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T14:08:58.817 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:08:58 smithi069 ceph-mon[29472]: from='mgr.24749 172.21.15.122:0/1892077634' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T14:08:58.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:08:58 smithi122 ceph-mon[25721]: from='mgr.24749 172.21.15.122:0/1892077634' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T14:08:58.845 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Apr 14 14:08:58 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-prometheus-a[49277]: ts=2024-04-14T14:08:58.659Z caller=manager.go:609 level=warn component="rule manager" group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on(ceph_daemon) group_left(hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.69\", device_class=\"ssd\", hostname=\"smithi069\", instance=\"172.21.15.122:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.69\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster=\"dc326fc2-fa65-11ee-bc8e-c7b262605968\", cluster_addr=\"172.21.15.69\", device_class=\"ssd\", hostname=\"smithi069\", instance=\"smithi122:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.69\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-04-14T14:08:59.817 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:08:59 smithi069 ceph-mon[24818]: pgmap v168: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 569 B/s rd, 0 op/s 2024-04-14T14:08:59.817 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:08:59 smithi069 ceph-mon[29472]: pgmap v168: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 569 B/s rd, 0 op/s 2024-04-14T14:08:59.844 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:08:59 smithi122 ceph-mon[25721]: pgmap v168: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 569 B/s rd, 0 op/s 2024-04-14T14:09:01.817 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:09:01 smithi069 ceph-mon[24818]: pgmap v169: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-04-14T14:09:01.818 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:09:01 smithi069 ceph-mon[29472]: pgmap v169: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-04-14T14:09:01.844 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:09:01 smithi122 ceph-mon[25721]: pgmap v169: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-04-14T14:09:03.095 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Apr 14 14:09:02 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-x[45487]: ::ffff:172.21.15.122 - - [14/Apr/2024:14:09:02] "GET /metrics HTTP/1.1" 200 37969 "" "Prometheus/2.33.4" 2024-04-14T14:09:03.817 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:09:03 smithi069 ceph-mon[24818]: pgmap v170: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-04-14T14:09:03.817 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:09:03 smithi069 ceph-mon[29472]: pgmap v170: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-04-14T14:09:03.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:09:03 smithi122 ceph-mon[25721]: pgmap v170: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-04-14T14:09:03.845 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Apr 14 14:09:03 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-prometheus-a[49277]: ts=2024-04-14T14:09:03.504Z caller=manager.go:609 level=warn component="rule manager" group=nodes msg="Evaluating rule failed" rule="alert: CephNodeDiskspaceWarning\nexpr: predict_linear(node_filesystem_free_bytes{device=~\"/.*\"}[2d], 3600 * 24 * 5)\n * on(instance) group_left(nodename) node_uname_info < 0\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.8.4\n severity: warning\n type: ceph_default\nannotations:\n description: Mountpoint {{ $labels.mountpoint }} on {{ $labels.nodename }} will\n be full in less than 5 days based on the 48 hour trailing fill rate.\n summary: Host filesystem free space is getting low\n" err="found duplicate series for the match group {instance=\"smithi069\"} on the right hand-side of the operation: [{__name__=\"node_uname_info\", domainname=\"(none)\", instance=\"smithi069\", job=\"node\", machine=\"x86_64\", nodename=\"smithi069\", release=\"5.14.0-435.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Fri Mar 29 18:46:37 UTC 2024\"}, {__name__=\"node_uname_info\", cluster=\"dc326fc2-fa65-11ee-bc8e-c7b262605968\", domainname=\"(none)\", instance=\"smithi069\", job=\"node\", machine=\"x86_64\", nodename=\"smithi069\", release=\"5.14.0-435.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Fri Mar 29 18:46:37 UTC 2024\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-04-14T14:09:05.817 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:09:05 smithi069 ceph-mon[24818]: pgmap v171: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-04-14T14:09:05.817 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:09:05 smithi069 ceph-mon[29472]: pgmap v171: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-04-14T14:09:05.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:09:05 smithi122 ceph-mon[25721]: pgmap v171: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-04-14T14:09:06.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:09:06 smithi122 ceph-mon[25721]: from='client.15177 -' entity='client.iscsi.foo.smithi069.yuumtr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T14:09:07.067 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:09:06 smithi069 ceph-mon[24818]: from='client.15177 -' entity='client.iscsi.foo.smithi069.yuumtr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T14:09:07.067 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:09:06 smithi069 ceph-mon[29472]: from='client.15177 -' entity='client.iscsi.foo.smithi069.yuumtr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T14:09:07.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:09:07 smithi122 ceph-mon[25721]: pgmap v172: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-14T14:09:08.067 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:09:07 smithi069 ceph-mon[24818]: pgmap v172: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-14T14:09:08.067 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:09:07 smithi069 ceph-mon[29472]: pgmap v172: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-14T14:09:09.095 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Apr 14 14:09:08 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-prometheus-a[49277]: ts=2024-04-14T14:09:08.660Z caller=manager.go:609 level=warn component="rule manager" group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on(ceph_daemon) group_left(hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.69\", device_class=\"ssd\", hostname=\"smithi069\", instance=\"172.21.15.122:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.69\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster=\"dc326fc2-fa65-11ee-bc8e-c7b262605968\", cluster_addr=\"172.21.15.69\", device_class=\"ssd\", hostname=\"smithi069\", instance=\"smithi122:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.69\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-04-14T14:09:09.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:09:09 smithi122 ceph-mon[25721]: pgmap v173: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:09:10.067 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:09:09 smithi069 ceph-mon[24818]: pgmap v173: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:09:10.067 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:09:09 smithi069 ceph-mon[29472]: pgmap v173: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:09:11.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:09:11 smithi122 ceph-mon[25721]: pgmap v174: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:09:12.067 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:09:11 smithi069 ceph-mon[24818]: pgmap v174: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:09:12.067 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:09:11 smithi069 ceph-mon[29472]: pgmap v174: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:09:13.095 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Apr 14 14:09:12 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-x[45487]: ::ffff:172.21.15.122 - - [14/Apr/2024:14:09:12] "GET /metrics HTTP/1.1" 200 37967 "" "Prometheus/2.33.4" 2024-04-14T14:09:13.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:09:13 smithi122 ceph-mon[25721]: pgmap v175: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:09:13.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:09:13 smithi122 ceph-mon[25721]: from='mgr.24749 172.21.15.122:0/1892077634' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T14:09:13.845 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Apr 14 14:09:13 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-prometheus-a[49277]: ts=2024-04-14T14:09:13.504Z caller=manager.go:609 level=warn component="rule manager" group=nodes msg="Evaluating rule failed" rule="alert: CephNodeDiskspaceWarning\nexpr: predict_linear(node_filesystem_free_bytes{device=~\"/.*\"}[2d], 3600 * 24 * 5)\n * on(instance) group_left(nodename) node_uname_info < 0\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.8.4\n severity: warning\n type: ceph_default\nannotations:\n description: Mountpoint {{ $labels.mountpoint }} on {{ $labels.nodename }} will\n be full in less than 5 days based on the 48 hour trailing fill rate.\n summary: Host filesystem free space is getting low\n" err="found duplicate series for the match group {instance=\"smithi069\"} on the right hand-side of the operation: [{__name__=\"node_uname_info\", domainname=\"(none)\", instance=\"smithi069\", job=\"node\", machine=\"x86_64\", nodename=\"smithi069\", release=\"5.14.0-435.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Fri Mar 29 18:46:37 UTC 2024\"}, {__name__=\"node_uname_info\", cluster=\"dc326fc2-fa65-11ee-bc8e-c7b262605968\", domainname=\"(none)\", instance=\"smithi069\", job=\"node\", machine=\"x86_64\", nodename=\"smithi069\", release=\"5.14.0-435.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Fri Mar 29 18:46:37 UTC 2024\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-04-14T14:09:14.067 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:09:13 smithi069 ceph-mon[24818]: pgmap v175: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:09:14.067 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:09:13 smithi069 ceph-mon[24818]: from='mgr.24749 172.21.15.122:0/1892077634' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T14:09:14.067 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:09:13 smithi069 ceph-mon[29472]: pgmap v175: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:09:14.067 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:09:13 smithi069 ceph-mon[29472]: from='mgr.24749 172.21.15.122:0/1892077634' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T14:09:15.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:09:15 smithi122 ceph-mon[25721]: pgmap v176: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:09:15.869 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:09:15 smithi069 ceph-mon[24818]: pgmap v176: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:09:15.869 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:09:15 smithi069 ceph-mon[29472]: pgmap v176: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:09:16.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:09:16 smithi122 ceph-mon[25721]: from='client.15177 -' entity='client.iscsi.foo.smithi069.yuumtr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T14:09:16.887 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:09:16 smithi069 ceph-mon[24818]: from='client.15177 -' entity='client.iscsi.foo.smithi069.yuumtr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T14:09:16.887 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:09:16 smithi069 ceph-mon[29472]: from='client.15177 -' entity='client.iscsi.foo.smithi069.yuumtr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T14:09:18.037 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:09:17 smithi069 ceph-mon[24818]: pgmap v177: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:09:18.038 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:09:17 smithi069 ceph-mon[29472]: pgmap v177: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:09:18.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:09:17 smithi122 ceph-mon[25721]: pgmap v177: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:09:19.095 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Apr 14 14:09:18 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-prometheus-a[49277]: ts=2024-04-14T14:09:18.659Z caller=manager.go:609 level=warn component="rule manager" group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on(ceph_daemon) group_left(hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.69\", device_class=\"ssd\", hostname=\"smithi069\", instance=\"172.21.15.122:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.69\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster=\"dc326fc2-fa65-11ee-bc8e-c7b262605968\", cluster_addr=\"172.21.15.69\", device_class=\"ssd\", hostname=\"smithi069\", instance=\"smithi122:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.69\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-04-14T14:09:20.067 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:09:19 smithi069 ceph-mon[24818]: pgmap v178: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:09:20.067 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:09:19 smithi069 ceph-mon[29472]: pgmap v178: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:09:20.094 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:09:19 smithi122 ceph-mon[25721]: pgmap v178: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:09:22.067 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:09:21 smithi069 ceph-mon[29472]: pgmap v179: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:09:22.067 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:09:21 smithi069 ceph-mon[24818]: pgmap v179: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:09:22.094 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:09:21 smithi122 ceph-mon[25721]: pgmap v179: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:09:23.095 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Apr 14 14:09:22 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-x[45487]: ::ffff:172.21.15.122 - - [14/Apr/2024:14:09:22] "GET /metrics HTTP/1.1" 200 37967 "" "Prometheus/2.33.4" 2024-04-14T14:09:23.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:09:23 smithi122 ceph-mon[25721]: pgmap v180: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:09:23.845 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Apr 14 14:09:23 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-prometheus-a[49277]: ts=2024-04-14T14:09:23.504Z caller=manager.go:609 level=warn component="rule manager" group=nodes msg="Evaluating rule failed" rule="alert: CephNodeDiskspaceWarning\nexpr: predict_linear(node_filesystem_free_bytes{device=~\"/.*\"}[2d], 3600 * 24 * 5)\n * on(instance) group_left(nodename) node_uname_info < 0\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.8.4\n severity: warning\n type: ceph_default\nannotations:\n description: Mountpoint {{ $labels.mountpoint }} on {{ $labels.nodename }} will\n be full in less than 5 days based on the 48 hour trailing fill rate.\n summary: Host filesystem free space is getting low\n" err="found duplicate series for the match group {instance=\"smithi069\"} on the right hand-side of the operation: [{__name__=\"node_uname_info\", domainname=\"(none)\", instance=\"smithi069\", job=\"node\", machine=\"x86_64\", nodename=\"smithi069\", release=\"5.14.0-435.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Fri Mar 29 18:46:37 UTC 2024\"}, {__name__=\"node_uname_info\", cluster=\"dc326fc2-fa65-11ee-bc8e-c7b262605968\", domainname=\"(none)\", instance=\"smithi069\", job=\"node\", machine=\"x86_64\", nodename=\"smithi069\", release=\"5.14.0-435.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Fri Mar 29 18:46:37 UTC 2024\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-04-14T14:09:24.067 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:09:23 smithi069 ceph-mon[24818]: pgmap v180: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:09:24.067 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:09:23 smithi069 ceph-mon[29472]: pgmap v180: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:09:26.067 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:09:25 smithi069 ceph-mon[24818]: pgmap v181: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:09:26.067 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:09:25 smithi069 ceph-mon[29472]: pgmap v181: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:09:26.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:09:25 smithi122 ceph-mon[25721]: pgmap v181: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:09:27.067 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:09:26 smithi069 ceph-mon[24818]: from='client.15177 -' entity='client.iscsi.foo.smithi069.yuumtr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T14:09:27.067 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:09:26 smithi069 ceph-mon[29472]: from='client.15177 -' entity='client.iscsi.foo.smithi069.yuumtr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T14:09:27.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:09:26 smithi122 ceph-mon[25721]: from='client.15177 -' entity='client.iscsi.foo.smithi069.yuumtr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T14:09:27.817 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:09:27 smithi069 ceph-mon[24818]: pgmap v182: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:09:27.817 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:09:27 smithi069 ceph-mon[29472]: pgmap v182: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:09:28.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:09:27 smithi122 ceph-mon[25721]: pgmap v182: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:09:29.067 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:09:28 smithi069 ceph-mon[24818]: from='mgr.24749 172.21.15.122:0/1892077634' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T14:09:29.067 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:09:28 smithi069 ceph-mon[29472]: from='mgr.24749 172.21.15.122:0/1892077634' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T14:09:29.094 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:09:28 smithi122 ceph-mon[25721]: from='mgr.24749 172.21.15.122:0/1892077634' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T14:09:30.067 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:09:29 smithi069 ceph-mon[24818]: pgmap v183: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:09:30.067 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:09:29 smithi069 ceph-mon[29472]: pgmap v183: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:09:30.094 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:09:29 smithi122 ceph-mon[25721]: pgmap v183: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:09:32.067 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:09:31 smithi069 ceph-mon[24818]: pgmap v184: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:09:32.067 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:09:31 smithi069 ceph-mon[29472]: pgmap v184: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:09:32.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:09:31 smithi122 ceph-mon[25721]: pgmap v184: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:09:33.095 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Apr 14 14:09:32 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-x[45487]: ::ffff:172.21.15.122 - - [14/Apr/2024:14:09:32] "GET /metrics HTTP/1.1" 200 37968 "" "Prometheus/2.33.4" 2024-04-14T14:09:34.067 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:09:33 smithi069 ceph-mon[24818]: pgmap v185: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:09:34.067 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:09:33 smithi069 ceph-mon[29472]: pgmap v185: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:09:34.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:09:33 smithi122 ceph-mon[25721]: pgmap v185: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:09:36.067 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:09:35 smithi069 ceph-mon[24818]: pgmap v186: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:09:36.067 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:09:35 smithi069 ceph-mon[29472]: pgmap v186: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:09:36.094 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:09:35 smithi122 ceph-mon[25721]: pgmap v186: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:09:37.067 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:09:36 smithi069 ceph-mon[29472]: from='client.15177 -' entity='client.iscsi.foo.smithi069.yuumtr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T14:09:37.067 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:09:36 smithi069 ceph-mon[24818]: from='client.15177 -' entity='client.iscsi.foo.smithi069.yuumtr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T14:09:37.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:09:36 smithi122 ceph-mon[25721]: from='client.15177 -' entity='client.iscsi.foo.smithi069.yuumtr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T14:09:38.067 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:09:37 smithi069 ceph-mon[24818]: pgmap v187: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:09:38.067 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:09:37 smithi069 ceph-mon[29472]: pgmap v187: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:09:38.094 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:09:37 smithi122 ceph-mon[25721]: pgmap v187: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:09:40.067 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:09:39 smithi069 ceph-mon[24818]: pgmap v188: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:09:40.067 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:09:39 smithi069 ceph-mon[29472]: pgmap v188: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:09:40.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:09:39 smithi122 ceph-mon[25721]: pgmap v188: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:09:42.067 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:09:41 smithi069 ceph-mon[24818]: pgmap v189: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:09:42.067 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:09:41 smithi069 ceph-mon[29472]: pgmap v189: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:09:42.094 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:09:41 smithi122 ceph-mon[25721]: pgmap v189: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:09:43.067 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:09:42 smithi069 ceph-mon[24818]: from='mgr.24749 172.21.15.122:0/1892077634' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T14:09:43.067 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:09:42 smithi069 ceph-mon[29472]: from='mgr.24749 172.21.15.122:0/1892077634' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T14:09:43.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:09:42 smithi122 ceph-mon[25721]: from='mgr.24749 172.21.15.122:0/1892077634' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T14:09:43.095 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Apr 14 14:09:42 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-x[45487]: ::ffff:172.21.15.122 - - [14/Apr/2024:14:09:42] "GET /metrics HTTP/1.1" 200 37969 "" "Prometheus/2.33.4" 2024-04-14T14:09:44.067 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:09:43 smithi069 ceph-mon[29472]: pgmap v190: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:09:44.067 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:09:43 smithi069 ceph-mon[24818]: pgmap v190: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:09:44.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:09:43 smithi122 ceph-mon[25721]: pgmap v190: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:09:46.067 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:09:45 smithi069 ceph-mon[24818]: pgmap v191: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:09:46.068 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:09:45 smithi069 ceph-mon[29472]: pgmap v191: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:09:46.094 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:09:45 smithi122 ceph-mon[25721]: pgmap v191: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:09:47.067 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:09:46 smithi069 ceph-mon[24818]: from='client.15177 -' entity='client.iscsi.foo.smithi069.yuumtr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T14:09:47.067 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:09:46 smithi069 ceph-mon[29472]: from='client.15177 -' entity='client.iscsi.foo.smithi069.yuumtr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T14:09:47.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:09:46 smithi122 ceph-mon[25721]: from='client.15177 -' entity='client.iscsi.foo.smithi069.yuumtr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T14:09:48.067 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:09:47 smithi069 ceph-mon[24818]: pgmap v192: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:09:48.067 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:09:47 smithi069 ceph-mon[29472]: pgmap v192: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:09:48.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:09:47 smithi122 ceph-mon[25721]: pgmap v192: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:09:50.067 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:09:49 smithi069 ceph-mon[24818]: pgmap v193: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:09:50.067 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:09:49 smithi069 ceph-mon[29472]: pgmap v193: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:09:50.094 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:09:49 smithi122 ceph-mon[25721]: pgmap v193: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:09:52.067 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:09:51 smithi069 ceph-mon[24818]: pgmap v194: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:09:52.067 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:09:51 smithi069 ceph-mon[29472]: pgmap v194: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:09:52.094 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:09:51 smithi122 ceph-mon[25721]: pgmap v194: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:09:53.095 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Apr 14 14:09:52 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-x[45487]: ::ffff:172.21.15.122 - - [14/Apr/2024:14:09:52] "GET /metrics HTTP/1.1" 200 37969 "" "Prometheus/2.33.4" 2024-04-14T14:09:54.067 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:09:53 smithi069 ceph-mon[24818]: pgmap v195: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:09:54.067 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:09:53 smithi069 ceph-mon[29472]: pgmap v195: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:09:54.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:09:53 smithi122 ceph-mon[25721]: pgmap v195: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:09:56.067 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:09:55 smithi069 ceph-mon[24818]: pgmap v196: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:09:56.067 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:09:55 smithi069 ceph-mon[29472]: pgmap v196: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:09:56.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:09:55 smithi122 ceph-mon[25721]: pgmap v196: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:09:57.067 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:09:56 smithi069 ceph-mon[24818]: from='client.15177 -' entity='client.iscsi.foo.smithi069.yuumtr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T14:09:57.067 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:09:56 smithi069 ceph-mon[29472]: from='client.15177 -' entity='client.iscsi.foo.smithi069.yuumtr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T14:09:57.094 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:09:56 smithi122 ceph-mon[25721]: from='client.15177 -' entity='client.iscsi.foo.smithi069.yuumtr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T14:09:58.067 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:09:57 smithi069 ceph-mon[24818]: pgmap v197: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:09:58.067 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:09:57 smithi069 ceph-mon[24818]: from='mgr.24749 172.21.15.122:0/1892077634' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T14:09:58.067 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:09:57 smithi069 ceph-mon[24818]: from='mgr.24749 172.21.15.122:0/1892077634' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-14T14:09:58.067 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:09:57 smithi069 ceph-mon[24818]: pgmap v198: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 948 B/s rd, 0 op/s 2024-04-14T14:09:58.067 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:09:57 smithi069 ceph-mon[24818]: pgmap v199: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-04-14T14:09:58.067 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:09:57 smithi069 ceph-mon[24818]: from='mgr.24749 ' entity='mgr.x' 2024-04-14T14:09:58.067 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:09:57 smithi069 ceph-mon[24818]: from='mgr.24749 172.21.15.122:0/1892077634' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T14:09:58.068 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:09:57 smithi069 ceph-mon[29472]: pgmap v197: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:09:58.068 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:09:57 smithi069 ceph-mon[29472]: from='mgr.24749 172.21.15.122:0/1892077634' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T14:09:58.068 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:09:57 smithi069 ceph-mon[29472]: from='mgr.24749 172.21.15.122:0/1892077634' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-14T14:09:58.068 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:09:57 smithi069 ceph-mon[29472]: pgmap v198: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 948 B/s rd, 0 op/s 2024-04-14T14:09:58.068 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:09:57 smithi069 ceph-mon[29472]: pgmap v199: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-04-14T14:09:58.068 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:09:57 smithi069 ceph-mon[29472]: from='mgr.24749 ' entity='mgr.x' 2024-04-14T14:09:58.068 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:09:57 smithi069 ceph-mon[29472]: from='mgr.24749 172.21.15.122:0/1892077634' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T14:09:58.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:09:57 smithi122 ceph-mon[25721]: pgmap v197: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:09:58.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:09:57 smithi122 ceph-mon[25721]: from='mgr.24749 172.21.15.122:0/1892077634' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T14:09:58.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:09:57 smithi122 ceph-mon[25721]: from='mgr.24749 172.21.15.122:0/1892077634' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-14T14:09:58.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:09:57 smithi122 ceph-mon[25721]: pgmap v198: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 948 B/s rd, 0 op/s 2024-04-14T14:09:58.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:09:57 smithi122 ceph-mon[25721]: pgmap v199: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-04-14T14:09:58.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:09:57 smithi122 ceph-mon[25721]: from='mgr.24749 ' entity='mgr.x' 2024-04-14T14:09:58.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:09:57 smithi122 ceph-mon[25721]: from='mgr.24749 172.21.15.122:0/1892077634' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T14:09:59.317 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:09:58 smithi069 ceph-mon[24818]: pgmap v200: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-04-14T14:09:59.317 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:09:58 smithi069 ceph-mon[29472]: pgmap v200: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-04-14T14:09:59.345 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:09:58 smithi122 ceph-mon[25721]: pgmap v200: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-04-14T14:10:00.317 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:10:00 smithi069 ceph-mon[24818]: Health detail: HEALTH_WARN 1 failed cephadm daemon(s) 2024-04-14T14:10:00.317 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:10:00 smithi069 ceph-mon[24818]: [WRN] CEPHADM_FAILED_DAEMON: 1 failed cephadm daemon(s) 2024-04-14T14:10:00.317 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:10:00 smithi069 ceph-mon[24818]: daemon grafana.a on smithi122 is in error state 2024-04-14T14:10:00.317 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:10:00 smithi069 ceph-mon[29472]: Health detail: HEALTH_WARN 1 failed cephadm daemon(s) 2024-04-14T14:10:00.317 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:10:00 smithi069 ceph-mon[29472]: [WRN] CEPHADM_FAILED_DAEMON: 1 failed cephadm daemon(s) 2024-04-14T14:10:00.318 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:10:00 smithi069 ceph-mon[29472]: daemon grafana.a on smithi122 is in error state 2024-04-14T14:10:00.345 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:10:00 smithi122 ceph-mon[25721]: Health detail: HEALTH_WARN 1 failed cephadm daemon(s) 2024-04-14T14:10:00.345 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:10:00 smithi122 ceph-mon[25721]: [WRN] CEPHADM_FAILED_DAEMON: 1 failed cephadm daemon(s) 2024-04-14T14:10:00.345 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:10:00 smithi122 ceph-mon[25721]: daemon grafana.a on smithi122 is in error state 2024-04-14T14:10:01.817 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:10:01 smithi069 ceph-mon[24818]: pgmap v201: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-04-14T14:10:01.817 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:10:01 smithi069 ceph-mon[29472]: pgmap v201: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-04-14T14:10:01.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:10:01 smithi122 ceph-mon[25721]: pgmap v201: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-04-14T14:10:03.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:10:02 smithi122 ceph-mon[25721]: pgmap v202: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 581 B/s rd, 0 op/s 2024-04-14T14:10:03.095 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Apr 14 14:10:02 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-x[45487]: ::ffff:172.21.15.122 - - [14/Apr/2024:14:10:02] "GET /metrics HTTP/1.1" 200 37964 "" "Prometheus/2.33.4" 2024-04-14T14:10:03.317 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:10:02 smithi069 ceph-mon[29472]: pgmap v202: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 581 B/s rd, 0 op/s 2024-04-14T14:10:03.317 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:10:02 smithi069 ceph-mon[24818]: pgmap v202: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 581 B/s rd, 0 op/s 2024-04-14T14:10:05.317 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:10:04 smithi069 ceph-mon[29472]: pgmap v203: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-04-14T14:10:05.317 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:10:04 smithi069 ceph-mon[24818]: pgmap v203: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-04-14T14:10:05.345 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:10:04 smithi122 ceph-mon[25721]: pgmap v203: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-04-14T14:10:06.817 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:10:06 smithi069 ceph-mon[24818]: from='client.15177 -' entity='client.iscsi.foo.smithi069.yuumtr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T14:10:06.817 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:10:06 smithi069 ceph-mon[29472]: from='client.15177 -' entity='client.iscsi.foo.smithi069.yuumtr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T14:10:06.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:10:06 smithi122 ceph-mon[25721]: from='client.15177 -' entity='client.iscsi.foo.smithi069.yuumtr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T14:10:07.817 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:10:07 smithi069 ceph-mon[24818]: pgmap v204: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-14T14:10:07.817 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:10:07 smithi069 ceph-mon[29472]: pgmap v204: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-14T14:10:07.846 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:10:07 smithi122 ceph-mon[25721]: pgmap v204: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-14T14:10:08.858 DEBUG:teuthology.orchestra.run.smithi069:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid dc326fc2-fa65-11ee-bc8e-c7b262605968 -e sha1=a971fd5f5719038c70b817bafacd6d5400b1d888 -- bash -c 'ceph orch ps' 2024-04-14T14:10:09.175 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:10:08 smithi069 ceph-mon[24818]: pgmap v205: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:10:09.175 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:10:08 smithi069 ceph-mon[29472]: pgmap v205: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:10:09.345 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:10:08 smithi122 ceph-mon[25721]: pgmap v205: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:10:10.893 INFO:teuthology.orchestra.run.smithi069.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-14T14:10:10.894 INFO:teuthology.orchestra.run.smithi069.stdout:alertmanager.a smithi069 *:9093,9094 running (5m) 75s ago 13m 23.5M - 0.23.0 ba2b418f427c e300fc36e997 2024-04-14T14:10:10.894 INFO:teuthology.orchestra.run.smithi069.stdout:grafana.a smithi122 *:3000 error 2m ago 13m - - 2024-04-14T14:10:10.894 INFO:teuthology.orchestra.run.smithi069.stdout:iscsi.foo.smithi069.yuumtr smithi069 running (4m) 75s ago 12m 49.2M - 3.5 e1d6a67b021e ce79fedc7dfd 2024-04-14T14:10:10.894 INFO:teuthology.orchestra.run.smithi069.stdout:mgr.x smithi122 *:8443,9283 running (9m) 2m ago 18m 515M - 19.0.0-2778-ga971fd5f b96c509eef84 813744c2b615 2024-04-14T14:10:10.894 INFO:teuthology.orchestra.run.smithi069.stdout:mgr.y smithi069 *:8443,9283,8765 running (2m) 75s ago 19m 450M - 19.0.0-2778-ga971fd5f b96c509eef84 7af3d29d0e5f 2024-04-14T14:10:10.894 INFO:teuthology.orchestra.run.smithi069.stdout:mon.a smithi069 running (19m) 75s ago 19m 57.0M 2048M 17.2.0 e1d6a67b021e d3c553f91913 2024-04-14T14:10:10.894 INFO:teuthology.orchestra.run.smithi069.stdout:mon.b smithi122 running (18m) 2m ago 18m 50.6M 2048M 17.2.0 e1d6a67b021e 3b766bfce4be 2024-04-14T14:10:10.894 INFO:teuthology.orchestra.run.smithi069.stdout:mon.c smithi069 running (18m) 75s ago 18m 43.1M 2048M 17.2.0 e1d6a67b021e 23c078b9d1e5 2024-04-14T14:10:10.894 INFO:teuthology.orchestra.run.smithi069.stdout:node-exporter.a smithi069 *:9100 running (6m) 75s ago 14m 18.7M - 1.3.1 1dbe0e931976 ca074142ab73 2024-04-14T14:10:10.894 INFO:teuthology.orchestra.run.smithi069.stdout:node-exporter.b smithi122 *:9100 running (5m) 2m ago 14m 16.5M - 1.3.1 1dbe0e931976 996115353a0b 2024-04-14T14:10:10.894 INFO:teuthology.orchestra.run.smithi069.stdout:osd.0 smithi069 running (17m) 75s ago 17m 55.3M 2992M 17.2.0 e1d6a67b021e 46654654b6b6 2024-04-14T14:10:10.894 INFO:teuthology.orchestra.run.smithi069.stdout:osd.1 smithi069 running (17m) 75s ago 17m 52.6M 2992M 17.2.0 e1d6a67b021e cdba54a59609 2024-04-14T14:10:10.895 INFO:teuthology.orchestra.run.smithi069.stdout:osd.2 smithi069 running (17m) 75s ago 17m 49.9M 2992M 17.2.0 e1d6a67b021e bc37a1a9cba0 2024-04-14T14:10:10.895 INFO:teuthology.orchestra.run.smithi069.stdout:osd.3 smithi069 running (16m) 75s ago 16m 54.5M 2992M 17.2.0 e1d6a67b021e 0925ba3da801 2024-04-14T14:10:10.895 INFO:teuthology.orchestra.run.smithi069.stdout:osd.4 smithi122 running (16m) 2m ago 16m 53.4M 3248M 17.2.0 e1d6a67b021e c00975caf771 2024-04-14T14:10:10.895 INFO:teuthology.orchestra.run.smithi069.stdout:osd.5 smithi122 running (15m) 2m ago 15m 53.4M 3248M 17.2.0 e1d6a67b021e 5f52d217ceb8 2024-04-14T14:10:10.895 INFO:teuthology.orchestra.run.smithi069.stdout:osd.6 smithi122 running (15m) 2m ago 15m 48.7M 3248M 17.2.0 e1d6a67b021e a2acfec61475 2024-04-14T14:10:10.895 INFO:teuthology.orchestra.run.smithi069.stdout:osd.7 smithi122 running (15m) 2m ago 15m 54.3M 3248M 17.2.0 e1d6a67b021e 62411cd06801 2024-04-14T14:10:10.895 INFO:teuthology.orchestra.run.smithi069.stdout:prometheus.a smithi122 *:9095 running (5m) 2m ago 14m 71.3M - 2.33.4 514e6a882f6e 2bdf6dae6e1e 2024-04-14T14:10:10.895 INFO:teuthology.orchestra.run.smithi069.stdout:rgw.foo.smithi069.wesedz smithi069 *:8000 running (12m) 75s ago 12m 75.0M - 17.2.0 e1d6a67b021e 16de491440f3 2024-04-14T14:10:10.895 INFO:teuthology.orchestra.run.smithi069.stdout:rgw.foo.smithi122.jiobyv smithi122 *:8000 running (12m) 2m ago 12m 71.5M - 17.2.0 e1d6a67b021e 6d16de445dba 2024-04-14T14:10:11.567 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:10:11 smithi069 ceph-mon[24818]: from='client.15216 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T14:10:11.567 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:10:11 smithi069 ceph-mon[24818]: pgmap v206: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:10:11.567 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:10:11 smithi069 ceph-mon[29472]: from='client.15216 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T14:10:11.567 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:10:11 smithi069 ceph-mon[29472]: pgmap v206: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:10:11.624 DEBUG:teuthology.orchestra.run.smithi069:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid dc326fc2-fa65-11ee-bc8e-c7b262605968 -e sha1=a971fd5f5719038c70b817bafacd6d5400b1d888 -- bash -c 'ceph versions' 2024-04-14T14:10:11.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:10:11 smithi122 ceph-mon[25721]: from='client.15216 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T14:10:11.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:10:11 smithi122 ceph-mon[25721]: pgmap v206: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:10:12.948 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:10:12 smithi069 ceph-mon[24818]: from='mgr.24749 172.21.15.122:0/1892077634' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T14:10:12.948 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:10:12 smithi069 ceph-mon[29472]: from='mgr.24749 172.21.15.122:0/1892077634' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T14:10:13.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:10:12 smithi122 ceph-mon[25721]: from='mgr.24749 172.21.15.122:0/1892077634' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T14:10:13.095 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Apr 14 14:10:12 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-x[45487]: ::ffff:172.21.15.122 - - [14/Apr/2024:14:10:12] "GET /metrics HTTP/1.1" 200 37963 "" "Prometheus/2.33.4" 2024-04-14T14:10:13.748 INFO:teuthology.orchestra.run.smithi069.stdout:{ 2024-04-14T14:10:13.748 INFO:teuthology.orchestra.run.smithi069.stdout: "mon": { 2024-04-14T14:10:13.748 INFO:teuthology.orchestra.run.smithi069.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 3 2024-04-14T14:10:13.749 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-14T14:10:13.749 INFO:teuthology.orchestra.run.smithi069.stdout: "mgr": { 2024-04-14T14:10:13.749 INFO:teuthology.orchestra.run.smithi069.stdout: "ceph version 19.0.0-2778-ga971fd5f (a971fd5f5719038c70b817bafacd6d5400b1d888) squid (dev)": 2 2024-04-14T14:10:13.749 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-14T14:10:13.749 INFO:teuthology.orchestra.run.smithi069.stdout: "osd": { 2024-04-14T14:10:13.749 INFO:teuthology.orchestra.run.smithi069.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 8 2024-04-14T14:10:13.749 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-14T14:10:13.749 INFO:teuthology.orchestra.run.smithi069.stdout: "mds": {}, 2024-04-14T14:10:13.749 INFO:teuthology.orchestra.run.smithi069.stdout: "rgw": { 2024-04-14T14:10:13.749 INFO:teuthology.orchestra.run.smithi069.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-04-14T14:10:13.749 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-14T14:10:13.749 INFO:teuthology.orchestra.run.smithi069.stdout: "overall": { 2024-04-14T14:10:13.749 INFO:teuthology.orchestra.run.smithi069.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 13, 2024-04-14T14:10:13.750 INFO:teuthology.orchestra.run.smithi069.stdout: "ceph version 19.0.0-2778-ga971fd5f (a971fd5f5719038c70b817bafacd6d5400b1d888) squid (dev)": 2 2024-04-14T14:10:13.750 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-14T14:10:13.750 INFO:teuthology.orchestra.run.smithi069.stdout:} 2024-04-14T14:10:14.066 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:10:13 smithi069 ceph-mon[24818]: pgmap v207: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:10:14.067 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:10:13 smithi069 ceph-mon[29472]: pgmap v207: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:10:14.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:10:13 smithi122 ceph-mon[25721]: pgmap v207: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:10:14.422 DEBUG:teuthology.orchestra.run.smithi069:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid dc326fc2-fa65-11ee-bc8e-c7b262605968 -e sha1=a971fd5f5719038c70b817bafacd6d5400b1d888 -- bash -c 'ceph health detail' 2024-04-14T14:10:15.016 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:10:14 smithi069 ceph-mon[24818]: from='client.? 172.21.15.69:0/234772500' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T14:10:15.016 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:10:14 smithi069 ceph-mon[29472]: from='client.? 172.21.15.69:0/234772500' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T14:10:15.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:10:14 smithi122 ceph-mon[25721]: from='client.? 172.21.15.69:0/234772500' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T14:10:15.947 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:10:15 smithi069 ceph-mon[29472]: pgmap v208: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:10:15.948 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:10:15 smithi069 ceph-mon[24818]: pgmap v208: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:10:16.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:10:15 smithi122 ceph-mon[25721]: pgmap v208: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:10:16.469 INFO:teuthology.orchestra.run.smithi069.stdout:HEALTH_WARN 1 failed cephadm daemon(s) 2024-04-14T14:10:16.469 INFO:teuthology.orchestra.run.smithi069.stdout:[WRN] CEPHADM_FAILED_DAEMON: 1 failed cephadm daemon(s) 2024-04-14T14:10:16.469 INFO:teuthology.orchestra.run.smithi069.stdout: daemon grafana.a on smithi122 is in error state 2024-04-14T14:10:16.817 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:10:16 smithi069 ceph-mon[24818]: from='client.15177 -' entity='client.iscsi.foo.smithi069.yuumtr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T14:10:16.817 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:10:16 smithi069 ceph-mon[24818]: from='client.? 172.21.15.69:0/4195844421' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-14T14:10:16.818 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:10:16 smithi069 ceph-mon[29472]: from='client.15177 -' entity='client.iscsi.foo.smithi069.yuumtr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T14:10:16.818 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:10:16 smithi069 ceph-mon[29472]: from='client.? 172.21.15.69:0/4195844421' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-14T14:10:17.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:10:16 smithi122 ceph-mon[25721]: from='client.15177 -' entity='client.iscsi.foo.smithi069.yuumtr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T14:10:17.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:10:16 smithi122 ceph-mon[25721]: from='client.? 172.21.15.69:0/4195844421' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-14T14:10:17.260 DEBUG:teuthology.orchestra.run.smithi069:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid dc326fc2-fa65-11ee-bc8e-c7b262605968 -e sha1=a971fd5f5719038c70b817bafacd6d5400b1d888 -- bash -c 'ceph -s' 2024-04-14T14:10:17.817 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:10:17 smithi069 ceph-mon[24818]: pgmap v209: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:10:17.817 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:10:17 smithi069 ceph-mon[29472]: pgmap v209: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:10:18.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:10:17 smithi122 ceph-mon[25721]: pgmap v209: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:10:19.067 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:10:18 smithi069 ceph-mon[24818]: pgmap v210: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:10:19.067 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:10:18 smithi069 ceph-mon[29472]: pgmap v210: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:10:19.302 INFO:teuthology.orchestra.run.smithi069.stdout: cluster: 2024-04-14T14:10:19.302 INFO:teuthology.orchestra.run.smithi069.stdout: id: dc326fc2-fa65-11ee-bc8e-c7b262605968 2024-04-14T14:10:19.302 INFO:teuthology.orchestra.run.smithi069.stdout: health: HEALTH_WARN 2024-04-14T14:10:19.302 INFO:teuthology.orchestra.run.smithi069.stdout: 1 failed cephadm daemon(s) 2024-04-14T14:10:19.302 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-14T14:10:19.303 INFO:teuthology.orchestra.run.smithi069.stdout: services: 2024-04-14T14:10:19.303 INFO:teuthology.orchestra.run.smithi069.stdout: mon: 3 daemons, quorum a,c,b (age 18m) 2024-04-14T14:10:19.303 INFO:teuthology.orchestra.run.smithi069.stdout: mgr: x(active, since 6m), standbys: y 2024-04-14T14:10:19.303 INFO:teuthology.orchestra.run.smithi069.stdout: osd: 8 osds: 8 up (since 15m), 8 in (since 15m) 2024-04-14T14:10:19.303 INFO:teuthology.orchestra.run.smithi069.stdout: rgw: 2 daemons active (2 hosts, 1 zones) 2024-04-14T14:10:19.303 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-14T14:10:19.303 INFO:teuthology.orchestra.run.smithi069.stdout: data: 2024-04-14T14:10:19.303 INFO:teuthology.orchestra.run.smithi069.stdout: pools: 6 pools, 161 pgs 2024-04-14T14:10:19.303 INFO:teuthology.orchestra.run.smithi069.stdout: objects: 209 objects, 585 KiB 2024-04-14T14:10:19.303 INFO:teuthology.orchestra.run.smithi069.stdout: usage: 94 MiB used, 715 GiB / 715 GiB avail 2024-04-14T14:10:19.303 INFO:teuthology.orchestra.run.smithi069.stdout: pgs: 161 active+clean 2024-04-14T14:10:19.303 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-14T14:10:19.303 INFO:teuthology.orchestra.run.smithi069.stdout: io: 2024-04-14T14:10:19.303 INFO:teuthology.orchestra.run.smithi069.stdout: client: 1.2 KiB/s rd, 1 op/s rd, 0 op/s wr 2024-04-14T14:10:19.303 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-14T14:10:19.345 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:10:18 smithi122 ceph-mon[25721]: pgmap v210: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:10:19.887 DEBUG:teuthology.orchestra.run.smithi069:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid dc326fc2-fa65-11ee-bc8e-c7b262605968 -e sha1=a971fd5f5719038c70b817bafacd6d5400b1d888 -- bash -c 'ceph mgr fail' 2024-04-14T14:10:20.317 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:10:19 smithi069 ceph-mon[24818]: from='client.? 172.21.15.69:0/43871643' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch 2024-04-14T14:10:20.317 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:10:19 smithi069 ceph-mon[29472]: from='client.? 172.21.15.69:0/43871643' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch 2024-04-14T14:10:20.345 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:10:19 smithi122 ceph-mon[25721]: from='client.? 172.21.15.69:0/43871643' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch 2024-04-14T14:10:21.817 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:10:21 smithi069 ceph-mon[24818]: pgmap v211: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:10:21.817 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:10:21 smithi069 ceph-mon[29472]: pgmap v211: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:10:21.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:10:21 smithi122 ceph-mon[25721]: pgmap v211: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:10:22.730 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:10:22 smithi069 ceph-mon[24818]: from='client.? 172.21.15.69:0/95972623' entity='client.admin' cmd=[{"prefix": "mgr fail"}]: dispatch 2024-04-14T14:10:22.730 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:10:22 smithi069 ceph-mon[24818]: osdmap e102: 8 total, 8 up, 8 in 2024-04-14T14:10:22.730 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:10:22 smithi069 ceph-mon[24818]: Standby manager daemon x started 2024-04-14T14:10:22.730 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 14 14:10:22 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-y[63038]: [14/Apr/2024:14:10:22] ENGINE Bus STOPPING 2024-04-14T14:10:22.731 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 14 14:10:22 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-y[63038]: [14/Apr/2024:14:10:22] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-04-14T14:10:22.731 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 14 14:10:22 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-y[63038]: [14/Apr/2024:14:10:22] ENGINE Bus STOPPED 2024-04-14T14:10:22.731 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 14 14:10:22 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-y[63038]: [14/Apr/2024:14:10:22] ENGINE Bus STARTING 2024-04-14T14:10:22.731 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 14 14:10:22 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-y[63038]: [14/Apr/2024:14:10:22] ENGINE Serving on http://:::9283 2024-04-14T14:10:22.731 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 14 14:10:22 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-y[63038]: [14/Apr/2024:14:10:22] ENGINE Bus STARTED 2024-04-14T14:10:22.731 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:10:22 smithi069 ceph-mon[29472]: from='client.? 172.21.15.69:0/95972623' entity='client.admin' cmd=[{"prefix": "mgr fail"}]: dispatch 2024-04-14T14:10:22.731 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:10:22 smithi069 ceph-mon[29472]: osdmap e102: 8 total, 8 up, 8 in 2024-04-14T14:10:22.731 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:10:22 smithi069 ceph-mon[29472]: Standby manager daemon x started 2024-04-14T14:10:22.732 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:10:22 smithi122 ceph-mon[25721]: from='client.? 172.21.15.69:0/95972623' entity='client.admin' cmd=[{"prefix": "mgr fail"}]: dispatch 2024-04-14T14:10:22.732 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:10:22 smithi122 ceph-mon[25721]: osdmap e102: 8 total, 8 up, 8 in 2024-04-14T14:10:22.732 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:10:22 smithi122 ceph-mon[25721]: Standby manager daemon x started 2024-04-14T14:10:22.732 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Apr 14 14:10:22 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-x[45487]: 2024-04-14T14:10:22.488+0000 7f468b56a700 -1 mgr handle_mgr_map I was active but no longer am 2024-04-14T14:10:22.732 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Apr 14 14:10:22 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-x[45487]: ignoring --setuser ceph since I am not root 2024-04-14T14:10:22.732 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Apr 14 14:10:22 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-x[45487]: ignoring --setgroup ceph since I am not root 2024-04-14T14:10:22.732 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Apr 14 14:10:22 smithi122 ceph-mgr[45491]: -- 172.21.15.122:0/2222608874 <== mon.2 v2:172.21.15.122:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x558f59d3a5a0 con 0x558f59d0f000 2024-04-14T14:10:23.136 DEBUG:teuthology.orchestra.run.smithi069:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid dc326fc2-fa65-11ee-bc8e-c7b262605968 -e sha1=a971fd5f5719038c70b817bafacd6d5400b1d888 -- bash -c 'sleep 180' 2024-04-14T14:10:23.345 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Apr 14 14:10:23 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-x[45487]: 2024-04-14T14:10:23.086+0000 7f9611748200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-14T14:10:23.345 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Apr 14 14:10:23 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-x[45487]: 2024-04-14T14:10:23.186+0000 7f9611748200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-14T14:10:23.714 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:10:23 smithi069 ceph-mon[24818]: from='client.? 172.21.15.69:0/95972623' entity='client.admin' cmd='[{"prefix": "mgr fail"}]': finished 2024-04-14T14:10:23.714 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:10:23 smithi069 ceph-mon[24818]: mgrmap e27: y(active, starting, since 0.522389s), standbys: x 2024-04-14T14:10:23.715 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:10:23 smithi069 ceph-mon[24818]: from='mgr.25105 172.21.15.69:0/4213893117' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-14T14:10:23.715 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:10:23 smithi069 ceph-mon[24818]: from='mgr.25105 172.21.15.69:0/4213893117' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-14T14:10:23.715 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:10:23 smithi069 ceph-mon[24818]: from='mgr.25105 172.21.15.69:0/4213893117' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-14T14:10:23.715 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:10:23 smithi069 ceph-mon[24818]: from='mgr.25105 172.21.15.69:0/4213893117' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-04-14T14:10:23.715 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:10:23 smithi069 ceph-mon[24818]: from='mgr.25105 172.21.15.69:0/4213893117' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-04-14T14:10:23.715 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:10:23 smithi069 ceph-mon[24818]: from='mgr.25105 172.21.15.69:0/4213893117' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-14T14:10:23.715 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:10:23 smithi069 ceph-mon[24818]: from='mgr.25105 172.21.15.69:0/4213893117' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-14T14:10:23.715 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:10:23 smithi069 ceph-mon[24818]: from='mgr.25105 172.21.15.69:0/4213893117' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-14T14:10:23.715 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:10:23 smithi069 ceph-mon[24818]: from='mgr.25105 172.21.15.69:0/4213893117' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-14T14:10:23.715 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:10:23 smithi069 ceph-mon[24818]: from='mgr.25105 172.21.15.69:0/4213893117' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-14T14:10:23.715 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:10:23 smithi069 ceph-mon[24818]: from='mgr.25105 172.21.15.69:0/4213893117' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-14T14:10:23.715 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:10:23 smithi069 ceph-mon[24818]: from='mgr.25105 172.21.15.69:0/4213893117' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-14T14:10:23.715 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:10:23 smithi069 ceph-mon[24818]: from='mgr.25105 172.21.15.69:0/4213893117' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-14T14:10:23.715 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:10:23 smithi069 ceph-mon[24818]: from='mgr.25105 172.21.15.69:0/4213893117' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-04-14T14:10:23.716 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:10:23 smithi069 ceph-mon[24818]: from='mgr.25105 172.21.15.69:0/4213893117' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-04-14T14:10:23.716 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:10:23 smithi069 ceph-mon[24818]: from='mgr.25105 172.21.15.69:0/4213893117' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-04-14T14:10:23.716 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:10:23 smithi069 ceph-mon[24818]: Manager daemon y is now available 2024-04-14T14:10:23.716 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:10:23 smithi069 ceph-mon[24818]: from='mgr.25105 172.21.15.69:0/4213893117' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T14:10:23.716 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:10:23 smithi069 ceph-mon[24818]: from='mgr.25105 172.21.15.69:0/4213893117' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-04-14T14:10:23.716 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:10:23 smithi069 ceph-mon[24818]: from='mgr.25105 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-04-14T14:10:23.716 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:10:23 smithi069 ceph-mon[24818]: from='mgr.25105 172.21.15.69:0/4213893117' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-04-14T14:10:23.716 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:10:23 smithi069 ceph-mon[24818]: from='mgr.25105 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-04-14T14:10:23.716 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:10:23 smithi069 ceph-mon[29472]: from='client.? 172.21.15.69:0/95972623' entity='client.admin' cmd='[{"prefix": "mgr fail"}]': finished 2024-04-14T14:10:23.717 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:10:23 smithi069 ceph-mon[29472]: mgrmap e27: y(active, starting, since 0.522389s), standbys: x 2024-04-14T14:10:23.717 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:10:23 smithi069 ceph-mon[29472]: from='mgr.25105 172.21.15.69:0/4213893117' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-14T14:10:23.717 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:10:23 smithi069 ceph-mon[29472]: from='mgr.25105 172.21.15.69:0/4213893117' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-14T14:10:23.717 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:10:23 smithi069 ceph-mon[29472]: from='mgr.25105 172.21.15.69:0/4213893117' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-14T14:10:23.717 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:10:23 smithi069 ceph-mon[29472]: from='mgr.25105 172.21.15.69:0/4213893117' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-04-14T14:10:23.717 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:10:23 smithi069 ceph-mon[29472]: from='mgr.25105 172.21.15.69:0/4213893117' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-04-14T14:10:23.717 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:10:23 smithi069 ceph-mon[29472]: from='mgr.25105 172.21.15.69:0/4213893117' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-14T14:10:23.717 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:10:23 smithi069 ceph-mon[29472]: from='mgr.25105 172.21.15.69:0/4213893117' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-14T14:10:23.717 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:10:23 smithi069 ceph-mon[29472]: from='mgr.25105 172.21.15.69:0/4213893117' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-14T14:10:23.717 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:10:23 smithi069 ceph-mon[29472]: from='mgr.25105 172.21.15.69:0/4213893117' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-14T14:10:23.717 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:10:23 smithi069 ceph-mon[29472]: from='mgr.25105 172.21.15.69:0/4213893117' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-14T14:10:23.717 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:10:23 smithi069 ceph-mon[29472]: from='mgr.25105 172.21.15.69:0/4213893117' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-14T14:10:23.717 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:10:23 smithi069 ceph-mon[29472]: from='mgr.25105 172.21.15.69:0/4213893117' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-14T14:10:23.718 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:10:23 smithi069 ceph-mon[29472]: from='mgr.25105 172.21.15.69:0/4213893117' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-14T14:10:23.718 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:10:23 smithi069 ceph-mon[29472]: from='mgr.25105 172.21.15.69:0/4213893117' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-04-14T14:10:23.718 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:10:23 smithi069 ceph-mon[29472]: from='mgr.25105 172.21.15.69:0/4213893117' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-04-14T14:10:23.718 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:10:23 smithi069 ceph-mon[29472]: from='mgr.25105 172.21.15.69:0/4213893117' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-04-14T14:10:23.718 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:10:23 smithi069 ceph-mon[29472]: Manager daemon y is now available 2024-04-14T14:10:23.718 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:10:23 smithi069 ceph-mon[29472]: from='mgr.25105 172.21.15.69:0/4213893117' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T14:10:23.718 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:10:23 smithi069 ceph-mon[29472]: from='mgr.25105 172.21.15.69:0/4213893117' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-04-14T14:10:23.718 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:10:23 smithi069 ceph-mon[29472]: from='mgr.25105 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-04-14T14:10:23.718 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:10:23 smithi069 ceph-mon[29472]: from='mgr.25105 172.21.15.69:0/4213893117' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-04-14T14:10:23.718 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:10:23 smithi069 ceph-mon[29472]: from='mgr.25105 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-04-14T14:10:23.810 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:10:23 smithi122 ceph-mon[25721]: from='client.? 172.21.15.69:0/95972623' entity='client.admin' cmd='[{"prefix": "mgr fail"}]': finished 2024-04-14T14:10:23.810 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:10:23 smithi122 ceph-mon[25721]: mgrmap e27: y(active, starting, since 0.522389s), standbys: x 2024-04-14T14:10:23.810 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:10:23 smithi122 ceph-mon[25721]: from='mgr.25105 172.21.15.69:0/4213893117' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-14T14:10:23.810 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:10:23 smithi122 ceph-mon[25721]: from='mgr.25105 172.21.15.69:0/4213893117' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-14T14:10:23.811 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:10:23 smithi122 ceph-mon[25721]: from='mgr.25105 172.21.15.69:0/4213893117' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-14T14:10:23.811 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:10:23 smithi122 ceph-mon[25721]: from='mgr.25105 172.21.15.69:0/4213893117' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-04-14T14:10:23.811 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:10:23 smithi122 ceph-mon[25721]: from='mgr.25105 172.21.15.69:0/4213893117' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-04-14T14:10:23.811 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:10:23 smithi122 ceph-mon[25721]: from='mgr.25105 172.21.15.69:0/4213893117' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-14T14:10:23.811 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:10:23 smithi122 ceph-mon[25721]: from='mgr.25105 172.21.15.69:0/4213893117' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-14T14:10:23.811 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:10:23 smithi122 ceph-mon[25721]: from='mgr.25105 172.21.15.69:0/4213893117' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-14T14:10:23.811 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:10:23 smithi122 ceph-mon[25721]: from='mgr.25105 172.21.15.69:0/4213893117' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-14T14:10:23.811 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:10:23 smithi122 ceph-mon[25721]: from='mgr.25105 172.21.15.69:0/4213893117' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-14T14:10:23.811 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:10:23 smithi122 ceph-mon[25721]: from='mgr.25105 172.21.15.69:0/4213893117' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-14T14:10:23.812 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:10:23 smithi122 ceph-mon[25721]: from='mgr.25105 172.21.15.69:0/4213893117' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-14T14:10:23.812 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:10:23 smithi122 ceph-mon[25721]: from='mgr.25105 172.21.15.69:0/4213893117' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-14T14:10:23.812 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:10:23 smithi122 ceph-mon[25721]: from='mgr.25105 172.21.15.69:0/4213893117' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-04-14T14:10:23.812 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:10:23 smithi122 ceph-mon[25721]: from='mgr.25105 172.21.15.69:0/4213893117' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-04-14T14:10:23.812 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:10:23 smithi122 ceph-mon[25721]: from='mgr.25105 172.21.15.69:0/4213893117' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-04-14T14:10:23.812 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:10:23 smithi122 ceph-mon[25721]: Manager daemon y is now available 2024-04-14T14:10:23.812 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:10:23 smithi122 ceph-mon[25721]: from='mgr.25105 172.21.15.69:0/4213893117' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T14:10:23.812 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:10:23 smithi122 ceph-mon[25721]: from='mgr.25105 172.21.15.69:0/4213893117' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-04-14T14:10:23.812 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:10:23 smithi122 ceph-mon[25721]: from='mgr.25105 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-04-14T14:10:23.813 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:10:23 smithi122 ceph-mon[25721]: from='mgr.25105 172.21.15.69:0/4213893117' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-04-14T14:10:23.813 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:10:23 smithi122 ceph-mon[25721]: from='mgr.25105 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-04-14T14:10:24.095 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Apr 14 14:10:23 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-x[45487]: 2024-04-14T14:10:23.890+0000 7f9611748200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-14T14:10:24.095 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Apr 14 14:10:23 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-x[45487]: 2024-04-14T14:10:23.997+0000 7f9611748200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-14T14:10:24.430 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Apr 14 14:10:24 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-x[45487]: 2024-04-14T14:10:24.143+0000 7f9611748200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-14T14:10:24.430 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Apr 14 14:10:24 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-x[45487]: 2024-04-14T14:10:24.250+0000 7f9611748200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-14T14:10:24.430 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Apr 14 14:10:24 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-x[45487]: 2024-04-14T14:10:24.332+0000 7f9611748200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-14T14:10:24.476 INFO:journalctl@ceph.alertmanager.a.smithi069.stdout:Apr 14 14:10:24 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-alertmanager-a[58923]: level=warn ts=2024-04-14T14:10:24.254Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi122.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.122:8443: connect: connection refused" 2024-04-14T14:10:24.721 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Apr 14 14:10:24 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-x[45487]: 2024-04-14T14:10:24.609+0000 7f9611748200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-14T14:10:24.722 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:10:24 smithi122 ceph-mon[25721]: mgrmap e28: y(active, since 1.53064s), standbys: x 2024-04-14T14:10:24.722 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:10:24 smithi122 ceph-mon[25721]: [14/Apr/2024:14:10:23] ENGINE Bus STARTING 2024-04-14T14:10:24.722 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:10:24 smithi122 ceph-mon[25721]: [14/Apr/2024:14:10:23] ENGINE Serving on http://172.21.15.69:8765 2024-04-14T14:10:24.757 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:10:24 smithi069 ceph-mon[24818]: mgrmap e28: y(active, since 1.53064s), standbys: x 2024-04-14T14:10:24.758 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:10:24 smithi069 ceph-mon[24818]: [14/Apr/2024:14:10:23] ENGINE Bus STARTING 2024-04-14T14:10:24.758 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:10:24 smithi069 ceph-mon[24818]: [14/Apr/2024:14:10:23] ENGINE Serving on http://172.21.15.69:8765 2024-04-14T14:10:24.758 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:10:24 smithi069 ceph-mon[29472]: mgrmap e28: y(active, since 1.53064s), standbys: x 2024-04-14T14:10:24.759 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:10:24 smithi069 ceph-mon[29472]: [14/Apr/2024:14:10:23] ENGINE Bus STARTING 2024-04-14T14:10:24.759 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:10:24 smithi069 ceph-mon[29472]: [14/Apr/2024:14:10:23] ENGINE Serving on http://172.21.15.69:8765 2024-04-14T14:10:25.095 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Apr 14 14:10:24 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-x[45487]: 2024-04-14T14:10:24.719+0000 7f9611748200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-14T14:10:25.095 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Apr 14 14:10:24 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-x[45487]: 2024-04-14T14:10:24.807+0000 7f9611748200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-14T14:10:25.095 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Apr 14 14:10:24 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-x[45487]: 2024-04-14T14:10:24.918+0000 7f9611748200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-14T14:10:25.378 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Apr 14 14:10:25 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-x[45487]: 2024-04-14T14:10:25.110+0000 7f9611748200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-14T14:10:25.817 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:10:25 smithi069 ceph-mon[29472]: [14/Apr/2024:14:10:23] ENGINE Serving on https://172.21.15.69:7150 2024-04-14T14:10:25.817 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:10:25 smithi069 ceph-mon[29472]: [14/Apr/2024:14:10:23] ENGINE Bus STARTED 2024-04-14T14:10:25.817 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:10:25 smithi069 ceph-mon[29472]: Health check cleared: CEPHADM_FAILED_DAEMON (was: 1 failed cephadm daemon(s)) 2024-04-14T14:10:25.817 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:10:25 smithi069 ceph-mon[29472]: Cluster is now healthy 2024-04-14T14:10:25.817 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:10:25 smithi069 ceph-mon[29472]: pgmap v4: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail 2024-04-14T14:10:25.817 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:10:25 smithi069 ceph-mon[29472]: mgrmap e29: y(active, since 2s), standbys: x 2024-04-14T14:10:25.817 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:10:25 smithi069 ceph-mon[24818]: [14/Apr/2024:14:10:23] ENGINE Serving on https://172.21.15.69:7150 2024-04-14T14:10:25.818 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:10:25 smithi069 ceph-mon[24818]: [14/Apr/2024:14:10:23] ENGINE Bus STARTED 2024-04-14T14:10:25.818 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:10:25 smithi069 ceph-mon[24818]: Health check cleared: CEPHADM_FAILED_DAEMON (was: 1 failed cephadm daemon(s)) 2024-04-14T14:10:25.818 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:10:25 smithi069 ceph-mon[24818]: Cluster is now healthy 2024-04-14T14:10:25.818 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:10:25 smithi069 ceph-mon[24818]: pgmap v4: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail 2024-04-14T14:10:25.818 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:10:25 smithi069 ceph-mon[24818]: mgrmap e29: y(active, since 2s), standbys: x 2024-04-14T14:10:25.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:10:25 smithi122 ceph-mon[25721]: [14/Apr/2024:14:10:23] ENGINE Serving on https://172.21.15.69:7150 2024-04-14T14:10:25.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:10:25 smithi122 ceph-mon[25721]: [14/Apr/2024:14:10:23] ENGINE Bus STARTED 2024-04-14T14:10:25.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:10:25 smithi122 ceph-mon[25721]: Health check cleared: CEPHADM_FAILED_DAEMON (was: 1 failed cephadm daemon(s)) 2024-04-14T14:10:25.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:10:25 smithi122 ceph-mon[25721]: Cluster is now healthy 2024-04-14T14:10:25.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:10:25 smithi122 ceph-mon[25721]: pgmap v4: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail 2024-04-14T14:10:25.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:10:25 smithi122 ceph-mon[25721]: mgrmap e29: y(active, since 2s), standbys: x 2024-04-14T14:10:26.254 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Apr 14 14:10:26 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-x[45487]: 2024-04-14T14:10:26.119+0000 7f9611748200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-14T14:10:26.254 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Apr 14 14:10:26 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-x[45487]: 2024-04-14T14:10:26.252+0000 7f9611748200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-14T14:10:26.918 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Apr 14 14:10:26 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-x[45487]: 2024-04-14T14:10:26.601+0000 7f9611748200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-14T14:10:26.919 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Apr 14 14:10:26 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-x[45487]: 2024-04-14T14:10:26.704+0000 7f9611748200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-14T14:10:26.919 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Apr 14 14:10:26 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-x[45487]: 2024-04-14T14:10:26.916+0000 7f9611748200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-14T14:10:27.345 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Apr 14 14:10:27 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-x[45487]: 2024-04-14T14:10:27.112+0000 7f9611748200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-14T14:10:27.687 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Apr 14 14:10:27 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-x[45487]: 2024-04-14T14:10:27.414+0000 7f9611748200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-14T14:10:27.687 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:10:27 smithi122 ceph-mon[25721]: from='client.15177 -' entity='client.iscsi.foo.smithi069.yuumtr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T14:10:27.688 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:10:27 smithi122 ceph-mon[25721]: pgmap v5: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail 2024-04-14T14:10:27.688 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:10:27 smithi122 ceph-mon[25721]: mgrmap e30: y(active, since 4s), standbys: x 2024-04-14T14:10:27.688 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:10:27 smithi122 ceph-mon[25721]: from='mgr.25105 ' entity='mgr.y' 2024-04-14T14:10:27.688 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:10:27 smithi122 ceph-mon[25721]: from='mgr.25105 ' entity='mgr.y' 2024-04-14T14:10:27.817 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:10:27 smithi069 ceph-mon[24818]: from='client.15177 -' entity='client.iscsi.foo.smithi069.yuumtr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T14:10:27.817 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:10:27 smithi069 ceph-mon[24818]: pgmap v5: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail 2024-04-14T14:10:27.817 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:10:27 smithi069 ceph-mon[24818]: mgrmap e30: y(active, since 4s), standbys: x 2024-04-14T14:10:27.817 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:10:27 smithi069 ceph-mon[24818]: from='mgr.25105 ' entity='mgr.y' 2024-04-14T14:10:27.817 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:10:27 smithi069 ceph-mon[24818]: from='mgr.25105 ' entity='mgr.y' 2024-04-14T14:10:27.817 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:10:27 smithi069 ceph-mon[29472]: from='client.15177 -' entity='client.iscsi.foo.smithi069.yuumtr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T14:10:27.817 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:10:27 smithi069 ceph-mon[29472]: pgmap v5: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail 2024-04-14T14:10:27.818 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:10:27 smithi069 ceph-mon[29472]: mgrmap e30: y(active, since 4s), standbys: x 2024-04-14T14:10:27.818 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:10:27 smithi069 ceph-mon[29472]: from='mgr.25105 ' entity='mgr.y' 2024-04-14T14:10:27.818 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:10:27 smithi069 ceph-mon[29472]: from='mgr.25105 ' entity='mgr.y' 2024-04-14T14:10:27.950 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Apr 14 14:10:27 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-x[45487]: 2024-04-14T14:10:27.685+0000 7f9611748200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-14T14:10:29.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:10:28 smithi122 ceph-mon[25721]: from='mgr.25105 ' entity='mgr.y' 2024-04-14T14:10:29.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:10:28 smithi122 ceph-mon[25721]: from='mgr.25105 ' entity='mgr.y' 2024-04-14T14:10:29.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:10:28 smithi122 ceph-mon[25721]: pgmap v6: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail 2024-04-14T14:10:29.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:10:28 smithi122 ceph-mon[25721]: from='mgr.25105 ' entity='mgr.y' 2024-04-14T14:10:29.096 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:10:28 smithi122 ceph-mon[25721]: from='mgr.25105 ' entity='mgr.y' 2024-04-14T14:10:29.096 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Apr 14 14:10:27 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-x[45487]: 2024-04-14T14:10:27.948+0000 7f9611748200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-14T14:10:29.096 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Apr 14 14:10:28 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-x[45487]: 2024-04-14T14:10:28.752+0000 7f9611748200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-14T14:10:29.317 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:10:28 smithi069 ceph-mon[29472]: from='mgr.25105 ' entity='mgr.y' 2024-04-14T14:10:29.317 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:10:28 smithi069 ceph-mon[29472]: from='mgr.25105 ' entity='mgr.y' 2024-04-14T14:10:29.317 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:10:28 smithi069 ceph-mon[29472]: pgmap v6: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail 2024-04-14T14:10:29.317 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:10:28 smithi069 ceph-mon[29472]: from='mgr.25105 ' entity='mgr.y' 2024-04-14T14:10:29.317 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:10:28 smithi069 ceph-mon[29472]: from='mgr.25105 ' entity='mgr.y' 2024-04-14T14:10:29.317 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:10:28 smithi069 ceph-mon[24818]: from='mgr.25105 ' entity='mgr.y' 2024-04-14T14:10:29.317 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:10:28 smithi069 ceph-mon[24818]: from='mgr.25105 ' entity='mgr.y' 2024-04-14T14:10:29.317 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:10:28 smithi069 ceph-mon[24818]: pgmap v6: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail 2024-04-14T14:10:29.317 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:10:28 smithi069 ceph-mon[24818]: from='mgr.25105 ' entity='mgr.y' 2024-04-14T14:10:29.318 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:10:28 smithi069 ceph-mon[24818]: from='mgr.25105 ' entity='mgr.y' 2024-04-14T14:10:29.594 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Apr 14 14:10:29 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-x[45487]: 2024-04-14T14:10:29.317+0000 7f9611748200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-14T14:10:29.595 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Apr 14 14:10:29 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-x[45487]: 2024-04-14T14:10:29.415+0000 7f9611748200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-14T14:10:30.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:10:29 smithi122 ceph-mon[25721]: Standby manager daemon x restarted 2024-04-14T14:10:30.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:10:29 smithi122 ceph-mon[25721]: Standby manager daemon x started 2024-04-14T14:10:30.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:10:29 smithi122 ceph-mon[25721]: from='mgr.? 172.21.15.122:0/87298362' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-04-14T14:10:30.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:10:29 smithi122 ceph-mon[25721]: from='mgr.? 172.21.15.122:0/87298362' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-14T14:10:30.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:10:29 smithi122 ceph-mon[25721]: from='mgr.? 172.21.15.122:0/87298362' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-04-14T14:10:30.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:10:29 smithi122 ceph-mon[25721]: from='mgr.? 172.21.15.122:0/87298362' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-14T14:10:30.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:10:29 smithi122 ceph-mon[25721]: from='mgr.25105 ' entity='mgr.y' 2024-04-14T14:10:30.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:10:29 smithi122 ceph-mon[25721]: from='mgr.25105 ' entity='mgr.y' 2024-04-14T14:10:30.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:10:29 smithi122 ceph-mon[25721]: from='mgr.25105 172.21.15.69:0/4213893117' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T14:10:30.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:10:29 smithi122 ceph-mon[25721]: from='mgr.25105 172.21.15.69:0/4213893117' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-14T14:10:30.096 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:10:29 smithi122 ceph-mon[25721]: Updating smithi069:/etc/ceph/ceph.conf 2024-04-14T14:10:30.096 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:10:29 smithi122 ceph-mon[25721]: Updating smithi122:/etc/ceph/ceph.conf 2024-04-14T14:10:30.096 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Apr 14 14:10:29 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-x[45487]: 2024-04-14T14:10:29.634+0000 7f9611748200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-14T14:10:30.096 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Apr 14 14:10:29 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-x[45487]: [14/Apr/2024:14:10:29] ENGINE Bus STARTING 2024-04-14T14:10:30.096 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Apr 14 14:10:29 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-x[45487]: CherryPy Checker: 2024-04-14T14:10:30.096 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Apr 14 14:10:29 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-x[45487]: The Application mounted at '' has an empty config. 2024-04-14T14:10:30.096 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Apr 14 14:10:29 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-x[45487]: 2024-04-14T14:10:30.096 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Apr 14 14:10:29 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-x[45487]: [14/Apr/2024:14:10:29] ENGINE Serving on http://:::9283 2024-04-14T14:10:30.096 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Apr 14 14:10:29 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-x[45487]: [14/Apr/2024:14:10:29] ENGINE Bus STARTED 2024-04-14T14:10:30.317 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:10:29 smithi069 ceph-mon[24818]: Standby manager daemon x restarted 2024-04-14T14:10:30.317 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:10:29 smithi069 ceph-mon[24818]: Standby manager daemon x started 2024-04-14T14:10:30.317 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:10:29 smithi069 ceph-mon[24818]: from='mgr.? 172.21.15.122:0/87298362' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-04-14T14:10:30.317 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:10:29 smithi069 ceph-mon[24818]: from='mgr.? 172.21.15.122:0/87298362' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-14T14:10:30.317 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:10:29 smithi069 ceph-mon[24818]: from='mgr.? 172.21.15.122:0/87298362' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-04-14T14:10:30.317 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:10:29 smithi069 ceph-mon[24818]: from='mgr.? 172.21.15.122:0/87298362' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-14T14:10:30.317 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:10:29 smithi069 ceph-mon[24818]: from='mgr.25105 ' entity='mgr.y' 2024-04-14T14:10:30.317 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:10:29 smithi069 ceph-mon[24818]: from='mgr.25105 ' entity='mgr.y' 2024-04-14T14:10:30.317 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:10:29 smithi069 ceph-mon[24818]: from='mgr.25105 172.21.15.69:0/4213893117' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T14:10:30.317 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:10:29 smithi069 ceph-mon[24818]: from='mgr.25105 172.21.15.69:0/4213893117' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-14T14:10:30.318 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:10:29 smithi069 ceph-mon[24818]: Updating smithi069:/etc/ceph/ceph.conf 2024-04-14T14:10:30.318 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:10:29 smithi069 ceph-mon[24818]: Updating smithi122:/etc/ceph/ceph.conf 2024-04-14T14:10:30.318 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:10:29 smithi069 ceph-mon[29472]: Standby manager daemon x restarted 2024-04-14T14:10:30.318 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:10:29 smithi069 ceph-mon[29472]: Standby manager daemon x started 2024-04-14T14:10:30.318 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:10:29 smithi069 ceph-mon[29472]: from='mgr.? 172.21.15.122:0/87298362' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-04-14T14:10:30.318 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:10:29 smithi069 ceph-mon[29472]: from='mgr.? 172.21.15.122:0/87298362' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-14T14:10:30.318 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:10:29 smithi069 ceph-mon[29472]: from='mgr.? 172.21.15.122:0/87298362' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-04-14T14:10:30.318 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:10:29 smithi069 ceph-mon[29472]: from='mgr.? 172.21.15.122:0/87298362' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-14T14:10:30.318 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:10:29 smithi069 ceph-mon[29472]: from='mgr.25105 ' entity='mgr.y' 2024-04-14T14:10:30.318 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:10:29 smithi069 ceph-mon[29472]: from='mgr.25105 ' entity='mgr.y' 2024-04-14T14:10:30.318 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:10:29 smithi069 ceph-mon[29472]: from='mgr.25105 172.21.15.69:0/4213893117' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T14:10:30.319 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:10:29 smithi069 ceph-mon[29472]: from='mgr.25105 172.21.15.69:0/4213893117' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-14T14:10:30.319 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:10:29 smithi069 ceph-mon[29472]: Updating smithi069:/etc/ceph/ceph.conf 2024-04-14T14:10:30.319 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:10:29 smithi069 ceph-mon[29472]: Updating smithi122:/etc/ceph/ceph.conf 2024-04-14T14:10:31.317 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:10:30 smithi069 ceph-mon[24818]: mgrmap e31: y(active, since 7s), standbys: x 2024-04-14T14:10:31.317 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:10:30 smithi069 ceph-mon[24818]: Updating smithi122:/var/lib/ceph/dc326fc2-fa65-11ee-bc8e-c7b262605968/config/ceph.conf 2024-04-14T14:10:31.317 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:10:30 smithi069 ceph-mon[24818]: Updating smithi069:/var/lib/ceph/dc326fc2-fa65-11ee-bc8e-c7b262605968/config/ceph.conf 2024-04-14T14:10:31.317 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:10:30 smithi069 ceph-mon[24818]: Updating smithi122:/etc/ceph/ceph.client.admin.keyring 2024-04-14T14:10:31.317 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:10:30 smithi069 ceph-mon[24818]: Updating smithi069:/etc/ceph/ceph.client.admin.keyring 2024-04-14T14:10:31.317 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:10:30 smithi069 ceph-mon[24818]: pgmap v7: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 30 KiB/s rd, 0 B/s wr, 17 op/s 2024-04-14T14:10:31.317 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:10:30 smithi069 ceph-mon[24818]: Updating smithi069:/var/lib/ceph/dc326fc2-fa65-11ee-bc8e-c7b262605968/config/ceph.client.admin.keyring 2024-04-14T14:10:31.318 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:10:30 smithi069 ceph-mon[24818]: Updating smithi122:/var/lib/ceph/dc326fc2-fa65-11ee-bc8e-c7b262605968/config/ceph.client.admin.keyring 2024-04-14T14:10:31.318 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:10:30 smithi069 ceph-mon[24818]: from='mgr.25105 ' entity='mgr.y' 2024-04-14T14:10:31.318 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:10:30 smithi069 ceph-mon[24818]: from='mgr.25105 ' entity='mgr.y' 2024-04-14T14:10:31.318 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:10:30 smithi069 ceph-mon[24818]: from='mgr.25105 ' entity='mgr.y' 2024-04-14T14:10:31.318 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:10:30 smithi069 ceph-mon[24818]: from='mgr.25105 ' entity='mgr.y' 2024-04-14T14:10:31.318 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:10:30 smithi069 ceph-mon[24818]: pgmap v8: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 29 KiB/s rd, 0 B/s wr, 16 op/s 2024-04-14T14:10:31.318 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:10:30 smithi069 ceph-mon[24818]: from='mgr.25105 ' entity='mgr.y' 2024-04-14T14:10:31.319 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:10:30 smithi069 ceph-mon[29472]: mgrmap e31: y(active, since 7s), standbys: x 2024-04-14T14:10:31.319 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:10:30 smithi069 ceph-mon[29472]: Updating smithi122:/var/lib/ceph/dc326fc2-fa65-11ee-bc8e-c7b262605968/config/ceph.conf 2024-04-14T14:10:31.319 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:10:30 smithi069 ceph-mon[29472]: Updating smithi069:/var/lib/ceph/dc326fc2-fa65-11ee-bc8e-c7b262605968/config/ceph.conf 2024-04-14T14:10:31.319 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:10:30 smithi069 ceph-mon[29472]: Updating smithi122:/etc/ceph/ceph.client.admin.keyring 2024-04-14T14:10:31.319 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:10:30 smithi069 ceph-mon[29472]: Updating smithi069:/etc/ceph/ceph.client.admin.keyring 2024-04-14T14:10:31.319 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:10:30 smithi069 ceph-mon[29472]: pgmap v7: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 30 KiB/s rd, 0 B/s wr, 17 op/s 2024-04-14T14:10:31.319 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:10:30 smithi069 ceph-mon[29472]: Updating smithi069:/var/lib/ceph/dc326fc2-fa65-11ee-bc8e-c7b262605968/config/ceph.client.admin.keyring 2024-04-14T14:10:31.319 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:10:30 smithi069 ceph-mon[29472]: Updating smithi122:/var/lib/ceph/dc326fc2-fa65-11ee-bc8e-c7b262605968/config/ceph.client.admin.keyring 2024-04-14T14:10:31.320 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:10:30 smithi069 ceph-mon[29472]: from='mgr.25105 ' entity='mgr.y' 2024-04-14T14:10:31.320 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:10:30 smithi069 ceph-mon[29472]: from='mgr.25105 ' entity='mgr.y' 2024-04-14T14:10:31.320 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:10:30 smithi069 ceph-mon[29472]: from='mgr.25105 ' entity='mgr.y' 2024-04-14T14:10:31.320 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:10:30 smithi069 ceph-mon[29472]: from='mgr.25105 ' entity='mgr.y' 2024-04-14T14:10:31.320 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:10:30 smithi069 ceph-mon[29472]: pgmap v8: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 29 KiB/s rd, 0 B/s wr, 16 op/s 2024-04-14T14:10:31.320 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:10:30 smithi069 ceph-mon[29472]: from='mgr.25105 ' entity='mgr.y' 2024-04-14T14:10:31.345 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:10:30 smithi122 ceph-mon[25721]: mgrmap e31: y(active, since 7s), standbys: x 2024-04-14T14:10:31.345 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:10:30 smithi122 ceph-mon[25721]: Updating smithi122:/var/lib/ceph/dc326fc2-fa65-11ee-bc8e-c7b262605968/config/ceph.conf 2024-04-14T14:10:31.345 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:10:30 smithi122 ceph-mon[25721]: Updating smithi069:/var/lib/ceph/dc326fc2-fa65-11ee-bc8e-c7b262605968/config/ceph.conf 2024-04-14T14:10:31.345 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:10:30 smithi122 ceph-mon[25721]: Updating smithi122:/etc/ceph/ceph.client.admin.keyring 2024-04-14T14:10:31.345 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:10:30 smithi122 ceph-mon[25721]: Updating smithi069:/etc/ceph/ceph.client.admin.keyring 2024-04-14T14:10:31.345 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:10:30 smithi122 ceph-mon[25721]: pgmap v7: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 30 KiB/s rd, 0 B/s wr, 17 op/s 2024-04-14T14:10:31.345 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:10:30 smithi122 ceph-mon[25721]: Updating smithi069:/var/lib/ceph/dc326fc2-fa65-11ee-bc8e-c7b262605968/config/ceph.client.admin.keyring 2024-04-14T14:10:31.346 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:10:30 smithi122 ceph-mon[25721]: Updating smithi122:/var/lib/ceph/dc326fc2-fa65-11ee-bc8e-c7b262605968/config/ceph.client.admin.keyring 2024-04-14T14:10:31.346 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:10:30 smithi122 ceph-mon[25721]: from='mgr.25105 ' entity='mgr.y' 2024-04-14T14:10:31.346 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:10:30 smithi122 ceph-mon[25721]: from='mgr.25105 ' entity='mgr.y' 2024-04-14T14:10:31.346 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:10:30 smithi122 ceph-mon[25721]: from='mgr.25105 ' entity='mgr.y' 2024-04-14T14:10:31.346 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:10:30 smithi122 ceph-mon[25721]: from='mgr.25105 ' entity='mgr.y' 2024-04-14T14:10:31.346 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:10:30 smithi122 ceph-mon[25721]: pgmap v8: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 29 KiB/s rd, 0 B/s wr, 16 op/s 2024-04-14T14:10:31.346 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:10:30 smithi122 ceph-mon[25721]: from='mgr.25105 ' entity='mgr.y' 2024-04-14T14:10:31.700 INFO:journalctl@ceph.alertmanager.a.smithi069.stdout:Apr 14 14:10:31 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-alertmanager-a[58923]: level=info ts=2024-04-14T14:10:31.565Z caller=notify.go:735 component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify success" attempts=7 2024-04-14T14:10:32.067 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:10:31 smithi069 ceph-mon[24818]: Reconfiguring iscsi.foo.smithi069.yuumtr (dependencies changed)... 2024-04-14T14:10:32.067 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:10:31 smithi069 ceph-mon[24818]: from='mgr.25105 172.21.15.69:0/4213893117' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi069.yuumtr", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-04-14T14:10:32.067 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:10:31 smithi069 ceph-mon[24818]: from='mgr.25105 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi069.yuumtr", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-04-14T14:10:32.067 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:10:31 smithi069 ceph-mon[24818]: from='mgr.25105 172.21.15.69:0/4213893117' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T14:10:32.067 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:10:31 smithi069 ceph-mon[24818]: Reconfiguring daemon iscsi.foo.smithi069.yuumtr on smithi069 2024-04-14T14:10:32.067 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:10:31 smithi069 ceph-mon[24818]: Health check failed: 1 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON) 2024-04-14T14:10:32.067 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:10:31 smithi069 ceph-mon[29472]: Reconfiguring iscsi.foo.smithi069.yuumtr (dependencies changed)... 2024-04-14T14:10:32.067 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:10:31 smithi069 ceph-mon[29472]: from='mgr.25105 172.21.15.69:0/4213893117' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi069.yuumtr", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-04-14T14:10:32.067 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:10:31 smithi069 ceph-mon[29472]: from='mgr.25105 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi069.yuumtr", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-04-14T14:10:32.068 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:10:31 smithi069 ceph-mon[29472]: from='mgr.25105 172.21.15.69:0/4213893117' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T14:10:32.068 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:10:31 smithi069 ceph-mon[29472]: Reconfiguring daemon iscsi.foo.smithi069.yuumtr on smithi069 2024-04-14T14:10:32.068 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:10:31 smithi069 ceph-mon[29472]: Health check failed: 1 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON) 2024-04-14T14:10:32.345 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:10:31 smithi122 ceph-mon[25721]: Reconfiguring iscsi.foo.smithi069.yuumtr (dependencies changed)... 2024-04-14T14:10:32.345 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:10:31 smithi122 ceph-mon[25721]: from='mgr.25105 172.21.15.69:0/4213893117' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi069.yuumtr", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-04-14T14:10:32.345 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:10:31 smithi122 ceph-mon[25721]: from='mgr.25105 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi069.yuumtr", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-04-14T14:10:32.345 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:10:31 smithi122 ceph-mon[25721]: from='mgr.25105 172.21.15.69:0/4213893117' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T14:10:32.345 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:10:31 smithi122 ceph-mon[25721]: Reconfiguring daemon iscsi.foo.smithi069.yuumtr on smithi069 2024-04-14T14:10:32.345 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:10:31 smithi122 ceph-mon[25721]: Health check failed: 1 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON) 2024-04-14T14:10:33.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:10:32 smithi122 ceph-mon[25721]: pgmap v9: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 23 KiB/s rd, 0 B/s wr, 13 op/s 2024-04-14T14:10:33.095 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Apr 14 14:10:32 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-x[45487]: ::ffff:172.21.15.122 - - [14/Apr/2024:14:10:32] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-04-14T14:10:33.214 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:10:32 smithi069 ceph-mon[24818]: pgmap v9: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 23 KiB/s rd, 0 B/s wr, 13 op/s 2024-04-14T14:10:33.215 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:10:32 smithi069 ceph-mon[29472]: pgmap v9: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 23 KiB/s rd, 0 B/s wr, 13 op/s 2024-04-14T14:10:36.274 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:10:35 smithi122 ceph-mon[25721]: pgmap v10: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 21 KiB/s rd, 0 B/s wr, 12 op/s 2024-04-14T14:10:36.275 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:10:35 smithi122 ceph-mon[25721]: from='mgr.25105 ' entity='mgr.y' 2024-04-14T14:10:36.275 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:10:35 smithi122 ceph-mon[25721]: from='mgr.25105 ' entity='mgr.y' 2024-04-14T14:10:36.275 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:10:35 smithi122 ceph-mon[25721]: from='client.? 172.21.15.69:0/2224171421' entity='client.iscsi.foo.smithi069.yuumtr' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-04-14T14:10:36.317 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:10:35 smithi069 ceph-mon[24818]: pgmap v10: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 21 KiB/s rd, 0 B/s wr, 12 op/s 2024-04-14T14:10:36.317 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:10:35 smithi069 ceph-mon[24818]: from='mgr.25105 ' entity='mgr.y' 2024-04-14T14:10:36.317 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:10:35 smithi069 ceph-mon[24818]: from='mgr.25105 ' entity='mgr.y' 2024-04-14T14:10:36.317 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:10:35 smithi069 ceph-mon[24818]: from='client.? 172.21.15.69:0/2224171421' entity='client.iscsi.foo.smithi069.yuumtr' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-04-14T14:10:36.317 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:10:35 smithi069 ceph-mon[29472]: pgmap v10: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 21 KiB/s rd, 0 B/s wr, 12 op/s 2024-04-14T14:10:36.318 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:10:35 smithi069 ceph-mon[29472]: from='mgr.25105 ' entity='mgr.y' 2024-04-14T14:10:36.318 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:10:35 smithi069 ceph-mon[29472]: from='mgr.25105 ' entity='mgr.y' 2024-04-14T14:10:36.318 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:10:35 smithi069 ceph-mon[29472]: from='client.? 172.21.15.69:0/2224171421' entity='client.iscsi.foo.smithi069.yuumtr' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-04-14T14:10:37.100 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:10:36 smithi122 ceph-mon[25721]: Reconfiguring prometheus.a (dependencies changed)... 2024-04-14T14:10:37.100 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:10:36 smithi122 ceph-mon[25721]: Reconfiguring daemon prometheus.a on smithi122 2024-04-14T14:10:37.100 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:10:36 smithi122 ceph-mon[25721]: pgmap v11: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 21 KiB/s rd, 0 B/s wr, 12 op/s 2024-04-14T14:10:37.100 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Apr 14 14:10:36 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-prometheus-a[49277]: ts=2024-04-14T14:10:36.654Z caller=refresh.go:98 level=error component="discovery manager scrape" discovery=http msg="Unable to refresh target groups" err="Get \"http://172.21.15.122:8765/sd/prometheus/sd-config?service=nvmeof\": dial tcp 172.21.15.122:8765: connect: connection refused" 2024-04-14T14:10:37.101 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Apr 14 14:10:36 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-prometheus-a[49277]: ts=2024-04-14T14:10:36.656Z caller=refresh.go:98 level=error component="discovery manager scrape" discovery=http msg="Unable to refresh target groups" err="Get \"http://172.21.15.122:8765/sd/prometheus/sd-config?service=node-exporter\": dial tcp 172.21.15.122:8765: connect: connection refused" 2024-04-14T14:10:37.101 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Apr 14 14:10:36 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-prometheus-a[49277]: ts=2024-04-14T14:10:36.658Z caller=refresh.go:98 level=error component="discovery manager scrape" discovery=http msg="Unable to refresh target groups" err="Get \"http://172.21.15.122:8765/sd/prometheus/sd-config?service=ceph-exporter\": dial tcp 172.21.15.122:8765: connect: connection refused" 2024-04-14T14:10:37.101 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Apr 14 14:10:36 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-prometheus-a[49277]: ts=2024-04-14T14:10:36.658Z caller=refresh.go:98 level=error component="discovery manager notify" discovery=http msg="Unable to refresh target groups" err="Get \"http://172.21.15.122:8765/sd/prometheus/sd-config?service=alertmanager\": dial tcp 172.21.15.122:8765: connect: connection refused" 2024-04-14T14:10:37.101 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Apr 14 14:10:36 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-prometheus-a[49277]: ts=2024-04-14T14:10:36.659Z caller=refresh.go:98 level=error component="discovery manager scrape" discovery=http msg="Unable to refresh target groups" err="Get \"http://172.21.15.122:8765/sd/prometheus/sd-config?service=mgr-prometheus\": dial tcp 172.21.15.122:8765: connect: connection refused" 2024-04-14T14:10:37.317 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:10:36 smithi069 ceph-mon[24818]: Reconfiguring prometheus.a (dependencies changed)... 2024-04-14T14:10:37.317 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:10:36 smithi069 ceph-mon[24818]: Reconfiguring daemon prometheus.a on smithi122 2024-04-14T14:10:37.317 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:10:36 smithi069 ceph-mon[24818]: pgmap v11: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 21 KiB/s rd, 0 B/s wr, 12 op/s 2024-04-14T14:10:37.317 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:10:36 smithi069 ceph-mon[29472]: Reconfiguring prometheus.a (dependencies changed)... 2024-04-14T14:10:37.317 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:10:36 smithi069 ceph-mon[29472]: Reconfiguring daemon prometheus.a on smithi122 2024-04-14T14:10:37.317 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:10:36 smithi069 ceph-mon[29472]: pgmap v11: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 21 KiB/s rd, 0 B/s wr, 12 op/s 2024-04-14T14:10:38.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:10:37 smithi122 ceph-mon[25721]: from='mgr.25105 172.21.15.69:0/4213893117' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T14:10:38.317 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:10:37 smithi069 ceph-mon[24818]: from='mgr.25105 172.21.15.69:0/4213893117' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T14:10:38.317 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:10:37 smithi069 ceph-mon[29472]: from='mgr.25105 172.21.15.69:0/4213893117' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T14:10:38.714 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Apr 14 14:10:38 smithi122 systemd[1]: Stopping Ceph prometheus.a for dc326fc2-fa65-11ee-bc8e-c7b262605968... 2024-04-14T14:10:38.714 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Apr 14 14:10:38 smithi122 bash[54931]: Error: no container with name or ID "ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-prometheus.a" found: no such container 2024-04-14T14:10:39.017 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:10:38 smithi122 ceph-mon[25721]: pgmap v12: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 21 KiB/s rd, 0 B/s wr, 12 op/s 2024-04-14T14:10:39.017 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Apr 14 14:10:38 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-prometheus-a[49277]: ts=2024-04-14T14:10:38.713Z caller=main.go:775 level=warn msg="Received SIGTERM, exiting gracefully..." 2024-04-14T14:10:39.017 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Apr 14 14:10:38 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-prometheus-a[49277]: ts=2024-04-14T14:10:38.713Z caller=main.go:798 level=info msg="Stopping scrape discovery manager..." 2024-04-14T14:10:39.017 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Apr 14 14:10:38 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-prometheus-a[49277]: ts=2024-04-14T14:10:38.713Z caller=main.go:812 level=info msg="Stopping notify discovery manager..." 2024-04-14T14:10:39.017 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Apr 14 14:10:38 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-prometheus-a[49277]: ts=2024-04-14T14:10:38.713Z caller=main.go:834 level=info msg="Stopping scrape manager..." 2024-04-14T14:10:39.018 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Apr 14 14:10:38 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-prometheus-a[49277]: ts=2024-04-14T14:10:38.713Z caller=main.go:794 level=info msg="Scrape discovery manager stopped" 2024-04-14T14:10:39.018 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Apr 14 14:10:38 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-prometheus-a[49277]: ts=2024-04-14T14:10:38.713Z caller=main.go:808 level=info msg="Notify discovery manager stopped" 2024-04-14T14:10:39.018 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Apr 14 14:10:38 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-prometheus-a[49277]: ts=2024-04-14T14:10:38.713Z caller=manager.go:945 level=info component="rule manager" msg="Stopping rule manager..." 2024-04-14T14:10:39.018 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Apr 14 14:10:38 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-prometheus-a[49277]: ts=2024-04-14T14:10:38.713Z caller=manager.go:955 level=info component="rule manager" msg="Rule manager stopped" 2024-04-14T14:10:39.018 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Apr 14 14:10:38 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-prometheus-a[49277]: ts=2024-04-14T14:10:38.713Z caller=main.go:828 level=info msg="Scrape manager stopped" 2024-04-14T14:10:39.018 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Apr 14 14:10:38 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-prometheus-a[49277]: ts=2024-04-14T14:10:38.713Z caller=notifier.go:600 level=info component=notifier msg="Stopping notification manager..." 2024-04-14T14:10:39.018 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Apr 14 14:10:38 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-prometheus-a[49277]: ts=2024-04-14T14:10:38.713Z caller=main.go:1054 level=info msg="Notifier manager stopped" 2024-04-14T14:10:39.018 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Apr 14 14:10:38 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-prometheus-a[49277]: ts=2024-04-14T14:10:38.713Z caller=main.go:1066 level=info msg="See you next time!" 2024-04-14T14:10:39.018 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Apr 14 14:10:38 smithi122 podman[54944]: 2024-04-14 14:10:38.723415146 +0000 UTC m=+0.115051769 container died 2bdf6dae6e1e624add99fc15caba8a64ab8495e8227a255fdce337ca01270932 (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-prometheus-a, maintainer=The Prometheus Authors ) 2024-04-14T14:10:39.317 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:10:38 smithi069 ceph-mon[24818]: pgmap v12: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 21 KiB/s rd, 0 B/s wr, 12 op/s 2024-04-14T14:10:39.317 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:10:38 smithi069 ceph-mon[29472]: pgmap v12: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 21 KiB/s rd, 0 B/s wr, 12 op/s 2024-04-14T14:10:39.345 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Apr 14 14:10:39 smithi122 podman[54944]: 2024-04-14 14:10:39.104882774 +0000 UTC m=+0.496519393 container cleanup 2bdf6dae6e1e624add99fc15caba8a64ab8495e8227a255fdce337ca01270932 (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-prometheus-a, maintainer=The Prometheus Authors ) 2024-04-14T14:10:39.346 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Apr 14 14:10:39 smithi122 bash[54944]: ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-prometheus-a 2024-04-14T14:10:39.346 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Apr 14 14:10:39 smithi122 bash[54970]: Error: no container with name or ID "ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-prometheus.a" found: no such container 2024-04-14T14:10:39.633 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Apr 14 14:10:39 smithi122 podman[54958]: 2024-04-14 14:10:39.364290742 +0000 UTC m=+0.642807765 container remove 2bdf6dae6e1e624add99fc15caba8a64ab8495e8227a255fdce337ca01270932 (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-prometheus-a, maintainer=The Prometheus Authors ) 2024-04-14T14:10:39.633 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Apr 14 14:10:39 smithi122 systemd[1]: ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968@prometheus.a.service: Deactivated successfully. 2024-04-14T14:10:39.633 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Apr 14 14:10:39 smithi122 systemd[1]: Stopped Ceph prometheus.a for dc326fc2-fa65-11ee-bc8e-c7b262605968. 2024-04-14T14:10:39.633 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Apr 14 14:10:39 smithi122 systemd[1]: ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968@prometheus.a.service: Consumed 2.030s CPU time. 2024-04-14T14:10:39.633 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Apr 14 14:10:39 smithi122 systemd[1]: Starting Ceph prometheus.a for dc326fc2-fa65-11ee-bc8e-c7b262605968... 2024-04-14T14:10:39.633 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Apr 14 14:10:39 smithi122 podman[55046]: 2024-04-14 14:10:39.510345836 +0000 UTC m=+0.018832678 image pull 514e6a882f6e74806a5856468489eeff8d7106095557578da96935e4d0ba4d9d quay.io/prometheus/prometheus:v2.33.4 2024-04-14T14:10:39.633 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Apr 14 14:10:39 smithi122 podman[55046]: 2024-04-14 14:10:39.632014419 +0000 UTC m=+0.140501261 container create 9ebe35247d1a39f019e49a526e82fee7cb72172eb0477901dbef3a479dc4f20c (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-prometheus-a, maintainer=The Prometheus Authors ) 2024-04-14T14:10:40.345 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Apr 14 14:10:39 smithi122 podman[55046]: 2024-04-14 14:10:39.884604779 +0000 UTC m=+0.393091621 container init 9ebe35247d1a39f019e49a526e82fee7cb72172eb0477901dbef3a479dc4f20c (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-prometheus-a, maintainer=The Prometheus Authors ) 2024-04-14T14:10:40.345 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Apr 14 14:10:39 smithi122 podman[55046]: 2024-04-14 14:10:39.887295655 +0000 UTC m=+0.395782508 container start 9ebe35247d1a39f019e49a526e82fee7cb72172eb0477901dbef3a479dc4f20c (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-prometheus-a, maintainer=The Prometheus Authors ) 2024-04-14T14:10:40.345 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Apr 14 14:10:39 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-prometheus-a[55061]: ts=2024-04-14T14:10:39.910Z caller=main.go:475 level=info msg="No time or size retention was set so using the default time retention" duration=15d 2024-04-14T14:10:40.346 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Apr 14 14:10:39 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-prometheus-a[55061]: ts=2024-04-14T14:10:39.910Z caller=main.go:512 level=info msg="Starting Prometheus" version="(version=2.33.4, branch=HEAD, revision=83032011a5d3e6102624fe58241a374a7201fee8)" 2024-04-14T14:10:40.346 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Apr 14 14:10:39 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-prometheus-a[55061]: ts=2024-04-14T14:10:39.910Z caller=main.go:517 level=info build_context="(go=go1.17.7, user=root@d13bf69e7be8, date=20220222-16:51:28)" 2024-04-14T14:10:40.346 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Apr 14 14:10:39 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-prometheus-a[55061]: ts=2024-04-14T14:10:39.910Z caller=main.go:518 level=info host_details="(Linux 5.14.0-435.el9.x86_64 #1 SMP PREEMPT_DYNAMIC Fri Mar 29 18:46:37 UTC 2024 x86_64 smithi122 (none))" 2024-04-14T14:10:40.346 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Apr 14 14:10:39 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-prometheus-a[55061]: ts=2024-04-14T14:10:39.910Z caller=main.go:519 level=info fd_limits="(soft=1048576, hard=1048576)" 2024-04-14T14:10:40.346 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Apr 14 14:10:39 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-prometheus-a[55061]: ts=2024-04-14T14:10:39.910Z caller=main.go:520 level=info vm_limits="(soft=unlimited, hard=unlimited)" 2024-04-14T14:10:40.346 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Apr 14 14:10:39 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-prometheus-a[55061]: ts=2024-04-14T14:10:39.912Z caller=web.go:570 level=info component=web msg="Start listening for connections" address=:9095 2024-04-14T14:10:40.346 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Apr 14 14:10:39 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-prometheus-a[55061]: ts=2024-04-14T14:10:39.912Z caller=main.go:923 level=info msg="Starting TSDB ..." 2024-04-14T14:10:40.346 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Apr 14 14:10:39 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-prometheus-a[55061]: ts=2024-04-14T14:10:39.913Z caller=tls_config.go:195 level=info component=web msg="TLS is disabled." http2=false 2024-04-14T14:10:40.346 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Apr 14 14:10:39 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-prometheus-a[55061]: ts=2024-04-14T14:10:39.914Z caller=head.go:493 level=info component=tsdb msg="Replaying on-disk memory mappable chunks if any" 2024-04-14T14:10:40.346 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Apr 14 14:10:39 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-prometheus-a[55061]: ts=2024-04-14T14:10:39.916Z caller=head.go:527 level=info component=tsdb msg="On-disk memory mappable chunks replay completed" duration=1.382349ms 2024-04-14T14:10:40.346 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Apr 14 14:10:39 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-prometheus-a[55061]: ts=2024-04-14T14:10:39.916Z caller=head.go:533 level=info component=tsdb msg="Replaying WAL, this may take a while" 2024-04-14T14:10:40.346 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Apr 14 14:10:39 smithi122 bash[55046]: 9ebe35247d1a39f019e49a526e82fee7cb72172eb0477901dbef3a479dc4f20c 2024-04-14T14:10:40.346 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Apr 14 14:10:39 smithi122 systemd[1]: Started Ceph prometheus.a for dc326fc2-fa65-11ee-bc8e-c7b262605968. 2024-04-14T14:10:40.347 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Apr 14 14:10:40 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-prometheus-a[55061]: ts=2024-04-14T14:10:40.119Z caller=head.go:604 level=info component=tsdb msg="WAL segment loaded" segment=0 maxSegment=3 2024-04-14T14:10:40.607 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Apr 14 14:10:40 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-prometheus-a[55061]: ts=2024-04-14T14:10:40.379Z caller=head.go:604 level=info component=tsdb msg="WAL segment loaded" segment=1 maxSegment=3 2024-04-14T14:10:40.607 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Apr 14 14:10:40 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-prometheus-a[55061]: ts=2024-04-14T14:10:40.605Z caller=head.go:604 level=info component=tsdb msg="WAL segment loaded" segment=2 maxSegment=3 2024-04-14T14:10:40.607 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Apr 14 14:10:40 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-prometheus-a[55061]: ts=2024-04-14T14:10:40.605Z caller=head.go:604 level=info component=tsdb msg="WAL segment loaded" segment=3 maxSegment=3 2024-04-14T14:10:40.607 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Apr 14 14:10:40 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-prometheus-a[55061]: ts=2024-04-14T14:10:40.605Z caller=head.go:610 level=info component=tsdb msg="WAL replay completed" checkpoint_replay_duration=49.419µs wal_replay_duration=689.880232ms total_replay_duration=691.333857ms 2024-04-14T14:10:40.932 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Apr 14 14:10:40 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-prometheus-a[55061]: ts=2024-04-14T14:10:40.611Z caller=main.go:944 level=info fs_type=XFS_SUPER_MAGIC 2024-04-14T14:10:40.932 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Apr 14 14:10:40 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-prometheus-a[55061]: ts=2024-04-14T14:10:40.611Z caller=main.go:947 level=info msg="TSDB started" 2024-04-14T14:10:40.932 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Apr 14 14:10:40 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-prometheus-a[55061]: ts=2024-04-14T14:10:40.611Z caller=main.go:1128 level=info msg="Loading configuration file" filename=/etc/prometheus/prometheus.yml 2024-04-14T14:10:40.932 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Apr 14 14:10:40 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-prometheus-a[55061]: ts=2024-04-14T14:10:40.638Z caller=main.go:1165 level=info msg="Completed loading of configuration file" filename=/etc/prometheus/prometheus.yml totalDuration=27.526278ms db_storage=913ns remote_storage=1.278µs web_handler=304ns query_engine=1.039µs scrape=519.278µs scrape_sd=141.714µs notify=12.15µs notify_sd=10.904µs rules=26.349651ms 2024-04-14T14:10:40.932 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Apr 14 14:10:40 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-prometheus-a[55061]: ts=2024-04-14T14:10:40.638Z caller=main.go:896 level=info msg="Server is ready to receive web requests." 2024-04-14T14:10:41.307 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:10:41 smithi122 ceph-mon[25721]: from='mgr.25105 ' entity='mgr.y' 2024-04-14T14:10:41.307 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:10:41 smithi122 ceph-mon[25721]: from='mgr.25105 ' entity='mgr.y' 2024-04-14T14:10:41.307 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:10:41 smithi122 ceph-mon[25721]: from='mgr.25105 172.21.15.69:0/4213893117' entity='mgr.y' cmd=[{"prefix": "dashboard iscsi-gateway-list"}]: dispatch 2024-04-14T14:10:41.307 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:10:41 smithi122 ceph-mon[25721]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard iscsi-gateway-list"}]: dispatch 2024-04-14T14:10:41.307 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:10:41 smithi122 ceph-mon[25721]: from='mgr.25105 ' entity='mgr.y' 2024-04-14T14:10:41.307 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:10:41 smithi122 ceph-mon[25721]: Adding iSCSI gateway http://:@172.21.15.69:5000 to Dashboard 2024-04-14T14:10:41.307 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:10:41 smithi122 ceph-mon[25721]: from='mgr.25105 172.21.15.69:0/4213893117' entity='mgr.y' cmd=[{"prefix": "dashboard set-iscsi-api-ssl-verification", "value": "true"}]: dispatch 2024-04-14T14:10:41.308 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:10:41 smithi122 ceph-mon[25721]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-iscsi-api-ssl-verification", "value": "true"}]: dispatch 2024-04-14T14:10:41.308 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:10:41 smithi122 ceph-mon[25721]: from='mgr.25105 172.21.15.69:0/4213893117' entity='mgr.y' cmd=[{"prefix": "dashboard iscsi-gateway-add", "name": "smithi069"}]: dispatch 2024-04-14T14:10:41.308 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:10:41 smithi122 ceph-mon[25721]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard iscsi-gateway-add", "name": "smithi069"}]: dispatch 2024-04-14T14:10:41.308 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:10:41 smithi122 ceph-mon[25721]: from='mgr.25105 ' entity='mgr.y' 2024-04-14T14:10:41.308 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:10:41 smithi122 ceph-mon[25721]: from='mgr.25105 172.21.15.69:0/4213893117' entity='mgr.y' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-04-14T14:10:41.308 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:10:41 smithi122 ceph-mon[25721]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-04-14T14:10:41.308 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:10:41 smithi122 ceph-mon[25721]: pgmap v13: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 893 B/s rd, 1 op/s 2024-04-14T14:10:41.317 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:10:41 smithi069 ceph-mon[24818]: from='mgr.25105 ' entity='mgr.y' 2024-04-14T14:10:41.317 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:10:41 smithi069 ceph-mon[24818]: from='mgr.25105 ' entity='mgr.y' 2024-04-14T14:10:41.317 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:10:41 smithi069 ceph-mon[24818]: from='mgr.25105 172.21.15.69:0/4213893117' entity='mgr.y' cmd=[{"prefix": "dashboard iscsi-gateway-list"}]: dispatch 2024-04-14T14:10:41.317 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:10:41 smithi069 ceph-mon[24818]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard iscsi-gateway-list"}]: dispatch 2024-04-14T14:10:41.318 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:10:41 smithi069 ceph-mon[24818]: from='mgr.25105 ' entity='mgr.y' 2024-04-14T14:10:41.318 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:10:41 smithi069 ceph-mon[24818]: Adding iSCSI gateway http://:@172.21.15.69:5000 to Dashboard 2024-04-14T14:10:41.318 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:10:41 smithi069 ceph-mon[24818]: from='mgr.25105 172.21.15.69:0/4213893117' entity='mgr.y' cmd=[{"prefix": "dashboard set-iscsi-api-ssl-verification", "value": "true"}]: dispatch 2024-04-14T14:10:41.318 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:10:41 smithi069 ceph-mon[24818]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-iscsi-api-ssl-verification", "value": "true"}]: dispatch 2024-04-14T14:10:41.318 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:10:41 smithi069 ceph-mon[24818]: from='mgr.25105 172.21.15.69:0/4213893117' entity='mgr.y' cmd=[{"prefix": "dashboard iscsi-gateway-add", "name": "smithi069"}]: dispatch 2024-04-14T14:10:41.318 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:10:41 smithi069 ceph-mon[24818]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard iscsi-gateway-add", "name": "smithi069"}]: dispatch 2024-04-14T14:10:41.318 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:10:41 smithi069 ceph-mon[24818]: from='mgr.25105 ' entity='mgr.y' 2024-04-14T14:10:41.318 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:10:41 smithi069 ceph-mon[24818]: from='mgr.25105 172.21.15.69:0/4213893117' entity='mgr.y' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-04-14T14:10:41.318 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:10:41 smithi069 ceph-mon[24818]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-04-14T14:10:41.318 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:10:41 smithi069 ceph-mon[24818]: pgmap v13: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 893 B/s rd, 1 op/s 2024-04-14T14:10:41.318 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:10:41 smithi069 ceph-mon[29472]: from='mgr.25105 ' entity='mgr.y' 2024-04-14T14:10:41.319 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:10:41 smithi069 ceph-mon[29472]: from='mgr.25105 ' entity='mgr.y' 2024-04-14T14:10:41.319 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:10:41 smithi069 ceph-mon[29472]: from='mgr.25105 172.21.15.69:0/4213893117' entity='mgr.y' cmd=[{"prefix": "dashboard iscsi-gateway-list"}]: dispatch 2024-04-14T14:10:41.319 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:10:41 smithi069 ceph-mon[29472]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard iscsi-gateway-list"}]: dispatch 2024-04-14T14:10:41.319 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:10:41 smithi069 ceph-mon[29472]: from='mgr.25105 ' entity='mgr.y' 2024-04-14T14:10:41.319 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:10:41 smithi069 ceph-mon[29472]: Adding iSCSI gateway http://:@172.21.15.69:5000 to Dashboard 2024-04-14T14:10:41.319 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:10:41 smithi069 ceph-mon[29472]: from='mgr.25105 172.21.15.69:0/4213893117' entity='mgr.y' cmd=[{"prefix": "dashboard set-iscsi-api-ssl-verification", "value": "true"}]: dispatch 2024-04-14T14:10:41.319 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:10:41 smithi069 ceph-mon[29472]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-iscsi-api-ssl-verification", "value": "true"}]: dispatch 2024-04-14T14:10:41.319 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:10:41 smithi069 ceph-mon[29472]: from='mgr.25105 172.21.15.69:0/4213893117' entity='mgr.y' cmd=[{"prefix": "dashboard iscsi-gateway-add", "name": "smithi069"}]: dispatch 2024-04-14T14:10:41.319 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:10:41 smithi069 ceph-mon[29472]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard iscsi-gateway-add", "name": "smithi069"}]: dispatch 2024-04-14T14:10:41.319 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:10:41 smithi069 ceph-mon[29472]: from='mgr.25105 ' entity='mgr.y' 2024-04-14T14:10:41.319 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:10:41 smithi069 ceph-mon[29472]: from='mgr.25105 172.21.15.69:0/4213893117' entity='mgr.y' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-04-14T14:10:41.319 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:10:41 smithi069 ceph-mon[29472]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-04-14T14:10:41.319 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:10:41 smithi069 ceph-mon[29472]: pgmap v13: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 893 B/s rd, 1 op/s 2024-04-14T14:10:44.317 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:10:43 smithi069 ceph-mon[24818]: pgmap v14: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-04-14T14:10:44.317 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:10:43 smithi069 ceph-mon[29472]: pgmap v14: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-04-14T14:10:44.345 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:10:43 smithi122 ceph-mon[25721]: pgmap v14: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-04-14T14:10:45.317 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:10:45 smithi069 ceph-mon[24818]: from='mgr.25105 ' entity='mgr.y' 2024-04-14T14:10:45.317 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:10:45 smithi069 ceph-mon[24818]: from='mgr.25105 ' entity='mgr.y' 2024-04-14T14:10:45.317 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:10:45 smithi069 ceph-mon[24818]: from='mgr.25105 ' entity='mgr.y' 2024-04-14T14:10:45.317 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:10:45 smithi069 ceph-mon[24818]: from='mgr.25105 ' entity='mgr.y' 2024-04-14T14:10:45.317 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:10:45 smithi069 ceph-mon[24818]: from='mgr.25105 172.21.15.69:0/4213893117' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T14:10:45.317 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:10:45 smithi069 ceph-mon[24818]: from='mgr.25105 172.21.15.69:0/4213893117' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-14T14:10:45.317 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:10:45 smithi069 ceph-mon[24818]: pgmap v15: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:10:45.317 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:10:45 smithi069 ceph-mon[24818]: pgmap v16: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 963 B/s rd, 1 op/s 2024-04-14T14:10:45.318 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:10:45 smithi069 ceph-mon[24818]: from='mgr.25105 ' entity='mgr.y' 2024-04-14T14:10:45.318 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:10:45 smithi069 ceph-mon[29472]: from='mgr.25105 ' entity='mgr.y' 2024-04-14T14:10:45.318 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:10:45 smithi069 ceph-mon[29472]: from='mgr.25105 ' entity='mgr.y' 2024-04-14T14:10:45.318 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:10:45 smithi069 ceph-mon[29472]: from='mgr.25105 ' entity='mgr.y' 2024-04-14T14:10:45.318 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:10:45 smithi069 ceph-mon[29472]: from='mgr.25105 ' entity='mgr.y' 2024-04-14T14:10:45.318 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:10:45 smithi069 ceph-mon[29472]: from='mgr.25105 172.21.15.69:0/4213893117' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T14:10:45.318 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:10:45 smithi069 ceph-mon[29472]: from='mgr.25105 172.21.15.69:0/4213893117' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-14T14:10:45.318 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:10:45 smithi069 ceph-mon[29472]: pgmap v15: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:10:45.318 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:10:45 smithi069 ceph-mon[29472]: pgmap v16: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 963 B/s rd, 1 op/s 2024-04-14T14:10:45.318 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:10:45 smithi069 ceph-mon[29472]: from='mgr.25105 ' entity='mgr.y' 2024-04-14T14:10:45.344 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:10:45 smithi122 ceph-mon[25721]: from='mgr.25105 ' entity='mgr.y' 2024-04-14T14:10:45.345 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:10:45 smithi122 ceph-mon[25721]: from='mgr.25105 ' entity='mgr.y' 2024-04-14T14:10:45.345 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:10:45 smithi122 ceph-mon[25721]: from='mgr.25105 ' entity='mgr.y' 2024-04-14T14:10:45.345 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:10:45 smithi122 ceph-mon[25721]: from='mgr.25105 ' entity='mgr.y' 2024-04-14T14:10:45.345 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:10:45 smithi122 ceph-mon[25721]: from='mgr.25105 172.21.15.69:0/4213893117' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T14:10:45.345 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:10:45 smithi122 ceph-mon[25721]: from='mgr.25105 172.21.15.69:0/4213893117' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-14T14:10:45.345 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:10:45 smithi122 ceph-mon[25721]: pgmap v15: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:10:45.345 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:10:45 smithi122 ceph-mon[25721]: pgmap v16: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 963 B/s rd, 1 op/s 2024-04-14T14:10:45.345 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:10:45 smithi122 ceph-mon[25721]: from='mgr.25105 ' entity='mgr.y' 2024-04-14T14:10:46.817 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:10:46 smithi069 ceph-mon[24818]: from='client.15285 -' entity='client.iscsi.foo.smithi069.yuumtr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T14:10:46.817 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 14 14:10:46 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-y[63038]: ::ffff:172.21.15.122 - - [14/Apr/2024:14:10:46] "GET /metrics HTTP/1.1" 200 37986 "" "Prometheus/2.33.4" 2024-04-14T14:10:46.818 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:10:46 smithi069 ceph-mon[29472]: from='client.15285 -' entity='client.iscsi.foo.smithi069.yuumtr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T14:10:46.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:10:46 smithi122 ceph-mon[25721]: from='client.15285 -' entity='client.iscsi.foo.smithi069.yuumtr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T14:10:47.817 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:10:47 smithi069 ceph-mon[24818]: pgmap v17: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 963 B/s rd, 1 op/s 2024-04-14T14:10:47.817 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:10:47 smithi069 ceph-mon[29472]: pgmap v17: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 963 B/s rd, 1 op/s 2024-04-14T14:10:47.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:10:47 smithi122 ceph-mon[25721]: pgmap v17: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 963 B/s rd, 1 op/s 2024-04-14T14:10:49.817 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:10:49 smithi069 ceph-mon[24818]: pgmap v18: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 535 B/s rd, 0 op/s 2024-04-14T14:10:49.817 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:10:49 smithi069 ceph-mon[29472]: pgmap v18: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 535 B/s rd, 0 op/s 2024-04-14T14:10:49.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:10:49 smithi122 ceph-mon[25721]: pgmap v18: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 535 B/s rd, 0 op/s 2024-04-14T14:10:51.817 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:10:51 smithi069 ceph-mon[24818]: pgmap v19: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-04-14T14:10:51.817 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:10:51 smithi069 ceph-mon[29472]: pgmap v19: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-04-14T14:10:51.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:10:51 smithi122 ceph-mon[25721]: pgmap v19: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-04-14T14:10:53.817 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:10:53 smithi069 ceph-mon[24818]: pgmap v20: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-04-14T14:10:53.817 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:10:53 smithi069 ceph-mon[24818]: from='mgr.25105 172.21.15.69:0/4213893117' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T14:10:53.817 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:10:53 smithi069 ceph-mon[29472]: pgmap v20: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-04-14T14:10:53.818 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:10:53 smithi069 ceph-mon[29472]: from='mgr.25105 172.21.15.69:0/4213893117' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T14:10:53.844 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:10:53 smithi122 ceph-mon[25721]: pgmap v20: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-04-14T14:10:53.844 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:10:53 smithi122 ceph-mon[25721]: from='mgr.25105 172.21.15.69:0/4213893117' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T14:10:55.817 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:10:55 smithi069 ceph-mon[29472]: pgmap v21: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-14T14:10:55.817 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:10:55 smithi069 ceph-mon[24818]: pgmap v21: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-14T14:10:55.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:10:55 smithi122 ceph-mon[25721]: pgmap v21: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-14T14:10:56.817 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:10:56 smithi069 ceph-mon[24818]: from='client.15285 -' entity='client.iscsi.foo.smithi069.yuumtr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T14:10:56.817 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 14 14:10:56 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-y[63038]: ::ffff:172.21.15.122 - - [14/Apr/2024:14:10:56] "GET /metrics HTTP/1.1" 200 37980 "" "Prometheus/2.33.4" 2024-04-14T14:10:56.818 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:10:56 smithi069 ceph-mon[29472]: from='client.15285 -' entity='client.iscsi.foo.smithi069.yuumtr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T14:10:56.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:10:56 smithi122 ceph-mon[25721]: from='client.15285 -' entity='client.iscsi.foo.smithi069.yuumtr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T14:10:57.817 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:10:57 smithi069 ceph-mon[24818]: pgmap v22: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:10:57.817 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:10:57 smithi069 ceph-mon[29472]: pgmap v22: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:10:57.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:10:57 smithi122 ceph-mon[25721]: pgmap v22: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:10:59.817 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:10:59 smithi069 ceph-mon[24818]: pgmap v23: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:10:59.817 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:10:59 smithi069 ceph-mon[29472]: pgmap v23: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:10:59.844 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:10:59 smithi122 ceph-mon[25721]: pgmap v23: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:11:01.817 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:11:01 smithi069 ceph-mon[24818]: pgmap v24: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:11:01.817 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:11:01 smithi069 ceph-mon[29472]: pgmap v24: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:11:01.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:11:01 smithi122 ceph-mon[25721]: pgmap v24: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:11:03.817 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:11:03 smithi069 ceph-mon[24818]: pgmap v25: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:11:03.817 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:11:03 smithi069 ceph-mon[29472]: pgmap v25: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:11:03.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:11:03 smithi122 ceph-mon[25721]: pgmap v25: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:11:05.817 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:11:05 smithi069 ceph-mon[24818]: pgmap v26: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:11:05.817 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:11:05 smithi069 ceph-mon[29472]: pgmap v26: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:11:05.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:11:05 smithi122 ceph-mon[25721]: pgmap v26: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:11:06.817 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:11:06 smithi069 ceph-mon[24818]: from='client.15285 -' entity='client.iscsi.foo.smithi069.yuumtr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T14:11:06.817 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 14 14:11:06 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-y[63038]: ::ffff:172.21.15.122 - - [14/Apr/2024:14:11:06] "GET /metrics HTTP/1.1" 200 37980 "" "Prometheus/2.33.4" 2024-04-14T14:11:06.817 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:11:06 smithi069 ceph-mon[29472]: from='client.15285 -' entity='client.iscsi.foo.smithi069.yuumtr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T14:11:06.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:11:06 smithi122 ceph-mon[25721]: from='client.15285 -' entity='client.iscsi.foo.smithi069.yuumtr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T14:11:07.817 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:11:07 smithi069 ceph-mon[24818]: pgmap v27: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:11:07.817 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:11:07 smithi069 ceph-mon[29472]: pgmap v27: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:11:07.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:11:07 smithi122 ceph-mon[25721]: pgmap v27: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:11:08.817 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:11:08 smithi069 ceph-mon[24818]: from='mgr.25105 172.21.15.69:0/4213893117' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T14:11:08.817 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:11:08 smithi069 ceph-mon[29472]: from='mgr.25105 172.21.15.69:0/4213893117' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T14:11:08.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:11:08 smithi122 ceph-mon[25721]: from='mgr.25105 172.21.15.69:0/4213893117' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T14:11:09.817 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:11:09 smithi069 ceph-mon[24818]: pgmap v28: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:11:09.817 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:11:09 smithi069 ceph-mon[29472]: pgmap v28: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:11:09.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:11:09 smithi122 ceph-mon[25721]: pgmap v28: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:11:11.817 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:11:11 smithi069 ceph-mon[24818]: pgmap v29: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:11:11.817 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:11:11 smithi069 ceph-mon[29472]: pgmap v29: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:11:11.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:11:11 smithi122 ceph-mon[25721]: pgmap v29: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:11:13.817 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:11:13 smithi069 ceph-mon[24818]: pgmap v30: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:11:13.817 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:11:13 smithi069 ceph-mon[29472]: pgmap v30: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:11:13.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:11:13 smithi122 ceph-mon[25721]: pgmap v30: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:11:15.817 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:11:15 smithi069 ceph-mon[24818]: pgmap v31: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:11:15.817 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:11:15 smithi069 ceph-mon[29472]: pgmap v31: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:11:15.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:11:15 smithi122 ceph-mon[25721]: pgmap v31: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:11:16.817 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:11:16 smithi069 ceph-mon[24818]: from='client.15285 -' entity='client.iscsi.foo.smithi069.yuumtr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T14:11:16.817 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:11:16 smithi069 ceph-mon[29472]: from='client.15285 -' entity='client.iscsi.foo.smithi069.yuumtr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T14:11:16.818 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 14 14:11:16 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-y[63038]: ::ffff:172.21.15.122 - - [14/Apr/2024:14:11:16] "GET /metrics HTTP/1.1" 200 37980 "" "Prometheus/2.33.4" 2024-04-14T14:11:16.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:11:16 smithi122 ceph-mon[25721]: from='client.15285 -' entity='client.iscsi.foo.smithi069.yuumtr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T14:11:17.817 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:11:17 smithi069 ceph-mon[24818]: pgmap v32: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:11:17.817 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:11:17 smithi069 ceph-mon[29472]: pgmap v32: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:11:17.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:11:17 smithi122 ceph-mon[25721]: pgmap v32: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:11:19.817 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:11:19 smithi069 ceph-mon[24818]: pgmap v33: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:11:19.817 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:11:19 smithi069 ceph-mon[29472]: pgmap v33: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:11:19.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:11:19 smithi122 ceph-mon[25721]: pgmap v33: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:11:21.817 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:11:21 smithi069 ceph-mon[24818]: pgmap v34: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:11:21.817 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:11:21 smithi069 ceph-mon[29472]: pgmap v34: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:11:21.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:11:21 smithi122 ceph-mon[25721]: pgmap v34: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:11:23.817 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:11:23 smithi069 ceph-mon[24818]: pgmap v35: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:11:23.817 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:11:23 smithi069 ceph-mon[24818]: from='mgr.25105 172.21.15.69:0/4213893117' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T14:11:23.817 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:11:23 smithi069 ceph-mon[29472]: pgmap v35: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:11:23.817 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:11:23 smithi069 ceph-mon[29472]: from='mgr.25105 172.21.15.69:0/4213893117' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T14:11:23.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:11:23 smithi122 ceph-mon[25721]: pgmap v35: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:11:23.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:11:23 smithi122 ceph-mon[25721]: from='mgr.25105 172.21.15.69:0/4213893117' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T14:11:25.817 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:11:25 smithi069 ceph-mon[24818]: pgmap v36: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:11:25.817 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:11:25 smithi069 ceph-mon[29472]: pgmap v36: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:11:25.844 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:11:25 smithi122 ceph-mon[25721]: pgmap v36: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:11:26.775 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 14 14:11:26 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-y[63038]: ::ffff:172.21.15.122 - - [14/Apr/2024:14:11:26] "GET /metrics HTTP/1.1" 200 37977 "" "Prometheus/2.33.4" 2024-04-14T14:11:26.776 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:11:26 smithi069 ceph-mon[24818]: from='client.15285 -' entity='client.iscsi.foo.smithi069.yuumtr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T14:11:26.776 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:11:26 smithi069 ceph-mon[29472]: from='client.15285 -' entity='client.iscsi.foo.smithi069.yuumtr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T14:11:26.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:11:26 smithi122 ceph-mon[25721]: from='client.15285 -' entity='client.iscsi.foo.smithi069.yuumtr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T14:11:27.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:11:27 smithi122 ceph-mon[25721]: pgmap v37: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:11:28.067 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:11:27 smithi069 ceph-mon[24818]: pgmap v37: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:11:28.067 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:11:27 smithi069 ceph-mon[29472]: pgmap v37: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:11:29.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:11:29 smithi122 ceph-mon[25721]: pgmap v38: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:11:30.067 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:11:29 smithi069 ceph-mon[24818]: pgmap v38: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:11:30.067 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:11:29 smithi069 ceph-mon[29472]: pgmap v38: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:11:31.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:11:31 smithi122 ceph-mon[25721]: pgmap v39: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:11:32.067 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:11:31 smithi069 ceph-mon[24818]: pgmap v39: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:11:32.067 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:11:31 smithi069 ceph-mon[29472]: pgmap v39: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:11:33.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:11:33 smithi122 ceph-mon[25721]: pgmap v40: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:11:34.067 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:11:33 smithi069 ceph-mon[24818]: pgmap v40: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:11:34.067 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:11:33 smithi069 ceph-mon[29472]: pgmap v40: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:11:35.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:11:35 smithi122 ceph-mon[25721]: pgmap v41: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:11:36.067 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:11:35 smithi069 ceph-mon[24818]: pgmap v41: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:11:36.067 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:11:35 smithi069 ceph-mon[29472]: pgmap v41: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:11:36.817 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:11:36 smithi069 ceph-mon[24818]: from='client.15285 -' entity='client.iscsi.foo.smithi069.yuumtr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T14:11:36.817 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 14 14:11:36 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-y[63038]: ::ffff:172.21.15.122 - - [14/Apr/2024:14:11:36] "GET /metrics HTTP/1.1" 200 37977 "" "Prometheus/2.33.4" 2024-04-14T14:11:36.817 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:11:36 smithi069 ceph-mon[29472]: from='client.15285 -' entity='client.iscsi.foo.smithi069.yuumtr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T14:11:36.844 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:11:36 smithi122 ceph-mon[25721]: from='client.15285 -' entity='client.iscsi.foo.smithi069.yuumtr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T14:11:37.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:11:37 smithi122 ceph-mon[25721]: pgmap v42: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:11:37.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:11:37 smithi122 ceph-mon[25721]: from='mgr.25105 172.21.15.69:0/4213893117' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T14:11:38.067 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:11:37 smithi069 ceph-mon[24818]: pgmap v42: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:11:38.067 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:11:37 smithi069 ceph-mon[24818]: from='mgr.25105 172.21.15.69:0/4213893117' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T14:11:38.067 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:11:37 smithi069 ceph-mon[29472]: pgmap v42: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:11:38.067 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:11:37 smithi069 ceph-mon[29472]: from='mgr.25105 172.21.15.69:0/4213893117' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T14:11:39.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:11:39 smithi122 ceph-mon[25721]: pgmap v43: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:11:40.067 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:11:39 smithi069 ceph-mon[24818]: pgmap v43: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:11:40.067 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:11:39 smithi069 ceph-mon[29472]: pgmap v43: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:11:42.067 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:11:41 smithi069 ceph-mon[24818]: pgmap v44: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:11:42.067 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:11:41 smithi069 ceph-mon[29472]: pgmap v44: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:11:42.094 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:11:41 smithi122 ceph-mon[25721]: pgmap v44: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:11:44.067 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:11:43 smithi069 ceph-mon[24818]: pgmap v45: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:11:44.067 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:11:43 smithi069 ceph-mon[29472]: pgmap v45: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:11:44.094 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:11:43 smithi122 ceph-mon[25721]: pgmap v45: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:11:46.067 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:11:45 smithi069 ceph-mon[24818]: pgmap v46: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:11:46.067 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:11:45 smithi069 ceph-mon[24818]: from='mgr.25105 ' entity='mgr.y' 2024-04-14T14:11:46.067 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:11:45 smithi069 ceph-mon[24818]: from='mgr.25105 ' entity='mgr.y' 2024-04-14T14:11:46.067 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:11:45 smithi069 ceph-mon[29472]: pgmap v46: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:11:46.067 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:11:45 smithi069 ceph-mon[29472]: from='mgr.25105 ' entity='mgr.y' 2024-04-14T14:11:46.067 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:11:45 smithi069 ceph-mon[29472]: from='mgr.25105 ' entity='mgr.y' 2024-04-14T14:11:46.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:11:45 smithi122 ceph-mon[25721]: pgmap v46: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:11:46.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:11:45 smithi122 ceph-mon[25721]: from='mgr.25105 ' entity='mgr.y' 2024-04-14T14:11:46.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:11:45 smithi122 ceph-mon[25721]: from='mgr.25105 ' entity='mgr.y' 2024-04-14T14:11:46.817 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 14 14:11:46 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-y[63038]: ::ffff:172.21.15.122 - - [14/Apr/2024:14:11:46] "GET /metrics HTTP/1.1" 200 37977 "" "Prometheus/2.33.4" 2024-04-14T14:11:46.817 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:11:46 smithi069 ceph-mon[24818]: from='client.15285 -' entity='client.iscsi.foo.smithi069.yuumtr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T14:11:46.817 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:11:46 smithi069 ceph-mon[24818]: from='mgr.25105 172.21.15.69:0/4213893117' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T14:11:46.817 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:11:46 smithi069 ceph-mon[24818]: from='mgr.25105 172.21.15.69:0/4213893117' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-14T14:11:46.817 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:11:46 smithi069 ceph-mon[24818]: pgmap v47: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 908 B/s rd, 0 op/s 2024-04-14T14:11:46.818 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:11:46 smithi069 ceph-mon[24818]: pgmap v48: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-04-14T14:11:46.818 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:11:46 smithi069 ceph-mon[24818]: from='mgr.25105 ' entity='mgr.y' 2024-04-14T14:11:46.818 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:11:46 smithi069 ceph-mon[29472]: from='client.15285 -' entity='client.iscsi.foo.smithi069.yuumtr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T14:11:46.818 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:11:46 smithi069 ceph-mon[29472]: from='mgr.25105 172.21.15.69:0/4213893117' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T14:11:46.818 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:11:46 smithi069 ceph-mon[29472]: from='mgr.25105 172.21.15.69:0/4213893117' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-14T14:11:46.818 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:11:46 smithi069 ceph-mon[29472]: pgmap v47: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 908 B/s rd, 0 op/s 2024-04-14T14:11:46.818 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:11:46 smithi069 ceph-mon[29472]: pgmap v48: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-04-14T14:11:46.818 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:11:46 smithi069 ceph-mon[29472]: from='mgr.25105 ' entity='mgr.y' 2024-04-14T14:11:47.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:11:46 smithi122 ceph-mon[25721]: from='client.15285 -' entity='client.iscsi.foo.smithi069.yuumtr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T14:11:47.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:11:46 smithi122 ceph-mon[25721]: from='mgr.25105 172.21.15.69:0/4213893117' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T14:11:47.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:11:46 smithi122 ceph-mon[25721]: from='mgr.25105 172.21.15.69:0/4213893117' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-14T14:11:47.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:11:46 smithi122 ceph-mon[25721]: pgmap v47: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 908 B/s rd, 0 op/s 2024-04-14T14:11:47.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:11:46 smithi122 ceph-mon[25721]: pgmap v48: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-04-14T14:11:47.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:11:46 smithi122 ceph-mon[25721]: from='mgr.25105 ' entity='mgr.y' 2024-04-14T14:11:49.067 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:11:48 smithi069 ceph-mon[24818]: pgmap v49: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-04-14T14:11:49.067 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:11:48 smithi069 ceph-mon[29472]: pgmap v49: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-04-14T14:11:49.094 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:11:48 smithi122 ceph-mon[25721]: pgmap v49: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-04-14T14:11:51.067 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:11:50 smithi069 ceph-mon[24818]: pgmap v50: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-04-14T14:11:51.067 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:11:50 smithi069 ceph-mon[29472]: pgmap v50: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-04-14T14:11:51.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:11:50 smithi122 ceph-mon[25721]: pgmap v50: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-04-14T14:11:53.067 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:11:52 smithi069 ceph-mon[24818]: pgmap v51: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-04-14T14:11:53.067 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:11:52 smithi069 ceph-mon[24818]: from='mgr.25105 172.21.15.69:0/4213893117' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T14:11:53.067 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:11:52 smithi069 ceph-mon[29472]: pgmap v51: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-04-14T14:11:53.067 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:11:52 smithi069 ceph-mon[29472]: from='mgr.25105 172.21.15.69:0/4213893117' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T14:11:53.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:11:52 smithi122 ceph-mon[25721]: pgmap v51: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-04-14T14:11:53.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:11:52 smithi122 ceph-mon[25721]: from='mgr.25105 172.21.15.69:0/4213893117' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T14:11:55.067 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:11:54 smithi069 ceph-mon[24818]: pgmap v52: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-04-14T14:11:55.067 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:11:54 smithi069 ceph-mon[29472]: pgmap v52: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-04-14T14:11:55.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:11:54 smithi122 ceph-mon[25721]: pgmap v52: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-04-14T14:11:56.817 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:11:56 smithi069 ceph-mon[24818]: from='client.15285 -' entity='client.iscsi.foo.smithi069.yuumtr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T14:11:56.817 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:11:56 smithi069 ceph-mon[24818]: pgmap v53: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-14T14:11:56.817 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 14 14:11:56 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-y[63038]: ::ffff:172.21.15.122 - - [14/Apr/2024:14:11:56] "GET /metrics HTTP/1.1" 200 37977 "" "Prometheus/2.33.4" 2024-04-14T14:11:56.817 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:11:56 smithi069 ceph-mon[29472]: from='client.15285 -' entity='client.iscsi.foo.smithi069.yuumtr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T14:11:56.818 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:11:56 smithi069 ceph-mon[29472]: pgmap v53: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-14T14:11:56.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:11:56 smithi122 ceph-mon[25721]: from='client.15285 -' entity='client.iscsi.foo.smithi069.yuumtr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T14:11:56.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:11:56 smithi122 ceph-mon[25721]: pgmap v53: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-14T14:11:59.067 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:11:58 smithi069 ceph-mon[24818]: pgmap v54: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:11:59.067 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:11:58 smithi069 ceph-mon[29472]: pgmap v54: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:11:59.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:11:58 smithi122 ceph-mon[25721]: pgmap v54: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:12:01.067 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:12:00 smithi069 ceph-mon[24818]: pgmap v55: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:12:01.067 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:12:00 smithi069 ceph-mon[29472]: pgmap v55: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:12:01.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:12:00 smithi122 ceph-mon[25721]: pgmap v55: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:12:03.067 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:12:02 smithi069 ceph-mon[24818]: pgmap v56: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:12:03.067 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:12:02 smithi069 ceph-mon[29472]: pgmap v56: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:12:03.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:12:02 smithi122 ceph-mon[25721]: pgmap v56: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:12:05.067 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:12:04 smithi069 ceph-mon[24818]: pgmap v57: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:12:05.067 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:12:04 smithi069 ceph-mon[29472]: pgmap v57: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:12:05.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:12:04 smithi122 ceph-mon[25721]: pgmap v57: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:12:06.817 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 14 14:12:06 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-y[63038]: ::ffff:172.21.15.122 - - [14/Apr/2024:14:12:06] "GET /metrics HTTP/1.1" 200 37977 "" "Prometheus/2.33.4" 2024-04-14T14:12:06.817 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:12:06 smithi069 ceph-mon[29472]: from='client.15285 -' entity='client.iscsi.foo.smithi069.yuumtr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T14:12:06.817 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:12:06 smithi069 ceph-mon[29472]: pgmap v58: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:12:06.817 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:12:06 smithi069 ceph-mon[24818]: from='client.15285 -' entity='client.iscsi.foo.smithi069.yuumtr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T14:12:06.818 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:12:06 smithi069 ceph-mon[24818]: pgmap v58: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:12:06.844 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:12:06 smithi122 ceph-mon[25721]: from='client.15285 -' entity='client.iscsi.foo.smithi069.yuumtr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T14:12:06.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:12:06 smithi122 ceph-mon[25721]: pgmap v58: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:12:08.067 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:12:07 smithi069 ceph-mon[24818]: from='mgr.25105 172.21.15.69:0/4213893117' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T14:12:08.067 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:12:07 smithi069 ceph-mon[29472]: from='mgr.25105 172.21.15.69:0/4213893117' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T14:12:08.094 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:12:07 smithi122 ceph-mon[25721]: from='mgr.25105 172.21.15.69:0/4213893117' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T14:12:09.067 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:12:08 smithi069 ceph-mon[24818]: pgmap v59: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:12:09.067 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:12:08 smithi069 ceph-mon[29472]: pgmap v59: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:12:09.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:12:08 smithi122 ceph-mon[25721]: pgmap v59: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:12:11.067 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:12:10 smithi069 ceph-mon[24818]: pgmap v60: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:12:11.067 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:12:10 smithi069 ceph-mon[29472]: pgmap v60: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:12:11.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:12:10 smithi122 ceph-mon[25721]: pgmap v60: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:12:13.067 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:12:12 smithi069 ceph-mon[24818]: pgmap v61: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:12:13.067 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:12:12 smithi069 ceph-mon[29472]: pgmap v61: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:12:13.094 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:12:12 smithi122 ceph-mon[25721]: pgmap v61: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:12:15.067 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:12:14 smithi069 ceph-mon[29472]: pgmap v62: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:12:15.067 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:12:14 smithi069 ceph-mon[24818]: pgmap v62: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:12:15.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:12:14 smithi122 ceph-mon[25721]: pgmap v62: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:12:16.817 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:12:16 smithi069 ceph-mon[24818]: from='client.15285 -' entity='client.iscsi.foo.smithi069.yuumtr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T14:12:16.817 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:12:16 smithi069 ceph-mon[24818]: pgmap v63: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:12:16.817 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 14 14:12:16 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-y[63038]: ::ffff:172.21.15.122 - - [14/Apr/2024:14:12:16] "GET /metrics HTTP/1.1" 200 37977 "" "Prometheus/2.33.4" 2024-04-14T14:12:16.818 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:12:16 smithi069 ceph-mon[29472]: from='client.15285 -' entity='client.iscsi.foo.smithi069.yuumtr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T14:12:16.818 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:12:16 smithi069 ceph-mon[29472]: pgmap v63: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:12:16.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:12:16 smithi122 ceph-mon[25721]: from='client.15285 -' entity='client.iscsi.foo.smithi069.yuumtr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T14:12:16.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:12:16 smithi122 ceph-mon[25721]: pgmap v63: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:12:19.067 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:12:18 smithi069 ceph-mon[24818]: pgmap v64: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:12:19.067 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:12:18 smithi069 ceph-mon[29472]: pgmap v64: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:12:19.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:12:18 smithi122 ceph-mon[25721]: pgmap v64: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:12:21.067 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:12:20 smithi069 ceph-mon[24818]: pgmap v65: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:12:21.067 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:12:20 smithi069 ceph-mon[29472]: pgmap v65: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:12:21.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:12:20 smithi122 ceph-mon[25721]: pgmap v65: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:12:23.067 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:12:22 smithi069 ceph-mon[24818]: pgmap v66: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:12:23.067 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:12:22 smithi069 ceph-mon[24818]: from='mgr.25105 172.21.15.69:0/4213893117' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T14:12:23.068 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:12:22 smithi069 ceph-mon[29472]: pgmap v66: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:12:23.068 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:12:22 smithi069 ceph-mon[29472]: from='mgr.25105 172.21.15.69:0/4213893117' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T14:12:23.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:12:22 smithi122 ceph-mon[25721]: pgmap v66: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:12:23.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:12:22 smithi122 ceph-mon[25721]: from='mgr.25105 172.21.15.69:0/4213893117' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T14:12:25.067 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:12:24 smithi069 ceph-mon[24818]: pgmap v67: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:12:25.067 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:12:24 smithi069 ceph-mon[29472]: pgmap v67: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:12:25.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:12:24 smithi122 ceph-mon[25721]: pgmap v67: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:12:26.817 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:12:26 smithi069 ceph-mon[24818]: from='client.15285 -' entity='client.iscsi.foo.smithi069.yuumtr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T14:12:26.817 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:12:26 smithi069 ceph-mon[24818]: pgmap v68: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:12:26.817 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 14 14:12:26 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-y[63038]: ::ffff:172.21.15.122 - - [14/Apr/2024:14:12:26] "GET /metrics HTTP/1.1" 200 37976 "" "Prometheus/2.33.4" 2024-04-14T14:12:26.818 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:12:26 smithi069 ceph-mon[29472]: from='client.15285 -' entity='client.iscsi.foo.smithi069.yuumtr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T14:12:26.818 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:12:26 smithi069 ceph-mon[29472]: pgmap v68: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:12:26.844 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:12:26 smithi122 ceph-mon[25721]: from='client.15285 -' entity='client.iscsi.foo.smithi069.yuumtr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T14:12:26.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:12:26 smithi122 ceph-mon[25721]: pgmap v68: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:12:29.067 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:12:28 smithi069 ceph-mon[24818]: pgmap v69: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:12:29.067 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:12:28 smithi069 ceph-mon[29472]: pgmap v69: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:12:29.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:12:28 smithi122 ceph-mon[25721]: pgmap v69: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:12:31.067 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:12:30 smithi069 ceph-mon[24818]: pgmap v70: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:12:31.067 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:12:30 smithi069 ceph-mon[29472]: pgmap v70: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:12:31.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:12:30 smithi122 ceph-mon[25721]: pgmap v70: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:12:33.067 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:12:32 smithi069 ceph-mon[24818]: pgmap v71: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:12:33.067 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:12:32 smithi069 ceph-mon[29472]: pgmap v71: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:12:33.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:12:32 smithi122 ceph-mon[25721]: pgmap v71: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:12:35.067 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:12:34 smithi069 ceph-mon[24818]: pgmap v72: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:12:35.067 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:12:34 smithi069 ceph-mon[29472]: pgmap v72: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:12:35.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:12:34 smithi122 ceph-mon[25721]: pgmap v72: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:12:36.817 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 14 14:12:36 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-y[63038]: ::ffff:172.21.15.122 - - [14/Apr/2024:14:12:36] "GET /metrics HTTP/1.1" 200 37976 "" "Prometheus/2.33.4" 2024-04-14T14:12:36.817 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:12:36 smithi069 ceph-mon[29472]: from='client.15285 -' entity='client.iscsi.foo.smithi069.yuumtr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T14:12:36.817 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:12:36 smithi069 ceph-mon[29472]: pgmap v73: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:12:36.817 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:12:36 smithi069 ceph-mon[24818]: from='client.15285 -' entity='client.iscsi.foo.smithi069.yuumtr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T14:12:36.818 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:12:36 smithi069 ceph-mon[24818]: pgmap v73: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:12:36.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:12:36 smithi122 ceph-mon[25721]: from='client.15285 -' entity='client.iscsi.foo.smithi069.yuumtr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T14:12:36.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:12:36 smithi122 ceph-mon[25721]: pgmap v73: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:12:38.067 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:12:37 smithi069 ceph-mon[24818]: from='mgr.25105 172.21.15.69:0/4213893117' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T14:12:38.067 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:12:37 smithi069 ceph-mon[29472]: from='mgr.25105 172.21.15.69:0/4213893117' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T14:12:38.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:12:37 smithi122 ceph-mon[25721]: from='mgr.25105 172.21.15.69:0/4213893117' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T14:12:39.067 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:12:38 smithi069 ceph-mon[24818]: pgmap v74: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:12:39.067 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:12:38 smithi069 ceph-mon[29472]: pgmap v74: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:12:39.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:12:38 smithi122 ceph-mon[25721]: pgmap v74: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:12:41.067 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:12:40 smithi069 ceph-mon[24818]: pgmap v75: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:12:41.067 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:12:40 smithi069 ceph-mon[29472]: pgmap v75: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:12:41.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:12:40 smithi122 ceph-mon[25721]: pgmap v75: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:12:43.067 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:12:42 smithi069 ceph-mon[24818]: pgmap v76: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:12:43.067 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:12:42 smithi069 ceph-mon[29472]: pgmap v76: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:12:43.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:12:42 smithi122 ceph-mon[25721]: pgmap v76: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:12:45.067 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:12:44 smithi069 ceph-mon[24818]: pgmap v77: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:12:45.067 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:12:44 smithi069 ceph-mon[29472]: pgmap v77: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:12:45.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:12:44 smithi122 ceph-mon[25721]: pgmap v77: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:12:46.817 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 14 14:12:46 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-y[63038]: ::ffff:172.21.15.122 - - [14/Apr/2024:14:12:46] "GET /metrics HTTP/1.1" 200 37988 "" "Prometheus/2.33.4" 2024-04-14T14:12:46.818 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:12:46 smithi069 ceph-mon[24818]: from='client.15285 -' entity='client.iscsi.foo.smithi069.yuumtr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T14:12:46.818 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:12:46 smithi069 ceph-mon[24818]: pgmap v78: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:12:46.818 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:12:46 smithi069 ceph-mon[24818]: from='mgr.25105 ' entity='mgr.y' 2024-04-14T14:12:46.818 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:12:46 smithi069 ceph-mon[24818]: from='mgr.25105 ' entity='mgr.y' 2024-04-14T14:12:46.819 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:12:46 smithi069 ceph-mon[29472]: from='client.15285 -' entity='client.iscsi.foo.smithi069.yuumtr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T14:12:46.819 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:12:46 smithi069 ceph-mon[29472]: pgmap v78: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:12:46.819 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:12:46 smithi069 ceph-mon[29472]: from='mgr.25105 ' entity='mgr.y' 2024-04-14T14:12:46.819 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:12:46 smithi069 ceph-mon[29472]: from='mgr.25105 ' entity='mgr.y' 2024-04-14T14:12:46.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:12:46 smithi122 ceph-mon[25721]: from='client.15285 -' entity='client.iscsi.foo.smithi069.yuumtr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T14:12:46.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:12:46 smithi122 ceph-mon[25721]: pgmap v78: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:12:46.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:12:46 smithi122 ceph-mon[25721]: from='mgr.25105 ' entity='mgr.y' 2024-04-14T14:12:46.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:12:46 smithi122 ceph-mon[25721]: from='mgr.25105 ' entity='mgr.y' 2024-04-14T14:12:47.817 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:12:47 smithi069 ceph-mon[24818]: from='mgr.25105 172.21.15.69:0/4213893117' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T14:12:47.817 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:12:47 smithi069 ceph-mon[24818]: from='mgr.25105 172.21.15.69:0/4213893117' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-14T14:12:47.817 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:12:47 smithi069 ceph-mon[24818]: pgmap v79: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 914 B/s rd, 0 op/s 2024-04-14T14:12:47.817 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:12:47 smithi069 ceph-mon[24818]: pgmap v80: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-04-14T14:12:47.817 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:12:47 smithi069 ceph-mon[24818]: from='mgr.25105 ' entity='mgr.y' 2024-04-14T14:12:47.818 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:12:47 smithi069 ceph-mon[29472]: from='mgr.25105 172.21.15.69:0/4213893117' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T14:12:47.818 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:12:47 smithi069 ceph-mon[29472]: from='mgr.25105 172.21.15.69:0/4213893117' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-14T14:12:47.818 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:12:47 smithi069 ceph-mon[29472]: pgmap v79: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 914 B/s rd, 0 op/s 2024-04-14T14:12:47.818 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:12:47 smithi069 ceph-mon[29472]: pgmap v80: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-04-14T14:12:47.818 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:12:47 smithi069 ceph-mon[29472]: from='mgr.25105 ' entity='mgr.y' 2024-04-14T14:12:47.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:12:47 smithi122 ceph-mon[25721]: from='mgr.25105 172.21.15.69:0/4213893117' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T14:12:47.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:12:47 smithi122 ceph-mon[25721]: from='mgr.25105 172.21.15.69:0/4213893117' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-14T14:12:47.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:12:47 smithi122 ceph-mon[25721]: pgmap v79: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 914 B/s rd, 0 op/s 2024-04-14T14:12:47.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:12:47 smithi122 ceph-mon[25721]: pgmap v80: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-04-14T14:12:47.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:12:47 smithi122 ceph-mon[25721]: from='mgr.25105 ' entity='mgr.y' 2024-04-14T14:12:49.317 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:12:48 smithi069 ceph-mon[24818]: pgmap v81: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-04-14T14:12:49.317 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:12:48 smithi069 ceph-mon[29472]: pgmap v81: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-04-14T14:12:49.345 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:12:48 smithi122 ceph-mon[25721]: pgmap v81: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-04-14T14:12:51.817 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:12:51 smithi069 ceph-mon[24818]: pgmap v82: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-04-14T14:12:51.817 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:12:51 smithi069 ceph-mon[29472]: pgmap v82: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-04-14T14:12:51.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:12:51 smithi122 ceph-mon[25721]: pgmap v82: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-04-14T14:12:53.067 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:12:52 smithi069 ceph-mon[24818]: from='mgr.25105 172.21.15.69:0/4213893117' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T14:12:53.067 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:12:52 smithi069 ceph-mon[29472]: from='mgr.25105 172.21.15.69:0/4213893117' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T14:12:53.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:12:52 smithi122 ceph-mon[25721]: from='mgr.25105 172.21.15.69:0/4213893117' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T14:12:54.067 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:12:53 smithi069 ceph-mon[24818]: pgmap v83: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 556 B/s rd, 0 op/s 2024-04-14T14:12:54.067 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:12:53 smithi069 ceph-mon[29472]: pgmap v83: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 556 B/s rd, 0 op/s 2024-04-14T14:12:54.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:12:53 smithi122 ceph-mon[25721]: pgmap v83: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 556 B/s rd, 0 op/s 2024-04-14T14:12:55.317 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:12:54 smithi069 ceph-mon[24818]: pgmap v84: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-04-14T14:12:55.317 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:12:54 smithi069 ceph-mon[29472]: pgmap v84: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-04-14T14:12:55.345 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:12:54 smithi122 ceph-mon[25721]: pgmap v84: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-04-14T14:12:56.817 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:12:56 smithi069 ceph-mon[24818]: from='client.15285 -' entity='client.iscsi.foo.smithi069.yuumtr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T14:12:56.817 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 14 14:12:56 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-y[63038]: ::ffff:172.21.15.122 - - [14/Apr/2024:14:12:56] "GET /metrics HTTP/1.1" 200 37985 "" "Prometheus/2.33.4" 2024-04-14T14:12:56.817 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:12:56 smithi069 ceph-mon[29472]: from='client.15285 -' entity='client.iscsi.foo.smithi069.yuumtr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T14:12:56.844 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:12:56 smithi122 ceph-mon[25721]: from='client.15285 -' entity='client.iscsi.foo.smithi069.yuumtr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T14:12:57.817 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:12:57 smithi069 ceph-mon[24818]: pgmap v85: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-14T14:12:57.817 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:12:57 smithi069 ceph-mon[29472]: pgmap v85: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-14T14:12:57.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:12:57 smithi122 ceph-mon[25721]: pgmap v85: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-14T14:12:59.317 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:12:58 smithi069 ceph-mon[24818]: pgmap v86: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:12:59.317 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:12:58 smithi069 ceph-mon[29472]: pgmap v86: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:12:59.345 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:12:58 smithi122 ceph-mon[25721]: pgmap v86: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:13:02.817 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:13:02 smithi069 ceph-mon[29472]: pgmap v87: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:13:02.817 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:13:02 smithi069 ceph-mon[24818]: pgmap v87: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:13:02.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:13:02 smithi122 ceph-mon[25721]: pgmap v87: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:13:04.817 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:13:04 smithi069 ceph-mon[29472]: pgmap v88: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:13:04.817 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:13:04 smithi069 ceph-mon[24818]: pgmap v88: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:13:04.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:13:04 smithi122 ceph-mon[25721]: pgmap v88: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:13:06.817 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 14 14:13:06 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-y[63038]: ::ffff:172.21.15.122 - - [14/Apr/2024:14:13:06] "GET /metrics HTTP/1.1" 200 37985 "" "Prometheus/2.33.4" 2024-04-14T14:13:06.817 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:13:06 smithi069 ceph-mon[29472]: pgmap v89: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:13:06.817 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:13:06 smithi069 ceph-mon[29472]: from='client.15285 -' entity='client.iscsi.foo.smithi069.yuumtr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T14:13:06.818 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:13:06 smithi069 ceph-mon[24818]: pgmap v89: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:13:06.818 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:13:06 smithi069 ceph-mon[24818]: from='client.15285 -' entity='client.iscsi.foo.smithi069.yuumtr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T14:13:06.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:13:06 smithi122 ceph-mon[25721]: pgmap v89: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:13:06.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:13:06 smithi122 ceph-mon[25721]: from='client.15285 -' entity='client.iscsi.foo.smithi069.yuumtr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T14:13:08.817 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:13:08 smithi069 ceph-mon[29472]: pgmap v90: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:13:08.817 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:13:08 smithi069 ceph-mon[29472]: from='mgr.25105 172.21.15.69:0/4213893117' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T14:13:08.817 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:13:08 smithi069 ceph-mon[24818]: pgmap v90: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:13:08.817 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:13:08 smithi069 ceph-mon[24818]: from='mgr.25105 172.21.15.69:0/4213893117' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T14:13:08.844 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:13:08 smithi122 ceph-mon[25721]: pgmap v90: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:13:08.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:13:08 smithi122 ceph-mon[25721]: from='mgr.25105 172.21.15.69:0/4213893117' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T14:13:10.817 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:13:10 smithi069 ceph-mon[29472]: pgmap v91: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:13:10.817 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:13:10 smithi069 ceph-mon[24818]: pgmap v91: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:13:10.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:13:10 smithi122 ceph-mon[25721]: pgmap v91: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:13:12.817 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:13:12 smithi069 ceph-mon[24818]: pgmap v92: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:13:12.817 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:13:12 smithi069 ceph-mon[29472]: pgmap v92: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:13:12.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:13:12 smithi122 ceph-mon[25721]: pgmap v92: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:13:14.817 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:13:14 smithi069 ceph-mon[24818]: pgmap v93: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:13:14.817 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:13:14 smithi069 ceph-mon[29472]: pgmap v93: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:13:14.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:13:14 smithi122 ceph-mon[25721]: pgmap v93: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:13:16.817 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:13:16 smithi069 ceph-mon[24818]: pgmap v94: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:13:16.817 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:13:16 smithi069 ceph-mon[24818]: from='client.15285 -' entity='client.iscsi.foo.smithi069.yuumtr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T14:13:16.817 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 14 14:13:16 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-y[63038]: ::ffff:172.21.15.122 - - [14/Apr/2024:14:13:16] "GET /metrics HTTP/1.1" 200 37986 "" "Prometheus/2.33.4" 2024-04-14T14:13:16.817 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:13:16 smithi069 ceph-mon[29472]: pgmap v94: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:13:16.818 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:13:16 smithi069 ceph-mon[29472]: from='client.15285 -' entity='client.iscsi.foo.smithi069.yuumtr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T14:13:16.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:13:16 smithi122 ceph-mon[25721]: pgmap v94: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:13:16.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:13:16 smithi122 ceph-mon[25721]: from='client.15285 -' entity='client.iscsi.foo.smithi069.yuumtr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T14:13:18.817 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:13:18 smithi069 ceph-mon[29472]: pgmap v95: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:13:18.817 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:13:18 smithi069 ceph-mon[24818]: pgmap v95: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:13:18.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:13:18 smithi122 ceph-mon[25721]: pgmap v95: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:13:20.817 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:13:20 smithi069 ceph-mon[24818]: pgmap v96: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:13:20.817 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:13:20 smithi069 ceph-mon[29472]: pgmap v96: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:13:20.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:13:20 smithi122 ceph-mon[25721]: pgmap v96: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:13:22.817 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:13:22 smithi069 ceph-mon[24818]: pgmap v97: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:13:22.817 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:13:22 smithi069 ceph-mon[29472]: pgmap v97: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:13:22.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:13:22 smithi122 ceph-mon[25721]: pgmap v97: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:13:23.817 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:13:23 smithi069 ceph-mon[24818]: from='mgr.25105 172.21.15.69:0/4213893117' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T14:13:23.817 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:13:23 smithi069 ceph-mon[29472]: from='mgr.25105 172.21.15.69:0/4213893117' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T14:13:23.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:13:23 smithi122 ceph-mon[25721]: from='mgr.25105 172.21.15.69:0/4213893117' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T14:13:24.817 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:13:24 smithi069 ceph-mon[24818]: pgmap v98: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:13:24.817 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:13:24 smithi069 ceph-mon[29472]: pgmap v98: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:13:24.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:13:24 smithi122 ceph-mon[25721]: pgmap v98: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:13:26.344 DEBUG:teuthology.orchestra.run.smithi069:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid dc326fc2-fa65-11ee-bc8e-c7b262605968 -e sha1=a971fd5f5719038c70b817bafacd6d5400b1d888 -- bash -c 'ceph orch ps' 2024-04-14T14:13:26.567 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 14 14:13:26 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-y[63038]: ::ffff:172.21.15.122 - - [14/Apr/2024:14:13:26] "GET /metrics HTTP/1.1" 200 37984 "" "Prometheus/2.33.4" 2024-04-14T14:13:26.567 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:13:26 smithi069 ceph-mon[24818]: pgmap v99: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:13:26.567 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:13:26 smithi069 ceph-mon[24818]: from='client.15285 -' entity='client.iscsi.foo.smithi069.yuumtr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T14:13:26.567 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:13:26 smithi069 ceph-mon[29472]: pgmap v99: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:13:26.567 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:13:26 smithi069 ceph-mon[29472]: from='client.15285 -' entity='client.iscsi.foo.smithi069.yuumtr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T14:13:26.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:13:26 smithi122 ceph-mon[25721]: pgmap v99: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:13:26.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:13:26 smithi122 ceph-mon[25721]: from='client.15285 -' entity='client.iscsi.foo.smithi069.yuumtr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T14:13:28.370 INFO:teuthology.orchestra.run.smithi069.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-14T14:13:28.370 INFO:teuthology.orchestra.run.smithi069.stdout:alertmanager.a smithi069 *:9093,9094 running (9m) 2m ago 16m 23.5M - 0.23.0 ba2b418f427c e300fc36e997 2024-04-14T14:13:28.370 INFO:teuthology.orchestra.run.smithi069.stdout:grafana.a smithi122 *:3000 error 2m ago 16m - - 2024-04-14T14:13:28.370 INFO:teuthology.orchestra.run.smithi069.stdout:iscsi.foo.smithi069.yuumtr smithi069 running (2m) 2m ago 16m 46.0M - 3.5 e1d6a67b021e ef90f6d8fe13 2024-04-14T14:13:28.370 INFO:teuthology.orchestra.run.smithi069.stdout:mgr.x smithi122 *:8443,9283 running (12m) 2m ago 21m 449M - 19.0.0-2778-ga971fd5f b96c509eef84 813744c2b615 2024-04-14T14:13:28.370 INFO:teuthology.orchestra.run.smithi069.stdout:mgr.y smithi069 *:8443,9283,8765 running (5m) 2m ago 22m 500M - 19.0.0-2778-ga971fd5f b96c509eef84 7af3d29d0e5f 2024-04-14T14:13:28.371 INFO:teuthology.orchestra.run.smithi069.stdout:mon.a smithi069 running (23m) 2m ago 23m 62.0M 2048M 17.2.0 e1d6a67b021e d3c553f91913 2024-04-14T14:13:28.371 INFO:teuthology.orchestra.run.smithi069.stdout:mon.b smithi122 running (21m) 2m ago 21m 49.2M 2048M 17.2.0 e1d6a67b021e 3b766bfce4be 2024-04-14T14:13:28.371 INFO:teuthology.orchestra.run.smithi069.stdout:mon.c smithi069 running (21m) 2m ago 21m 45.9M 2048M 17.2.0 e1d6a67b021e 23c078b9d1e5 2024-04-14T14:13:28.371 INFO:teuthology.orchestra.run.smithi069.stdout:node-exporter.a smithi069 *:9100 running (9m) 2m ago 17m 17.8M - 1.3.1 1dbe0e931976 ca074142ab73 2024-04-14T14:13:28.371 INFO:teuthology.orchestra.run.smithi069.stdout:node-exporter.b smithi122 *:9100 running (8m) 2m ago 17m 18.8M - 1.3.1 1dbe0e931976 996115353a0b 2024-04-14T14:13:28.371 INFO:teuthology.orchestra.run.smithi069.stdout:osd.0 smithi069 running (21m) 2m ago 21m 56.0M 2992M 17.2.0 e1d6a67b021e 46654654b6b6 2024-04-14T14:13:28.371 INFO:teuthology.orchestra.run.smithi069.stdout:osd.1 smithi069 running (20m) 2m ago 20m 53.0M 2992M 17.2.0 e1d6a67b021e cdba54a59609 2024-04-14T14:13:28.371 INFO:teuthology.orchestra.run.smithi069.stdout:osd.2 smithi069 running (20m) 2m ago 20m 50.2M 2992M 17.2.0 e1d6a67b021e bc37a1a9cba0 2024-04-14T14:13:28.371 INFO:teuthology.orchestra.run.smithi069.stdout:osd.3 smithi069 running (19m) 2m ago 19m 54.7M 2992M 17.2.0 e1d6a67b021e 0925ba3da801 2024-04-14T14:13:28.371 INFO:teuthology.orchestra.run.smithi069.stdout:osd.4 smithi122 running (19m) 2m ago 19m 52.9M 3248M 17.2.0 e1d6a67b021e c00975caf771 2024-04-14T14:13:28.371 INFO:teuthology.orchestra.run.smithi069.stdout:osd.5 smithi122 running (19m) 2m ago 19m 53.9M 3248M 17.2.0 e1d6a67b021e 5f52d217ceb8 2024-04-14T14:13:28.371 INFO:teuthology.orchestra.run.smithi069.stdout:osd.6 smithi122 running (18m) 2m ago 18m 49.5M 3248M 17.2.0 e1d6a67b021e a2acfec61475 2024-04-14T14:13:28.371 INFO:teuthology.orchestra.run.smithi069.stdout:osd.7 smithi122 running (18m) 2m ago 18m 55.0M 3248M 17.2.0 e1d6a67b021e 62411cd06801 2024-04-14T14:13:28.371 INFO:teuthology.orchestra.run.smithi069.stdout:prometheus.a smithi122 *:9095 running (2m) 2m ago 17m 42.5M - 2.33.4 514e6a882f6e 9ebe35247d1a 2024-04-14T14:13:28.372 INFO:teuthology.orchestra.run.smithi069.stdout:rgw.foo.smithi069.wesedz smithi069 *:8000 running (16m) 2m ago 16m 75.6M - 17.2.0 e1d6a67b021e 16de491440f3 2024-04-14T14:13:28.372 INFO:teuthology.orchestra.run.smithi069.stdout:rgw.foo.smithi122.jiobyv smithi122 *:8000 running (16m) 2m ago 16m 72.9M - 17.2.0 e1d6a67b021e 6d16de445dba 2024-04-14T14:13:28.580 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:13:28 smithi069 ceph-mon[24818]: pgmap v100: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:13:28.580 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:13:28 smithi069 ceph-mon[29472]: pgmap v100: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:13:28.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:13:28 smithi122 ceph-mon[25721]: pgmap v100: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:13:29.012 DEBUG:teuthology.orchestra.run.smithi069:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid dc326fc2-fa65-11ee-bc8e-c7b262605968 -e sha1=a971fd5f5719038c70b817bafacd6d5400b1d888 -- bash -c 'ceph versions' 2024-04-14T14:13:29.649 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:13:29 smithi069 ceph-mon[24818]: from='client.15297 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T14:13:29.650 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:13:29 smithi069 ceph-mon[29472]: from='client.15297 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T14:13:29.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:13:29 smithi122 ceph-mon[25721]: from='client.15297 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T14:13:30.817 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:13:30 smithi069 ceph-mon[24818]: pgmap v101: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:13:30.817 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:13:30 smithi069 ceph-mon[29472]: pgmap v101: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:13:30.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:13:30 smithi122 ceph-mon[25721]: pgmap v101: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:13:31.072 INFO:teuthology.orchestra.run.smithi069.stdout:{ 2024-04-14T14:13:31.072 INFO:teuthology.orchestra.run.smithi069.stdout: "mon": { 2024-04-14T14:13:31.072 INFO:teuthology.orchestra.run.smithi069.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 3 2024-04-14T14:13:31.072 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-14T14:13:31.072 INFO:teuthology.orchestra.run.smithi069.stdout: "mgr": { 2024-04-14T14:13:31.072 INFO:teuthology.orchestra.run.smithi069.stdout: "ceph version 19.0.0-2778-ga971fd5f (a971fd5f5719038c70b817bafacd6d5400b1d888) squid (dev)": 2 2024-04-14T14:13:31.072 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-14T14:13:31.072 INFO:teuthology.orchestra.run.smithi069.stdout: "osd": { 2024-04-14T14:13:31.072 INFO:teuthology.orchestra.run.smithi069.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 8 2024-04-14T14:13:31.072 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-14T14:13:31.073 INFO:teuthology.orchestra.run.smithi069.stdout: "mds": {}, 2024-04-14T14:13:31.073 INFO:teuthology.orchestra.run.smithi069.stdout: "rgw": { 2024-04-14T14:13:31.073 INFO:teuthology.orchestra.run.smithi069.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-04-14T14:13:31.073 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-14T14:13:31.073 INFO:teuthology.orchestra.run.smithi069.stdout: "overall": { 2024-04-14T14:13:31.073 INFO:teuthology.orchestra.run.smithi069.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 13, 2024-04-14T14:13:31.073 INFO:teuthology.orchestra.run.smithi069.stdout: "ceph version 19.0.0-2778-ga971fd5f (a971fd5f5719038c70b817bafacd6d5400b1d888) squid (dev)": 2 2024-04-14T14:13:31.073 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-14T14:13:31.073 INFO:teuthology.orchestra.run.smithi069.stdout:} 2024-04-14T14:13:31.734 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:13:31 smithi069 ceph-mon[24818]: from='client.? 172.21.15.69:0/3083608238' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T14:13:31.734 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:13:31 smithi069 ceph-mon[29472]: from='client.? 172.21.15.69:0/3083608238' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T14:13:31.757 DEBUG:teuthology.orchestra.run.smithi069:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid dc326fc2-fa65-11ee-bc8e-c7b262605968 -e sha1=a971fd5f5719038c70b817bafacd6d5400b1d888 -- bash -c 'ceph -s' 2024-04-14T14:13:31.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:13:31 smithi122 ceph-mon[25721]: from='client.? 172.21.15.69:0/3083608238' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T14:13:32.724 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:13:32 smithi069 ceph-mon[24818]: pgmap v102: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:13:32.724 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:13:32 smithi069 ceph-mon[29472]: pgmap v102: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:13:32.846 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:13:32 smithi122 ceph-mon[25721]: pgmap v102: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:13:33.778 INFO:teuthology.orchestra.run.smithi069.stdout: cluster: 2024-04-14T14:13:33.779 INFO:teuthology.orchestra.run.smithi069.stdout: id: dc326fc2-fa65-11ee-bc8e-c7b262605968 2024-04-14T14:13:33.779 INFO:teuthology.orchestra.run.smithi069.stdout: health: HEALTH_WARN 2024-04-14T14:13:33.779 INFO:teuthology.orchestra.run.smithi069.stdout: 1 failed cephadm daemon(s) 2024-04-14T14:13:33.779 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-14T14:13:33.779 INFO:teuthology.orchestra.run.smithi069.stdout: services: 2024-04-14T14:13:33.779 INFO:teuthology.orchestra.run.smithi069.stdout: mon: 3 daemons, quorum a,c,b (age 21m) 2024-04-14T14:13:33.779 INFO:teuthology.orchestra.run.smithi069.stdout: mgr: y(active, since 3m), standbys: x 2024-04-14T14:13:33.779 INFO:teuthology.orchestra.run.smithi069.stdout: osd: 8 osds: 8 up (since 18m), 8 in (since 18m) 2024-04-14T14:13:33.779 INFO:teuthology.orchestra.run.smithi069.stdout: rgw: 2 daemons active (2 hosts, 1 zones) 2024-04-14T14:13:33.779 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-14T14:13:33.779 INFO:teuthology.orchestra.run.smithi069.stdout: data: 2024-04-14T14:13:33.779 INFO:teuthology.orchestra.run.smithi069.stdout: pools: 6 pools, 161 pgs 2024-04-14T14:13:33.779 INFO:teuthology.orchestra.run.smithi069.stdout: objects: 209 objects, 585 KiB 2024-04-14T14:13:33.780 INFO:teuthology.orchestra.run.smithi069.stdout: usage: 94 MiB used, 715 GiB / 715 GiB avail 2024-04-14T14:13:33.780 INFO:teuthology.orchestra.run.smithi069.stdout: pgs: 161 active+clean 2024-04-14T14:13:33.780 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-14T14:13:33.780 INFO:teuthology.orchestra.run.smithi069.stdout: io: 2024-04-14T14:13:33.780 INFO:teuthology.orchestra.run.smithi069.stdout: client: 853 B/s rd, 0 op/s rd, 0 op/s wr 2024-04-14T14:13:33.780 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-14T14:13:34.305 DEBUG:teuthology.orchestra.run.smithi069:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid dc326fc2-fa65-11ee-bc8e-c7b262605968 -e sha1=a971fd5f5719038c70b817bafacd6d5400b1d888 -- bash -c 'ceph health detail' 2024-04-14T14:13:34.607 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:13:34 smithi069 ceph-mon[24818]: pgmap v103: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:13:34.607 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:13:34 smithi069 ceph-mon[24818]: from='client.? 172.21.15.69:0/2026644841' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch 2024-04-14T14:13:34.608 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:13:34 smithi069 ceph-mon[29472]: pgmap v103: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:13:34.608 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:13:34 smithi069 ceph-mon[29472]: from='client.? 172.21.15.69:0/2026644841' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch 2024-04-14T14:13:34.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:13:34 smithi122 ceph-mon[25721]: pgmap v103: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:13:34.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:13:34 smithi122 ceph-mon[25721]: from='client.? 172.21.15.69:0/2026644841' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch 2024-04-14T14:13:36.417 INFO:teuthology.orchestra.run.smithi069.stdout:HEALTH_WARN 1 failed cephadm daemon(s) 2024-04-14T14:13:36.418 INFO:teuthology.orchestra.run.smithi069.stdout:[WRN] CEPHADM_FAILED_DAEMON: 1 failed cephadm daemon(s) 2024-04-14T14:13:36.418 INFO:teuthology.orchestra.run.smithi069.stdout: daemon grafana.a on smithi122 is in error state 2024-04-14T14:13:36.802 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 14 14:13:36 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-y[63038]: ::ffff:172.21.15.122 - - [14/Apr/2024:14:13:36] "GET /metrics HTTP/1.1" 200 37984 "" "Prometheus/2.33.4" 2024-04-14T14:13:36.802 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:13:36 smithi069 ceph-mon[24818]: pgmap v104: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:13:36.802 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:13:36 smithi069 ceph-mon[24818]: from='client.15285 -' entity='client.iscsi.foo.smithi069.yuumtr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T14:13:36.802 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:13:36 smithi069 ceph-mon[24818]: from='client.? 172.21.15.69:0/1386557222' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-14T14:13:36.802 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:13:36 smithi069 ceph-mon[29472]: pgmap v104: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:13:36.803 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:13:36 smithi069 ceph-mon[29472]: from='client.15285 -' entity='client.iscsi.foo.smithi069.yuumtr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T14:13:36.803 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:13:36 smithi069 ceph-mon[29472]: from='client.? 172.21.15.69:0/1386557222' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-14T14:13:36.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:13:36 smithi122 ceph-mon[25721]: pgmap v104: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:13:36.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:13:36 smithi122 ceph-mon[25721]: from='client.15285 -' entity='client.iscsi.foo.smithi069.yuumtr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T14:13:36.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:13:36 smithi122 ceph-mon[25721]: from='client.? 172.21.15.69:0/1386557222' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-14T14:13:37.119 DEBUG:teuthology.orchestra.run.smithi069:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid dc326fc2-fa65-11ee-bc8e-c7b262605968 -e sha1=a971fd5f5719038c70b817bafacd6d5400b1d888 -- bash -c 'ceph versions | jq -e '"'"'.mgr | length == 1'"'"'' 2024-04-14T14:13:38.669 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:13:38 smithi069 ceph-mon[29472]: pgmap v105: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:13:38.669 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:13:38 smithi069 ceph-mon[29472]: from='mgr.25105 172.21.15.69:0/4213893117' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T14:13:38.670 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:13:38 smithi069 ceph-mon[24818]: pgmap v105: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:13:38.670 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:13:38 smithi069 ceph-mon[24818]: from='mgr.25105 172.21.15.69:0/4213893117' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T14:13:38.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:13:38 smithi122 ceph-mon[25721]: pgmap v105: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:13:38.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:13:38 smithi122 ceph-mon[25721]: from='mgr.25105 172.21.15.69:0/4213893117' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T14:13:39.205 INFO:teuthology.orchestra.run.smithi069.stdout:true 2024-04-14T14:13:39.820 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:13:39 smithi069 ceph-mon[24818]: from='client.? 172.21.15.69:0/772954748' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T14:13:39.820 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:13:39 smithi069 ceph-mon[29472]: from='client.? 172.21.15.69:0/772954748' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T14:13:39.822 DEBUG:teuthology.orchestra.run.smithi069:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid dc326fc2-fa65-11ee-bc8e-c7b262605968 -e sha1=a971fd5f5719038c70b817bafacd6d5400b1d888 -- bash -c 'ceph mgr fail' 2024-04-14T14:13:39.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:13:39 smithi122 ceph-mon[25721]: from='client.? 172.21.15.69:0/772954748' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T14:13:40.817 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:13:40 smithi069 ceph-mon[24818]: pgmap v106: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:13:40.817 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:13:40 smithi069 ceph-mon[29472]: pgmap v106: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:13:40.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:13:40 smithi122 ceph-mon[25721]: pgmap v106: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:13:42.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:13:42 smithi122 ceph-mon[25721]: pgmap v107: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:13:42.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:13:42 smithi122 ceph-mon[25721]: from='client.? 172.21.15.69:0/164029678' entity='client.admin' cmd=[{"prefix": "mgr fail"}]: dispatch 2024-04-14T14:13:42.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:13:42 smithi122 ceph-mon[25721]: osdmap e103: 8 total, 8 up, 8 in 2024-04-14T14:13:42.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:13:42 smithi122 ceph-mon[25721]: Standby manager daemon y started 2024-04-14T14:13:42.845 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Apr 14 14:13:42 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-x[45487]: [14/Apr/2024:14:13:42] ENGINE Bus STOPPING 2024-04-14T14:13:42.845 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Apr 14 14:13:42 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-x[45487]: [14/Apr/2024:14:13:42] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-04-14T14:13:42.845 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Apr 14 14:13:42 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-x[45487]: [14/Apr/2024:14:13:42] ENGINE Bus STOPPED 2024-04-14T14:13:42.846 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Apr 14 14:13:42 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-x[45487]: [14/Apr/2024:14:13:42] ENGINE Bus STARTING 2024-04-14T14:13:42.846 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Apr 14 14:13:42 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-x[45487]: [14/Apr/2024:14:13:42] ENGINE Serving on http://:::9283 2024-04-14T14:13:42.846 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Apr 14 14:13:42 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-x[45487]: [14/Apr/2024:14:13:42] ENGINE Bus STARTED 2024-04-14T14:13:43.012 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:13:42 smithi069 ceph-mon[24818]: pgmap v107: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:13:43.012 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:13:42 smithi069 ceph-mon[24818]: from='client.? 172.21.15.69:0/164029678' entity='client.admin' cmd=[{"prefix": "mgr fail"}]: dispatch 2024-04-14T14:13:43.012 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:13:42 smithi069 ceph-mon[24818]: osdmap e103: 8 total, 8 up, 8 in 2024-04-14T14:13:43.012 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:13:42 smithi069 ceph-mon[24818]: Standby manager daemon y started 2024-04-14T14:13:43.013 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 14 14:13:42 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-y[63038]: 2024-04-14T14:13:42.604+0000 7fc7882c3700 -1 mgr handle_mgr_map I was active but no longer am 2024-04-14T14:13:43.013 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 14 14:13:42 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-y[63038]: ignoring --setuser ceph since I am not root 2024-04-14T14:13:43.013 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 14 14:13:42 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-y[63038]: ignoring --setgroup ceph since I am not root 2024-04-14T14:13:43.013 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 14 14:13:42 smithi069 ceph-mgr[63042]: -- 172.21.15.69:0/313248830 <== mon.1 v2:172.21.15.69:3301/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x5594d97465a0 con 0x5594d971b000 2024-04-14T14:13:43.013 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:13:42 smithi069 ceph-mon[29472]: pgmap v107: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:13:43.013 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:13:42 smithi069 ceph-mon[29472]: from='client.? 172.21.15.69:0/164029678' entity='client.admin' cmd=[{"prefix": "mgr fail"}]: dispatch 2024-04-14T14:13:43.013 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:13:42 smithi069 ceph-mon[29472]: osdmap e103: 8 total, 8 up, 8 in 2024-04-14T14:13:43.013 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:13:42 smithi069 ceph-mon[29472]: Standby manager daemon y started 2024-04-14T14:13:43.296 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 14 14:13:43 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-y[63038]: 2024-04-14T14:13:43.204+0000 7ff9f0467200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-14T14:13:43.323 DEBUG:teuthology.orchestra.run.smithi069:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid dc326fc2-fa65-11ee-bc8e-c7b262605968 -e sha1=a971fd5f5719038c70b817bafacd6d5400b1d888 -- bash -c 'sleep 180' 2024-04-14T14:13:43.566 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 14 14:13:43 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-y[63038]: 2024-04-14T14:13:43.305+0000 7ff9f0467200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-14T14:13:43.881 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:13:43 smithi122 ceph-mon[25721]: from='mgr.24938 172.21.15.122:0/87298362' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-14T14:13:43.882 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:13:43 smithi122 ceph-mon[25721]: from='mgr.24938 172.21.15.122:0/87298362' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-14T14:13:43.882 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:13:43 smithi122 ceph-mon[25721]: from='mgr.24938 172.21.15.122:0/87298362' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-14T14:13:43.882 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:13:43 smithi122 ceph-mon[25721]: from='client.? 172.21.15.69:0/164029678' entity='client.admin' cmd='[{"prefix": "mgr fail"}]': finished 2024-04-14T14:13:43.882 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:13:43 smithi122 ceph-mon[25721]: mgrmap e32: x(active, starting, since 0.840359s), standbys: y 2024-04-14T14:13:43.882 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:13:43 smithi122 ceph-mon[25721]: from='mgr.24938 172.21.15.122:0/87298362' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-04-14T14:13:43.882 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:13:43 smithi122 ceph-mon[25721]: from='mgr.24938 172.21.15.122:0/87298362' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-04-14T14:13:43.882 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:13:43 smithi122 ceph-mon[25721]: from='mgr.24938 172.21.15.122:0/87298362' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-14T14:13:43.882 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:13:43 smithi122 ceph-mon[25721]: from='mgr.24938 172.21.15.122:0/87298362' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-14T14:13:43.883 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:13:43 smithi122 ceph-mon[25721]: from='mgr.24938 172.21.15.122:0/87298362' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-14T14:13:43.883 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:13:43 smithi122 ceph-mon[25721]: from='mgr.24938 172.21.15.122:0/87298362' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-14T14:13:43.883 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:13:43 smithi122 ceph-mon[25721]: from='mgr.24938 172.21.15.122:0/87298362' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-14T14:13:43.883 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:13:43 smithi122 ceph-mon[25721]: from='mgr.24938 172.21.15.122:0/87298362' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-14T14:13:43.883 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:13:43 smithi122 ceph-mon[25721]: from='mgr.24938 172.21.15.122:0/87298362' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-14T14:13:43.883 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:13:43 smithi122 ceph-mon[25721]: from='mgr.24938 172.21.15.122:0/87298362' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-14T14:13:43.883 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:13:43 smithi122 ceph-mon[25721]: from='mgr.24938 172.21.15.122:0/87298362' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-04-14T14:13:43.883 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:13:43 smithi122 ceph-mon[25721]: from='mgr.24938 172.21.15.122:0/87298362' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-04-14T14:13:43.883 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:13:43 smithi122 ceph-mon[25721]: from='mgr.24938 172.21.15.122:0/87298362' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-04-14T14:13:43.883 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:13:43 smithi122 ceph-mon[25721]: Manager daemon x is now available 2024-04-14T14:13:43.883 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:13:43 smithi122 ceph-mon[25721]: from='mgr.24938 172.21.15.122:0/87298362' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T14:13:43.883 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:13:43 smithi122 ceph-mon[25721]: from='mgr.24938 172.21.15.122:0/87298362' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-04-14T14:13:43.883 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:13:43 smithi122 ceph-mon[25721]: from='mgr.24938 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-04-14T14:13:43.883 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:13:43 smithi122 ceph-mon[25721]: from='mgr.24938 172.21.15.122:0/87298362' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-04-14T14:13:43.884 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:13:43 smithi122 ceph-mon[25721]: from='mgr.24938 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-04-14T14:13:43.884 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:13:43 smithi122 ceph-mon[25721]: [14/Apr/2024:14:13:43] ENGINE Bus STARTING 2024-04-14T14:13:43.884 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:13:43 smithi122 ceph-mon[25721]: [14/Apr/2024:14:13:43] ENGINE Serving on https://172.21.15.122:7150 2024-04-14T14:13:43.884 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:13:43 smithi122 ceph-mon[25721]: [14/Apr/2024:14:13:43] ENGINE Serving on http://172.21.15.122:8765 2024-04-14T14:13:43.884 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:13:43 smithi122 ceph-mon[25721]: [14/Apr/2024:14:13:43] ENGINE Bus STARTED 2024-04-14T14:13:43.891 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:13:43 smithi069 ceph-mon[24818]: from='mgr.24938 172.21.15.122:0/87298362' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-14T14:13:43.891 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:13:43 smithi069 ceph-mon[24818]: from='mgr.24938 172.21.15.122:0/87298362' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-14T14:13:43.891 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:13:43 smithi069 ceph-mon[24818]: from='mgr.24938 172.21.15.122:0/87298362' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-14T14:13:43.891 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:13:43 smithi069 ceph-mon[24818]: from='client.? 172.21.15.69:0/164029678' entity='client.admin' cmd='[{"prefix": "mgr fail"}]': finished 2024-04-14T14:13:43.891 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:13:43 smithi069 ceph-mon[24818]: mgrmap e32: x(active, starting, since 0.840359s), standbys: y 2024-04-14T14:13:43.892 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:13:43 smithi069 ceph-mon[24818]: from='mgr.24938 172.21.15.122:0/87298362' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-04-14T14:13:43.892 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:13:43 smithi069 ceph-mon[24818]: from='mgr.24938 172.21.15.122:0/87298362' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-04-14T14:13:43.892 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:13:43 smithi069 ceph-mon[24818]: from='mgr.24938 172.21.15.122:0/87298362' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-14T14:13:43.892 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:13:43 smithi069 ceph-mon[24818]: from='mgr.24938 172.21.15.122:0/87298362' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-14T14:13:43.892 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:13:43 smithi069 ceph-mon[24818]: from='mgr.24938 172.21.15.122:0/87298362' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-14T14:13:43.892 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:13:43 smithi069 ceph-mon[24818]: from='mgr.24938 172.21.15.122:0/87298362' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-14T14:13:43.892 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:13:43 smithi069 ceph-mon[24818]: from='mgr.24938 172.21.15.122:0/87298362' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-14T14:13:43.892 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:13:43 smithi069 ceph-mon[24818]: from='mgr.24938 172.21.15.122:0/87298362' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-14T14:13:43.892 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:13:43 smithi069 ceph-mon[24818]: from='mgr.24938 172.21.15.122:0/87298362' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-14T14:13:43.892 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:13:43 smithi069 ceph-mon[24818]: from='mgr.24938 172.21.15.122:0/87298362' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-14T14:13:43.892 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:13:43 smithi069 ceph-mon[24818]: from='mgr.24938 172.21.15.122:0/87298362' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-04-14T14:13:43.892 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:13:43 smithi069 ceph-mon[24818]: from='mgr.24938 172.21.15.122:0/87298362' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-04-14T14:13:43.892 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:13:43 smithi069 ceph-mon[24818]: from='mgr.24938 172.21.15.122:0/87298362' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-04-14T14:13:43.892 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:13:43 smithi069 ceph-mon[24818]: Manager daemon x is now available 2024-04-14T14:13:43.892 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:13:43 smithi069 ceph-mon[24818]: from='mgr.24938 172.21.15.122:0/87298362' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T14:13:43.893 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:13:43 smithi069 ceph-mon[24818]: from='mgr.24938 172.21.15.122:0/87298362' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-04-14T14:13:43.893 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:13:43 smithi069 ceph-mon[24818]: from='mgr.24938 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-04-14T14:13:43.893 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:13:43 smithi069 ceph-mon[24818]: from='mgr.24938 172.21.15.122:0/87298362' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-04-14T14:13:43.893 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:13:43 smithi069 ceph-mon[24818]: from='mgr.24938 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-04-14T14:13:43.893 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:13:43 smithi069 ceph-mon[24818]: [14/Apr/2024:14:13:43] ENGINE Bus STARTING 2024-04-14T14:13:43.893 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:13:43 smithi069 ceph-mon[24818]: [14/Apr/2024:14:13:43] ENGINE Serving on https://172.21.15.122:7150 2024-04-14T14:13:43.893 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:13:43 smithi069 ceph-mon[24818]: [14/Apr/2024:14:13:43] ENGINE Serving on http://172.21.15.122:8765 2024-04-14T14:13:43.893 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:13:43 smithi069 ceph-mon[24818]: [14/Apr/2024:14:13:43] ENGINE Bus STARTED 2024-04-14T14:13:43.893 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:13:43 smithi069 ceph-mon[29472]: from='mgr.24938 172.21.15.122:0/87298362' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-14T14:13:43.893 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:13:43 smithi069 ceph-mon[29472]: from='mgr.24938 172.21.15.122:0/87298362' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-14T14:13:43.893 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:13:43 smithi069 ceph-mon[29472]: from='mgr.24938 172.21.15.122:0/87298362' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-14T14:13:43.893 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:13:43 smithi069 ceph-mon[29472]: from='client.? 172.21.15.69:0/164029678' entity='client.admin' cmd='[{"prefix": "mgr fail"}]': finished 2024-04-14T14:13:43.894 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:13:43 smithi069 ceph-mon[29472]: mgrmap e32: x(active, starting, since 0.840359s), standbys: y 2024-04-14T14:13:43.894 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:13:43 smithi069 ceph-mon[29472]: from='mgr.24938 172.21.15.122:0/87298362' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-04-14T14:13:43.894 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:13:43 smithi069 ceph-mon[29472]: from='mgr.24938 172.21.15.122:0/87298362' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-04-14T14:13:43.894 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:13:43 smithi069 ceph-mon[29472]: from='mgr.24938 172.21.15.122:0/87298362' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-14T14:13:43.894 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:13:43 smithi069 ceph-mon[29472]: from='mgr.24938 172.21.15.122:0/87298362' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-14T14:13:43.894 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:13:43 smithi069 ceph-mon[29472]: from='mgr.24938 172.21.15.122:0/87298362' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-14T14:13:43.894 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:13:43 smithi069 ceph-mon[29472]: from='mgr.24938 172.21.15.122:0/87298362' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-14T14:13:43.894 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:13:43 smithi069 ceph-mon[29472]: from='mgr.24938 172.21.15.122:0/87298362' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-14T14:13:43.894 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:13:43 smithi069 ceph-mon[29472]: from='mgr.24938 172.21.15.122:0/87298362' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-14T14:13:43.894 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:13:43 smithi069 ceph-mon[29472]: from='mgr.24938 172.21.15.122:0/87298362' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-14T14:13:43.894 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:13:43 smithi069 ceph-mon[29472]: from='mgr.24938 172.21.15.122:0/87298362' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-14T14:13:43.894 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:13:43 smithi069 ceph-mon[29472]: from='mgr.24938 172.21.15.122:0/87298362' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-04-14T14:13:43.894 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:13:43 smithi069 ceph-mon[29472]: from='mgr.24938 172.21.15.122:0/87298362' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-04-14T14:13:43.894 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:13:43 smithi069 ceph-mon[29472]: from='mgr.24938 172.21.15.122:0/87298362' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-04-14T14:13:43.894 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:13:43 smithi069 ceph-mon[29472]: Manager daemon x is now available 2024-04-14T14:13:43.895 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:13:43 smithi069 ceph-mon[29472]: from='mgr.24938 172.21.15.122:0/87298362' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T14:13:43.895 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:13:43 smithi069 ceph-mon[29472]: from='mgr.24938 172.21.15.122:0/87298362' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-04-14T14:13:43.895 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:13:43 smithi069 ceph-mon[29472]: from='mgr.24938 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-04-14T14:13:43.895 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:13:43 smithi069 ceph-mon[29472]: from='mgr.24938 172.21.15.122:0/87298362' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-04-14T14:13:43.895 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:13:43 smithi069 ceph-mon[29472]: from='mgr.24938 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-04-14T14:13:43.895 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:13:43 smithi069 ceph-mon[29472]: [14/Apr/2024:14:13:43] ENGINE Bus STARTING 2024-04-14T14:13:43.895 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:13:43 smithi069 ceph-mon[29472]: [14/Apr/2024:14:13:43] ENGINE Serving on https://172.21.15.122:7150 2024-04-14T14:13:43.895 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:13:43 smithi069 ceph-mon[29472]: [14/Apr/2024:14:13:43] ENGINE Serving on http://172.21.15.122:8765 2024-04-14T14:13:43.895 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:13:43 smithi069 ceph-mon[29472]: [14/Apr/2024:14:13:43] ENGINE Bus STARTED 2024-04-14T14:13:44.272 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 14 14:13:44 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-y[63038]: 2024-04-14T14:13:44.028+0000 7ff9f0467200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-14T14:13:44.273 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 14 14:13:44 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-y[63038]: 2024-04-14T14:13:44.122+0000 7ff9f0467200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-14T14:13:44.273 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 14 14:13:44 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-y[63038]: 2024-04-14T14:13:44.270+0000 7ff9f0467200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-14T14:13:44.567 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 14 14:13:44 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-y[63038]: 2024-04-14T14:13:44.368+0000 7ff9f0467200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-14T14:13:44.567 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 14 14:13:44 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-y[63038]: 2024-04-14T14:13:44.451+0000 7ff9f0467200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-14T14:13:44.675 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:13:44 smithi122 ceph-mon[25721]: mgrmap e33: x(active, since 1.846s), standbys: y 2024-04-14T14:13:44.675 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:13:44 smithi122 ceph-mon[25721]: pgmap v3: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail 2024-04-14T14:13:44.845 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:13:44 smithi069 ceph-mon[24818]: mgrmap e33: x(active, since 1.846s), standbys: y 2024-04-14T14:13:44.845 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:13:44 smithi069 ceph-mon[24818]: pgmap v3: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail 2024-04-14T14:13:44.846 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 14 14:13:44 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-y[63038]: 2024-04-14T14:13:44.731+0000 7ff9f0467200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-14T14:13:44.846 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 14 14:13:44 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-y[63038]: 2024-04-14T14:13:44.843+0000 7ff9f0467200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-14T14:13:44.846 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:13:44 smithi069 ceph-mon[29472]: mgrmap e33: x(active, since 1.846s), standbys: y 2024-04-14T14:13:44.846 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:13:44 smithi069 ceph-mon[29472]: pgmap v3: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail 2024-04-14T14:13:45.215 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 14 14:13:44 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-y[63038]: 2024-04-14T14:13:44.934+0000 7ff9f0467200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-14T14:13:45.215 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 14 14:13:45 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-y[63038]: 2024-04-14T14:13:45.033+0000 7ff9f0467200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-14T14:13:45.567 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 14 14:13:45 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-y[63038]: 2024-04-14T14:13:45.212+0000 7ff9f0467200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-14T14:13:45.963 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:13:45 smithi069 ceph-mon[24818]: Health check cleared: CEPHADM_FAILED_DAEMON (was: 1 failed cephadm daemon(s)) 2024-04-14T14:13:45.963 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:13:45 smithi069 ceph-mon[24818]: Cluster is now healthy 2024-04-14T14:13:45.963 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:13:45 smithi069 ceph-mon[24818]: pgmap v4: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail 2024-04-14T14:13:45.963 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:13:45 smithi069 ceph-mon[24818]: mgrmap e34: x(active, since 2s), standbys: y 2024-04-14T14:13:45.963 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:13:45 smithi069 ceph-mon[24818]: from='client.15285 -' entity='client.iscsi.foo.smithi069.yuumtr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T14:13:45.963 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:13:45 smithi069 ceph-mon[29472]: Health check cleared: CEPHADM_FAILED_DAEMON (was: 1 failed cephadm daemon(s)) 2024-04-14T14:13:45.963 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:13:45 smithi069 ceph-mon[29472]: Cluster is now healthy 2024-04-14T14:13:45.963 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:13:45 smithi069 ceph-mon[29472]: pgmap v4: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail 2024-04-14T14:13:45.963 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:13:45 smithi069 ceph-mon[29472]: mgrmap e34: x(active, since 2s), standbys: y 2024-04-14T14:13:45.964 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:13:45 smithi069 ceph-mon[29472]: from='client.15285 -' entity='client.iscsi.foo.smithi069.yuumtr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T14:13:46.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:13:45 smithi122 ceph-mon[25721]: Health check cleared: CEPHADM_FAILED_DAEMON (was: 1 failed cephadm daemon(s)) 2024-04-14T14:13:46.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:13:45 smithi122 ceph-mon[25721]: Cluster is now healthy 2024-04-14T14:13:46.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:13:45 smithi122 ceph-mon[25721]: pgmap v4: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail 2024-04-14T14:13:46.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:13:45 smithi122 ceph-mon[25721]: mgrmap e34: x(active, since 2s), standbys: y 2024-04-14T14:13:46.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:13:45 smithi122 ceph-mon[25721]: from='client.15285 -' entity='client.iscsi.foo.smithi069.yuumtr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T14:13:46.567 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 14 14:13:46 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-y[63038]: 2024-04-14T14:13:46.230+0000 7ff9f0467200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-14T14:13:46.567 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 14 14:13:46 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-y[63038]: 2024-04-14T14:13:46.368+0000 7ff9f0467200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-14T14:13:46.943 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 14 14:13:46 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-y[63038]: 2024-04-14T14:13:46.759+0000 7ff9f0467200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-14T14:13:46.943 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 14 14:13:46 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-y[63038]: 2024-04-14T14:13:46.856+0000 7ff9f0467200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-14T14:13:47.270 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 14 14:13:47 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-y[63038]: 2024-04-14T14:13:47.072+0000 7ff9f0467200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-14T14:13:47.566 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 14 14:13:47 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-y[63038]: 2024-04-14T14:13:47.268+0000 7ff9f0467200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-14T14:13:47.854 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 14 14:13:47 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-y[63038]: 2024-04-14T14:13:47.583+0000 7ff9f0467200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-14T14:13:47.854 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:13:47 smithi069 ceph-mon[24818]: pgmap v5: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail 2024-04-14T14:13:47.855 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:13:47 smithi069 ceph-mon[24818]: mgrmap e35: x(active, since 4s), standbys: y 2024-04-14T14:13:47.855 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:13:47 smithi069 ceph-mon[24818]: from='mgr.24938 ' entity='mgr.x' 2024-04-14T14:13:47.855 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:13:47 smithi069 ceph-mon[24818]: from='mgr.24938 ' entity='mgr.x' 2024-04-14T14:13:47.855 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:13:47 smithi069 ceph-mon[29472]: pgmap v5: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail 2024-04-14T14:13:47.855 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:13:47 smithi069 ceph-mon[29472]: mgrmap e35: x(active, since 4s), standbys: y 2024-04-14T14:13:47.855 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:13:47 smithi069 ceph-mon[29472]: from='mgr.24938 ' entity='mgr.x' 2024-04-14T14:13:47.855 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:13:47 smithi069 ceph-mon[29472]: from='mgr.24938 ' entity='mgr.x' 2024-04-14T14:13:48.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:13:47 smithi122 ceph-mon[25721]: pgmap v5: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail 2024-04-14T14:13:48.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:13:47 smithi122 ceph-mon[25721]: mgrmap e35: x(active, since 4s), standbys: y 2024-04-14T14:13:48.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:13:47 smithi122 ceph-mon[25721]: from='mgr.24938 ' entity='mgr.x' 2024-04-14T14:13:48.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:13:47 smithi122 ceph-mon[25721]: from='mgr.24938 ' entity='mgr.x' 2024-04-14T14:13:48.105 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 14 14:13:47 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-y[63038]: 2024-04-14T14:13:47.852+0000 7ff9f0467200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-14T14:13:48.464 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 14 14:13:48 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-y[63038]: 2024-04-14T14:13:48.103+0000 7ff9f0467200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-14T14:13:49.067 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 14 14:13:48 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-y[63038]: 2024-04-14T14:13:48.982+0000 7ff9f0467200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-14T14:13:49.067 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:13:48 smithi069 ceph-mon[24818]: from='mgr.24938 ' entity='mgr.x' 2024-04-14T14:13:49.067 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:13:48 smithi069 ceph-mon[24818]: from='mgr.24938 ' entity='mgr.x' 2024-04-14T14:13:49.067 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:13:48 smithi069 ceph-mon[29472]: from='mgr.24938 ' entity='mgr.x' 2024-04-14T14:13:49.068 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:13:48 smithi069 ceph-mon[29472]: from='mgr.24938 ' entity='mgr.x' 2024-04-14T14:13:49.345 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:13:48 smithi122 ceph-mon[25721]: from='mgr.24938 ' entity='mgr.x' 2024-04-14T14:13:49.345 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:13:48 smithi122 ceph-mon[25721]: from='mgr.24938 ' entity='mgr.x' 2024-04-14T14:13:49.939 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 14 14:13:49 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-y[63038]: 2024-04-14T14:13:49.591+0000 7ff9f0467200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-14T14:13:49.939 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 14 14:13:49 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-y[63038]: 2024-04-14T14:13:49.697+0000 7ff9f0467200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-14T14:13:50.317 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:13:49 smithi069 ceph-mon[24818]: pgmap v6: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail 2024-04-14T14:13:50.317 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:13:49 smithi069 ceph-mon[24818]: from='mgr.24938 ' entity='mgr.x' 2024-04-14T14:13:50.317 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:13:49 smithi069 ceph-mon[24818]: from='mgr.24938 ' entity='mgr.x' 2024-04-14T14:13:50.317 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:13:49 smithi069 ceph-mon[24818]: Standby manager daemon y restarted 2024-04-14T14:13:50.317 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:13:49 smithi069 ceph-mon[24818]: Standby manager daemon y started 2024-04-14T14:13:50.317 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:13:49 smithi069 ceph-mon[24818]: from='mgr.? 172.21.15.69:0/1161558199' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-04-14T14:13:50.317 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:13:49 smithi069 ceph-mon[24818]: from='mgr.? 172.21.15.69:0/1161558199' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-14T14:13:50.318 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:13:49 smithi069 ceph-mon[24818]: from='mgr.? 172.21.15.69:0/1161558199' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-04-14T14:13:50.318 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:13:49 smithi069 ceph-mon[24818]: from='mgr.? 172.21.15.69:0/1161558199' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-14T14:13:50.318 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:13:49 smithi069 ceph-mon[29472]: pgmap v6: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail 2024-04-14T14:13:50.318 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:13:49 smithi069 ceph-mon[29472]: from='mgr.24938 ' entity='mgr.x' 2024-04-14T14:13:50.318 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:13:49 smithi069 ceph-mon[29472]: from='mgr.24938 ' entity='mgr.x' 2024-04-14T14:13:50.318 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:13:49 smithi069 ceph-mon[29472]: Standby manager daemon y restarted 2024-04-14T14:13:50.318 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:13:49 smithi069 ceph-mon[29472]: Standby manager daemon y started 2024-04-14T14:13:50.318 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:13:49 smithi069 ceph-mon[29472]: from='mgr.? 172.21.15.69:0/1161558199' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-04-14T14:13:50.318 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:13:49 smithi069 ceph-mon[29472]: from='mgr.? 172.21.15.69:0/1161558199' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-14T14:13:50.319 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:13:49 smithi069 ceph-mon[29472]: from='mgr.? 172.21.15.69:0/1161558199' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-04-14T14:13:50.319 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:13:49 smithi069 ceph-mon[29472]: from='mgr.? 172.21.15.69:0/1161558199' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-14T14:13:50.319 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 14 14:13:49 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-y[63038]: 2024-04-14T14:13:49.937+0000 7ff9f0467200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-14T14:13:50.319 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 14 14:13:49 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-y[63038]: [14/Apr/2024:14:13:49] ENGINE Bus STARTING 2024-04-14T14:13:50.319 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 14 14:13:50 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-y[63038]: CherryPy Checker: 2024-04-14T14:13:50.319 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 14 14:13:50 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-y[63038]: The Application mounted at '' has an empty config. 2024-04-14T14:13:50.319 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 14 14:13:50 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-y[63038]: 2024-04-14T14:13:50.319 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 14 14:13:50 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-y[63038]: [14/Apr/2024:14:13:50] ENGINE Serving on http://:::9283 2024-04-14T14:13:50.319 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 14 14:13:50 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-y[63038]: [14/Apr/2024:14:13:50] ENGINE Bus STARTED 2024-04-14T14:13:50.345 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:13:49 smithi122 ceph-mon[25721]: pgmap v6: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail 2024-04-14T14:13:50.345 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:13:49 smithi122 ceph-mon[25721]: from='mgr.24938 ' entity='mgr.x' 2024-04-14T14:13:50.345 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:13:49 smithi122 ceph-mon[25721]: from='mgr.24938 ' entity='mgr.x' 2024-04-14T14:13:50.345 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:13:49 smithi122 ceph-mon[25721]: Standby manager daemon y restarted 2024-04-14T14:13:50.345 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:13:49 smithi122 ceph-mon[25721]: Standby manager daemon y started 2024-04-14T14:13:50.345 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:13:49 smithi122 ceph-mon[25721]: from='mgr.? 172.21.15.69:0/1161558199' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-04-14T14:13:50.345 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:13:49 smithi122 ceph-mon[25721]: from='mgr.? 172.21.15.69:0/1161558199' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-14T14:13:50.345 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:13:49 smithi122 ceph-mon[25721]: from='mgr.? 172.21.15.69:0/1161558199' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-04-14T14:13:50.345 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:13:49 smithi122 ceph-mon[25721]: from='mgr.? 172.21.15.69:0/1161558199' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-14T14:13:51.317 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:13:50 smithi069 ceph-mon[24818]: mgrmap e36: x(active, since 8s), standbys: y 2024-04-14T14:13:51.317 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:13:50 smithi069 ceph-mon[24818]: from='mgr.24938 ' entity='mgr.x' 2024-04-14T14:13:51.317 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:13:50 smithi069 ceph-mon[24818]: from='mgr.24938 ' entity='mgr.x' 2024-04-14T14:13:51.317 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:13:50 smithi069 ceph-mon[24818]: from='mgr.24938 172.21.15.122:0/87298362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T14:13:51.317 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:13:50 smithi069 ceph-mon[24818]: from='mgr.24938 172.21.15.122:0/87298362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-14T14:13:51.317 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:13:50 smithi069 ceph-mon[24818]: Updating smithi069:/etc/ceph/ceph.conf 2024-04-14T14:13:51.317 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:13:50 smithi069 ceph-mon[24818]: Updating smithi122:/etc/ceph/ceph.conf 2024-04-14T14:13:51.318 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:13:50 smithi069 ceph-mon[24818]: Updating smithi069:/var/lib/ceph/dc326fc2-fa65-11ee-bc8e-c7b262605968/config/ceph.conf 2024-04-14T14:13:51.318 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:13:50 smithi069 ceph-mon[24818]: Updating smithi122:/var/lib/ceph/dc326fc2-fa65-11ee-bc8e-c7b262605968/config/ceph.conf 2024-04-14T14:13:51.318 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:13:50 smithi069 ceph-mon[29472]: mgrmap e36: x(active, since 8s), standbys: y 2024-04-14T14:13:51.318 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:13:50 smithi069 ceph-mon[29472]: from='mgr.24938 ' entity='mgr.x' 2024-04-14T14:13:51.318 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:13:50 smithi069 ceph-mon[29472]: from='mgr.24938 ' entity='mgr.x' 2024-04-14T14:13:51.318 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:13:50 smithi069 ceph-mon[29472]: from='mgr.24938 172.21.15.122:0/87298362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T14:13:51.318 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:13:50 smithi069 ceph-mon[29472]: from='mgr.24938 172.21.15.122:0/87298362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-14T14:13:51.318 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:13:50 smithi069 ceph-mon[29472]: Updating smithi069:/etc/ceph/ceph.conf 2024-04-14T14:13:51.318 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:13:50 smithi069 ceph-mon[29472]: Updating smithi122:/etc/ceph/ceph.conf 2024-04-14T14:13:51.318 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:13:50 smithi069 ceph-mon[29472]: Updating smithi069:/var/lib/ceph/dc326fc2-fa65-11ee-bc8e-c7b262605968/config/ceph.conf 2024-04-14T14:13:51.318 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:13:50 smithi069 ceph-mon[29472]: Updating smithi122:/var/lib/ceph/dc326fc2-fa65-11ee-bc8e-c7b262605968/config/ceph.conf 2024-04-14T14:13:51.345 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:13:50 smithi122 ceph-mon[25721]: mgrmap e36: x(active, since 8s), standbys: y 2024-04-14T14:13:51.345 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:13:50 smithi122 ceph-mon[25721]: from='mgr.24938 ' entity='mgr.x' 2024-04-14T14:13:51.345 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:13:50 smithi122 ceph-mon[25721]: from='mgr.24938 ' entity='mgr.x' 2024-04-14T14:13:51.345 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:13:50 smithi122 ceph-mon[25721]: from='mgr.24938 172.21.15.122:0/87298362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T14:13:51.345 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:13:50 smithi122 ceph-mon[25721]: from='mgr.24938 172.21.15.122:0/87298362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-14T14:13:51.345 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:13:50 smithi122 ceph-mon[25721]: Updating smithi069:/etc/ceph/ceph.conf 2024-04-14T14:13:51.345 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:13:50 smithi122 ceph-mon[25721]: Updating smithi122:/etc/ceph/ceph.conf 2024-04-14T14:13:51.345 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:13:50 smithi122 ceph-mon[25721]: Updating smithi069:/var/lib/ceph/dc326fc2-fa65-11ee-bc8e-c7b262605968/config/ceph.conf 2024-04-14T14:13:51.345 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:13:50 smithi122 ceph-mon[25721]: Updating smithi122:/var/lib/ceph/dc326fc2-fa65-11ee-bc8e-c7b262605968/config/ceph.conf 2024-04-14T14:13:52.567 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:13:52 smithi069 ceph-mon[24818]: pgmap v7: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 30 KiB/s rd, 0 B/s wr, 17 op/s 2024-04-14T14:13:52.567 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:13:52 smithi069 ceph-mon[24818]: Updating smithi069:/etc/ceph/ceph.client.admin.keyring 2024-04-14T14:13:52.567 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:13:52 smithi069 ceph-mon[24818]: Updating smithi122:/etc/ceph/ceph.client.admin.keyring 2024-04-14T14:13:52.568 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:13:52 smithi069 ceph-mon[24818]: Updating smithi069:/var/lib/ceph/dc326fc2-fa65-11ee-bc8e-c7b262605968/config/ceph.client.admin.keyring 2024-04-14T14:13:52.568 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:13:52 smithi069 ceph-mon[24818]: Updating smithi122:/var/lib/ceph/dc326fc2-fa65-11ee-bc8e-c7b262605968/config/ceph.client.admin.keyring 2024-04-14T14:13:52.568 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:13:52 smithi069 ceph-mon[24818]: from='mgr.24938 ' entity='mgr.x' 2024-04-14T14:13:52.568 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:13:52 smithi069 ceph-mon[24818]: from='mgr.24938 ' entity='mgr.x' 2024-04-14T14:13:52.568 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:13:52 smithi069 ceph-mon[24818]: from='mgr.24938 ' entity='mgr.x' 2024-04-14T14:13:52.568 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:13:52 smithi069 ceph-mon[24818]: from='mgr.24938 ' entity='mgr.x' 2024-04-14T14:13:52.568 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:13:52 smithi069 ceph-mon[24818]: pgmap v8: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 28 KiB/s rd, 0 B/s wr, 15 op/s 2024-04-14T14:13:52.568 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:13:52 smithi069 ceph-mon[24818]: from='mgr.24938 ' entity='mgr.x' 2024-04-14T14:13:52.568 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:13:52 smithi069 ceph-mon[24818]: Reconfiguring iscsi.foo.smithi069.yuumtr (dependencies changed)... 2024-04-14T14:13:52.568 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:13:52 smithi069 ceph-mon[24818]: from='mgr.24938 172.21.15.122:0/87298362' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi069.yuumtr", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-04-14T14:13:52.568 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:13:52 smithi069 ceph-mon[24818]: from='mgr.24938 ' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi069.yuumtr", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-04-14T14:13:52.568 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:13:52 smithi069 ceph-mon[24818]: from='mgr.24938 172.21.15.122:0/87298362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T14:13:52.568 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:13:52 smithi069 ceph-mon[24818]: Reconfiguring daemon iscsi.foo.smithi069.yuumtr on smithi069 2024-04-14T14:13:52.568 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:13:52 smithi069 ceph-mon[24818]: Health check failed: 1 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON) 2024-04-14T14:13:52.569 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:13:52 smithi069 ceph-mon[29472]: pgmap v7: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 30 KiB/s rd, 0 B/s wr, 17 op/s 2024-04-14T14:13:52.569 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:13:52 smithi069 ceph-mon[29472]: Updating smithi069:/etc/ceph/ceph.client.admin.keyring 2024-04-14T14:13:52.569 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:13:52 smithi069 ceph-mon[29472]: Updating smithi122:/etc/ceph/ceph.client.admin.keyring 2024-04-14T14:13:52.569 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:13:52 smithi069 ceph-mon[29472]: Updating smithi069:/var/lib/ceph/dc326fc2-fa65-11ee-bc8e-c7b262605968/config/ceph.client.admin.keyring 2024-04-14T14:13:52.569 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:13:52 smithi069 ceph-mon[29472]: Updating smithi122:/var/lib/ceph/dc326fc2-fa65-11ee-bc8e-c7b262605968/config/ceph.client.admin.keyring 2024-04-14T14:13:52.569 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:13:52 smithi069 ceph-mon[29472]: from='mgr.24938 ' entity='mgr.x' 2024-04-14T14:13:52.569 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:13:52 smithi069 ceph-mon[29472]: from='mgr.24938 ' entity='mgr.x' 2024-04-14T14:13:52.569 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:13:52 smithi069 ceph-mon[29472]: from='mgr.24938 ' entity='mgr.x' 2024-04-14T14:13:52.569 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:13:52 smithi069 ceph-mon[29472]: from='mgr.24938 ' entity='mgr.x' 2024-04-14T14:13:52.569 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:13:52 smithi069 ceph-mon[29472]: pgmap v8: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 28 KiB/s rd, 0 B/s wr, 15 op/s 2024-04-14T14:13:52.569 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:13:52 smithi069 ceph-mon[29472]: from='mgr.24938 ' entity='mgr.x' 2024-04-14T14:13:52.569 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:13:52 smithi069 ceph-mon[29472]: Reconfiguring iscsi.foo.smithi069.yuumtr (dependencies changed)... 2024-04-14T14:13:52.570 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:13:52 smithi069 ceph-mon[29472]: from='mgr.24938 172.21.15.122:0/87298362' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi069.yuumtr", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-04-14T14:13:52.570 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:13:52 smithi069 ceph-mon[29472]: from='mgr.24938 ' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi069.yuumtr", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-04-14T14:13:52.570 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:13:52 smithi069 ceph-mon[29472]: from='mgr.24938 172.21.15.122:0/87298362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T14:13:52.570 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:13:52 smithi069 ceph-mon[29472]: Reconfiguring daemon iscsi.foo.smithi069.yuumtr on smithi069 2024-04-14T14:13:52.570 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:13:52 smithi069 ceph-mon[29472]: Health check failed: 1 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON) 2024-04-14T14:13:52.595 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:13:52 smithi122 ceph-mon[25721]: pgmap v7: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 30 KiB/s rd, 0 B/s wr, 17 op/s 2024-04-14T14:13:52.595 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:13:52 smithi122 ceph-mon[25721]: Updating smithi069:/etc/ceph/ceph.client.admin.keyring 2024-04-14T14:13:52.595 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:13:52 smithi122 ceph-mon[25721]: Updating smithi122:/etc/ceph/ceph.client.admin.keyring 2024-04-14T14:13:52.595 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:13:52 smithi122 ceph-mon[25721]: Updating smithi069:/var/lib/ceph/dc326fc2-fa65-11ee-bc8e-c7b262605968/config/ceph.client.admin.keyring 2024-04-14T14:13:52.595 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:13:52 smithi122 ceph-mon[25721]: Updating smithi122:/var/lib/ceph/dc326fc2-fa65-11ee-bc8e-c7b262605968/config/ceph.client.admin.keyring 2024-04-14T14:13:52.595 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:13:52 smithi122 ceph-mon[25721]: from='mgr.24938 ' entity='mgr.x' 2024-04-14T14:13:52.595 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:13:52 smithi122 ceph-mon[25721]: from='mgr.24938 ' entity='mgr.x' 2024-04-14T14:13:52.595 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:13:52 smithi122 ceph-mon[25721]: from='mgr.24938 ' entity='mgr.x' 2024-04-14T14:13:52.596 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:13:52 smithi122 ceph-mon[25721]: from='mgr.24938 ' entity='mgr.x' 2024-04-14T14:13:52.596 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:13:52 smithi122 ceph-mon[25721]: pgmap v8: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 28 KiB/s rd, 0 B/s wr, 15 op/s 2024-04-14T14:13:52.596 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:13:52 smithi122 ceph-mon[25721]: from='mgr.24938 ' entity='mgr.x' 2024-04-14T14:13:52.596 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:13:52 smithi122 ceph-mon[25721]: Reconfiguring iscsi.foo.smithi069.yuumtr (dependencies changed)... 2024-04-14T14:13:52.596 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:13:52 smithi122 ceph-mon[25721]: from='mgr.24938 172.21.15.122:0/87298362' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi069.yuumtr", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-04-14T14:13:52.596 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:13:52 smithi122 ceph-mon[25721]: from='mgr.24938 ' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi069.yuumtr", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-04-14T14:13:52.596 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:13:52 smithi122 ceph-mon[25721]: from='mgr.24938 172.21.15.122:0/87298362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T14:13:52.596 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:13:52 smithi122 ceph-mon[25721]: Reconfiguring daemon iscsi.foo.smithi069.yuumtr on smithi069 2024-04-14T14:13:52.596 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:13:52 smithi122 ceph-mon[25721]: Health check failed: 1 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON) 2024-04-14T14:13:54.426 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:13:54 smithi069 ceph-mon[24818]: pgmap v9: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 22 KiB/s rd, 0 B/s wr, 12 op/s 2024-04-14T14:13:54.426 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:13:54 smithi069 ceph-mon[29472]: pgmap v9: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 22 KiB/s rd, 0 B/s wr, 12 op/s 2024-04-14T14:13:54.530 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:13:54 smithi122 ceph-mon[25721]: pgmap v9: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 22 KiB/s rd, 0 B/s wr, 12 op/s 2024-04-14T14:13:54.803 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Apr 14 14:13:54 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-x[45487]: 2024-04-14T14:13:54.527+0000 7f95977c4700 -1 log_channel(cephadm) log [ERR] : cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi069.yuumtr ... 2024-04-14T14:13:54.803 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Apr 14 14:13:54 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-x[45487]: Creating ceph-iscsi config... 2024-04-14T14:13:54.804 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Apr 14 14:13:54 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-x[45487]: Write file: /var/lib/ceph/dc326fc2-fa65-11ee-bc8e-c7b262605968/iscsi.foo.smithi069.yuumtr/iscsi-gateway.cfg 2024-04-14T14:13:54.804 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Apr 14 14:13:54 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-x[45487]: Write file: /var/lib/ceph/dc326fc2-fa65-11ee-bc8e-c7b262605968/iscsi.foo.smithi069.yuumtr/tcmu-runner-entrypoint.sh 2024-04-14T14:13:54.804 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Apr 14 14:13:54 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-x[45487]: Non-zero exit code 1 from systemctl restart ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968@iscsi.foo.smithi069.yuumtr 2024-04-14T14:13:54.804 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Apr 14 14:13:54 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-x[45487]: systemctl: stderr Job for ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968@iscsi.foo.smithi069.yuumtr.service failed because the control process exited with error code. 2024-04-14T14:13:54.804 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Apr 14 14:13:54 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-x[45487]: systemctl: stderr See "systemctl status ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968@iscsi.foo.smithi069.yuumtr.service" and "journalctl -xeu ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968@iscsi.foo.smithi069.yuumtr.service" for details. 2024-04-14T14:13:54.804 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Apr 14 14:13:54 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-x[45487]: Traceback (most recent call last): 2024-04-14T14:13:54.804 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Apr 14 14:13:54 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-x[45487]: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-04-14T14:13:54.804 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Apr 14 14:13:54 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-x[45487]: return _run_code(code, main_globals, None, 2024-04-14T14:13:54.804 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Apr 14 14:13:54 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-x[45487]: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-04-14T14:13:54.804 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Apr 14 14:13:54 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-x[45487]: exec(code, run_globals) 2024-04-14T14:13:54.804 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Apr 14 14:13:54 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-x[45487]: File "/var/lib/ceph/dc326fc2-fa65-11ee-bc8e-c7b262605968/cephadm.de2a8faf91349910526b51da93d8bd42d57efdd2e93c32cd09ae64ecc1748ae9/__main__.py", line 5608, in 2024-04-14T14:13:54.804 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Apr 14 14:13:54 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-x[45487]: File "/var/lib/ceph/dc326fc2-fa65-11ee-bc8e-c7b262605968/cephadm.de2a8faf91349910526b51da93d8bd42d57efdd2e93c32cd09ae64ecc1748ae9/__main__.py", line 5596, in main 2024-04-14T14:13:54.804 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Apr 14 14:13:54 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-x[45487]: File "/var/lib/ceph/dc326fc2-fa65-11ee-bc8e-c7b262605968/cephadm.de2a8faf91349910526b51da93d8bd42d57efdd2e93c32cd09ae64ecc1748ae9/__main__.py", line 3096, in command_deploy_from 2024-04-14T14:13:54.804 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Apr 14 14:13:54 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-x[45487]: File "/var/lib/ceph/dc326fc2-fa65-11ee-bc8e-c7b262605968/cephadm.de2a8faf91349910526b51da93d8bd42d57efdd2e93c32cd09ae64ecc1748ae9/__main__.py", line 3131, in _common_deploy 2024-04-14T14:13:54.805 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Apr 14 14:13:54 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-x[45487]: File "/var/lib/ceph/dc326fc2-fa65-11ee-bc8e-c7b262605968/cephadm.de2a8faf91349910526b51da93d8bd42d57efdd2e93c32cd09ae64ecc1748ae9/__main__.py", line 3151, in _deploy_daemon_container 2024-04-14T14:13:54.805 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Apr 14 14:13:54 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-x[45487]: File "/var/lib/ceph/dc326fc2-fa65-11ee-bc8e-c7b262605968/cephadm.de2a8faf91349910526b51da93d8bd42d57efdd2e93c32cd09ae64ecc1748ae9/__main__.py", line 1123, in deploy_daemon 2024-04-14T14:13:54.805 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Apr 14 14:13:54 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-x[45487]: File "/var/lib/ceph/dc326fc2-fa65-11ee-bc8e-c7b262605968/cephadm.de2a8faf91349910526b51da93d8bd42d57efdd2e93c32cd09ae64ecc1748ae9/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-04-14T14:13:54.805 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Apr 14 14:13:54 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-x[45487]: RuntimeError: Failed command: systemctl restart ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968@iscsi.foo.smithi069.yuumtr: Job for ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968@iscsi.foo.smithi069.yuumtr.service failed because the control process exited with error code. 2024-04-14T14:13:54.805 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Apr 14 14:13:54 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-x[45487]: See "systemctl status ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968@iscsi.foo.smithi069.yuumtr.service" and "journalctl -xeu ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968@iscsi.foo.smithi069.yuumtr.service" for details. 2024-04-14T14:13:54.805 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Apr 14 14:13:54 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-x[45487]: Traceback (most recent call last): 2024-04-14T14:13:54.805 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Apr 14 14:13:54 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-x[45487]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1114, in _check_daemons 2024-04-14T14:13:54.805 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Apr 14 14:13:54 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-x[45487]: self.mgr._daemon_action(daemon_spec, action=action) 2024-04-14T14:13:54.806 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Apr 14 14:13:54 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-x[45487]: File "/usr/share/ceph/mgr/cephadm/module.py", line 2424, in _daemon_action 2024-04-14T14:13:54.806 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Apr 14 14:13:54 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-x[45487]: CephadmServe(self)._create_daemon(daemon_spec, reconfig=(action == 'reconfig'))) 2024-04-14T14:13:54.806 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Apr 14 14:13:54 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-x[45487]: File "/usr/share/ceph/mgr/cephadm/module.py", line 779, in wait_async 2024-04-14T14:13:54.806 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Apr 14 14:13:54 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-x[45487]: return self.event_loop.get_result(coro, timeout) 2024-04-14T14:13:54.806 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Apr 14 14:13:54 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-x[45487]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 136, in get_result 2024-04-14T14:13:54.806 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Apr 14 14:13:54 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-x[45487]: return future.result(timeout) 2024-04-14T14:13:54.806 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Apr 14 14:13:54 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-x[45487]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2024-04-14T14:13:54.806 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Apr 14 14:13:54 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-x[45487]: return self.__get_result() 2024-04-14T14:13:54.806 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Apr 14 14:13:54 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-x[45487]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2024-04-14T14:13:54.806 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Apr 14 14:13:54 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-x[45487]: raise self._exception 2024-04-14T14:13:54.806 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Apr 14 14:13:54 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-x[45487]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1374, in _create_daemon 2024-04-14T14:13:54.806 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Apr 14 14:13:54 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-x[45487]: config_blobs=daemon_spec.final_config, 2024-04-14T14:13:54.806 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Apr 14 14:13:54 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-x[45487]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1685, in _run_cephadm 2024-04-14T14:13:54.806 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Apr 14 14:13:54 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-x[45487]: f'cephadm exited with an error code: {code}, stderr: {err}') 2024-04-14T14:13:54.807 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Apr 14 14:13:54 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-x[45487]: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi069.yuumtr ... 2024-04-14T14:13:54.807 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Apr 14 14:13:54 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-x[45487]: Creating ceph-iscsi config... 2024-04-14T14:13:54.807 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Apr 14 14:13:54 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-x[45487]: Write file: /var/lib/ceph/dc326fc2-fa65-11ee-bc8e-c7b262605968/iscsi.foo.smithi069.yuumtr/iscsi-gateway.cfg 2024-04-14T14:13:54.807 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Apr 14 14:13:54 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-x[45487]: Write file: /var/lib/ceph/dc326fc2-fa65-11ee-bc8e-c7b262605968/iscsi.foo.smithi069.yuumtr/tcmu-runner-entrypoint.sh 2024-04-14T14:13:54.807 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Apr 14 14:13:54 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-x[45487]: Non-zero exit code 1 from systemctl restart ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968@iscsi.foo.smithi069.yuumtr 2024-04-14T14:13:54.807 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Apr 14 14:13:54 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-x[45487]: systemctl: stderr Job for ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968@iscsi.foo.smithi069.yuumtr.service failed because the control process exited with error code. 2024-04-14T14:13:54.807 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Apr 14 14:13:54 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-x[45487]: systemctl: stderr See "systemctl status ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968@iscsi.foo.smithi069.yuumtr.service" and "journalctl -xeu ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968@iscsi.foo.smithi069.yuumtr.service" for details. 2024-04-14T14:13:54.807 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Apr 14 14:13:54 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-x[45487]: Traceback (most recent call last): 2024-04-14T14:13:54.807 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Apr 14 14:13:54 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-x[45487]: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-04-14T14:13:54.808 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Apr 14 14:13:54 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-x[45487]: return _run_code(code, main_globals, None, 2024-04-14T14:13:54.808 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Apr 14 14:13:54 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-x[45487]: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-04-14T14:13:54.808 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Apr 14 14:13:54 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-x[45487]: exec(code, run_globals) 2024-04-14T14:13:54.808 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Apr 14 14:13:54 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-x[45487]: File "/var/lib/ceph/dc326fc2-fa65-11ee-bc8e-c7b262605968/cephadm.de2a8faf91349910526b51da93d8bd42d57efdd2e93c32cd09ae64ecc1748ae9/__main__.py", line 5608, in 2024-04-14T14:13:54.808 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Apr 14 14:13:54 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-x[45487]: File "/var/lib/ceph/dc326fc2-fa65-11ee-bc8e-c7b262605968/cephadm.de2a8faf91349910526b51da93d8bd42d57efdd2e93c32cd09ae64ecc1748ae9/__main__.py", line 5596, in main 2024-04-14T14:13:54.808 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Apr 14 14:13:54 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-x[45487]: File "/var/lib/ceph/dc326fc2-fa65-11ee-bc8e-c7b262605968/cephadm.de2a8faf91349910526b51da93d8bd42d57efdd2e93c32cd09ae64ecc1748ae9/__main__.py", line 3096, in command_deploy_from 2024-04-14T14:13:54.808 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Apr 14 14:13:54 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-x[45487]: File "/var/lib/ceph/dc326fc2-fa65-11ee-bc8e-c7b262605968/cephadm.de2a8faf91349910526b51da93d8bd42d57efdd2e93c32cd09ae64ecc1748ae9/__main__.py", line 3131, in _common_deploy 2024-04-14T14:13:54.808 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Apr 14 14:13:54 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-x[45487]: File "/var/lib/ceph/dc326fc2-fa65-11ee-bc8e-c7b262605968/cephadm.de2a8faf91349910526b51da93d8bd42d57efdd2e93c32cd09ae64ecc1748ae9/__main__.py", line 3151, in _deploy_daemon_container 2024-04-14T14:13:54.808 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Apr 14 14:13:54 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-x[45487]: File "/var/lib/ceph/dc326fc2-fa65-11ee-bc8e-c7b262605968/cephadm.de2a8faf91349910526b51da93d8bd42d57efdd2e93c32cd09ae64ecc1748ae9/__main__.py", line 1123, in deploy_daemon 2024-04-14T14:13:54.808 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Apr 14 14:13:54 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-x[45487]: File "/var/lib/ceph/dc326fc2-fa65-11ee-bc8e-c7b262605968/cephadm.de2a8faf91349910526b51da93d8bd42d57efdd2e93c32cd09ae64ecc1748ae9/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-04-14T14:13:54.808 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Apr 14 14:13:54 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-x[45487]: RuntimeError: Failed command: systemctl restart ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968@iscsi.foo.smithi069.yuumtr: Job for ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968@iscsi.foo.smithi069.yuumtr.service failed because the control process exited with error code. 2024-04-14T14:13:54.808 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Apr 14 14:13:54 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-x[45487]: See "systemctl status ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968@iscsi.foo.smithi069.yuumtr.service" and "journalctl -xeu ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968@iscsi.foo.smithi069.yuumtr.service" for details. 2024-04-14T14:13:55.567 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:13:55 smithi069 ceph-mon[29472]: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi069.yuumtr ... 2024-04-14T14:13:55.567 INFO:journalctl@ceph.mon.c.smithi069.stdout: Creating ceph-iscsi config... 2024-04-14T14:13:55.567 INFO:journalctl@ceph.mon.c.smithi069.stdout: Write file: /var/lib/ceph/dc326fc2-fa65-11ee-bc8e-c7b262605968/iscsi.foo.smithi069.yuumtr/iscsi-gateway.cfg 2024-04-14T14:13:55.567 INFO:journalctl@ceph.mon.c.smithi069.stdout: Write file: /var/lib/ceph/dc326fc2-fa65-11ee-bc8e-c7b262605968/iscsi.foo.smithi069.yuumtr/tcmu-runner-entrypoint.sh 2024-04-14T14:13:55.567 INFO:journalctl@ceph.mon.c.smithi069.stdout: Non-zero exit code 1 from systemctl restart ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968@iscsi.foo.smithi069.yuumtr 2024-04-14T14:13:55.568 INFO:journalctl@ceph.mon.c.smithi069.stdout: systemctl: stderr Job for ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968@iscsi.foo.smithi069.yuumtr.service failed because the control process exited with error code. 2024-04-14T14:13:55.568 INFO:journalctl@ceph.mon.c.smithi069.stdout: systemctl: stderr See "systemctl status ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968@iscsi.foo.smithi069.yuumtr.service" and "journalctl -xeu ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968@iscsi.foo.smithi069.yuumtr.service" for details. 2024-04-14T14:13:55.568 INFO:journalctl@ceph.mon.c.smithi069.stdout: Traceback (most recent call last): 2024-04-14T14:13:55.568 INFO:journalctl@ceph.mon.c.smithi069.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-04-14T14:13:55.568 INFO:journalctl@ceph.mon.c.smithi069.stdout: return _run_code(code, main_globals, None, 2024-04-14T14:13:55.568 INFO:journalctl@ceph.mon.c.smithi069.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-04-14T14:13:55.568 INFO:journalctl@ceph.mon.c.smithi069.stdout: exec(code, run_globals) 2024-04-14T14:13:55.568 INFO:journalctl@ceph.mon.c.smithi069.stdout: File "/var/lib/ceph/dc326fc2-fa65-11ee-bc8e-c7b262605968/cephadm.de2a8faf91349910526b51da93d8bd42d57efdd2e93c32cd09ae64ecc1748ae9/__main__.py", line 5608, in 2024-04-14T14:13:55.568 INFO:journalctl@ceph.mon.c.smithi069.stdout: File "/var/lib/ceph/dc326fc2-fa65-11ee-bc8e-c7b262605968/cephadm.de2a8faf91349910526b51da93d8bd42d57efdd2e93c32cd09ae64ecc1748ae9/__main__.py", line 5596, in main 2024-04-14T14:13:55.568 INFO:journalctl@ceph.mon.c.smithi069.stdout: File "/var/lib/ceph/dc326fc2-fa65-11ee-bc8e-c7b262605968/cephadm.de2a8faf91349910526b51da93d8bd42d57efdd2e93c32cd09ae64ecc1748ae9/__main__.py", line 3096, in command_deploy_from 2024-04-14T14:13:55.568 INFO:journalctl@ceph.mon.c.smithi069.stdout: File "/var/lib/ceph/dc326fc2-fa65-11ee-bc8e-c7b262605968/cephadm.de2a8faf91349910526b51da93d8bd42d57efdd2e93c32cd09ae64ecc1748ae9/__main__.py", line 3131, in _common_deploy 2024-04-14T14:13:55.568 INFO:journalctl@ceph.mon.c.smithi069.stdout: File "/var/lib/ceph/dc326fc2-fa65-11ee-bc8e-c7b262605968/cephadm.de2a8faf91349910526b51da93d8bd42d57efdd2e93c32cd09ae64ecc1748ae9/__main__.py", line 3151, in _deploy_daemon_container 2024-04-14T14:13:55.568 INFO:journalctl@ceph.mon.c.smithi069.stdout: File "/var/lib/ceph/dc326fc2-fa65-11ee-bc8e-c7b262605968/cephadm.de2a8faf91349910526b51da93d8bd42d57efdd2e93c32cd09ae64ecc1748ae9/__main__.py", line 1123, in deploy_daemon 2024-04-14T14:13:55.568 INFO:journalctl@ceph.mon.c.smithi069.stdout: File "/var/lib/ceph/dc326fc2-fa65-11ee-bc8e-c7b262605968/cephadm.de2a8faf91349910526b51da93d8bd42d57efdd2e93c32cd09ae64ecc1748ae9/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-04-14T14:13:55.569 INFO:journalctl@ceph.mon.c.smithi069.stdout: RuntimeError: Failed command: systemctl restart ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968@iscsi.foo.smithi069.yuumtr: Job for ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968@iscsi.foo.smithi069.yuumtr.service failed because the control process exited with error code. 2024-04-14T14:13:55.569 INFO:journalctl@ceph.mon.c.smithi069.stdout: See "systemctl status ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968@iscsi.foo.smithi069.yuumtr.service" and "journalctl -xeu ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968@iscsi.foo.smithi069.yuumtr.service" for details. 2024-04-14T14:13:55.569 INFO:journalctl@ceph.mon.c.smithi069.stdout: Traceback (most recent call last): 2024-04-14T14:13:55.569 INFO:journalctl@ceph.mon.c.smithi069.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1114, in _check_daemons 2024-04-14T14:13:55.569 INFO:journalctl@ceph.mon.c.smithi069.stdout: self.mgr._daemon_action(daemon_spec, action=action) 2024-04-14T14:13:55.569 INFO:journalctl@ceph.mon.c.smithi069.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 2424, in _daemon_action 2024-04-14T14:13:55.569 INFO:journalctl@ceph.mon.c.smithi069.stdout: CephadmServe(self)._create_daemon(daemon_spec, reconfig=(action == 'reconfig'))) 2024-04-14T14:13:55.569 INFO:journalctl@ceph.mon.c.smithi069.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 779, in wait_async 2024-04-14T14:13:55.569 INFO:journalctl@ceph.mon.c.smithi069.stdout: return self.event_loop.get_result(coro, timeout) 2024-04-14T14:13:55.569 INFO:journalctl@ceph.mon.c.smithi069.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 136, in get_result 2024-04-14T14:13:55.569 INFO:journalctl@ceph.mon.c.smithi069.stdout: return future.result(timeout) 2024-04-14T14:13:55.569 INFO:journalctl@ceph.mon.c.smithi069.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2024-04-14T14:13:55.569 INFO:journalctl@ceph.mon.c.smithi069.stdout: return self.__get_result() 2024-04-14T14:13:55.569 INFO:journalctl@ceph.mon.c.smithi069.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2024-04-14T14:13:55.569 INFO:journalctl@ceph.mon.c.smithi069.stdout: raise self._exception 2024-04-14T14:13:55.570 INFO:journalctl@ceph.mon.c.smithi069.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1374, in _create_daemon 2024-04-14T14:13:55.570 INFO:journalctl@ceph.mon.c.smithi069.stdout: config_blobs=daemon_spec.final_config, 2024-04-14T14:13:55.570 INFO:journalctl@ceph.mon.c.smithi069.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1685, in _run_cephadm 2024-04-14T14:13:55.570 INFO:journalctl@ceph.mon.c.smithi069.stdout: f'cephadm exited with an error code: {code}, stderr: {err}') 2024-04-14T14:13:55.570 INFO:journalctl@ceph.mon.c.smithi069.stdout: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi069.yuumtr ... 2024-04-14T14:13:55.570 INFO:journalctl@ceph.mon.c.smithi069.stdout: Creating ceph-iscsi config... 2024-04-14T14:13:55.570 INFO:journalctl@ceph.mon.c.smithi069.stdout: Write file: /var/lib/ceph/dc326fc2-fa65-11ee-bc8e-c7b262605968/iscsi.foo.smithi069.yuumtr/iscsi-gateway.cfg 2024-04-14T14:13:55.570 INFO:journalctl@ceph.mon.c.smithi069.stdout: Write file: /var/lib/ceph/dc326fc2-fa65-11ee-bc8e-c7b262605968/iscsi.foo.smithi069.yuumtr/tcmu-runner-entrypoint.sh 2024-04-14T14:13:55.570 INFO:journalctl@ceph.mon.c.smithi069.stdout: Non-zero exit code 1 from systemctl restart ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968@iscsi.foo.smithi069.yuumtr 2024-04-14T14:13:55.570 INFO:journalctl@ceph.mon.c.smithi069.stdout: systemctl: stderr Job for ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968@iscsi.foo.smithi069.yuumtr.service failed because the control process exited with error code. 2024-04-14T14:13:55.570 INFO:journalctl@ceph.mon.c.smithi069.stdout: systemctl: stderr See "systemctl status ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968@iscsi.foo.smithi069.yuumtr.service" and "journalctl -xeu ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968@iscsi.foo.smithi069.yuumtr.service" for details. 2024-04-14T14:13:55.570 INFO:journalctl@ceph.mon.c.smithi069.stdout: Traceback (most recent call last): 2024-04-14T14:13:55.570 INFO:journalctl@ceph.mon.c.smithi069.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-04-14T14:13:55.570 INFO:journalctl@ceph.mon.c.smithi069.stdout: return _run_code(code, main_globals, None, 2024-04-14T14:13:55.570 INFO:journalctl@ceph.mon.c.smithi069.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-04-14T14:13:55.571 INFO:journalctl@ceph.mon.c.smithi069.stdout: exec(code, run_globals) 2024-04-14T14:13:55.571 INFO:journalctl@ceph.mon.c.smithi069.stdout: File "/var/lib/ceph/dc326fc2-fa65-11ee-bc8e-c7b262605968/cephadm.de2a8faf91349910526b51da93d8bd42d57efdd2e93c32cd09ae64ecc1748ae9/__main__.py", line 5608, in 2024-04-14T14:13:55.571 INFO:journalctl@ceph.mon.c.smithi069.stdout: File "/var/lib/ceph/dc326fc2-fa65-11ee-bc8e-c7b262605968/cephadm.de2a8faf91349910526b51da93d8bd42d57efdd2e93c32cd09ae64ecc1748ae9/__main__.py", line 5596, in main 2024-04-14T14:13:55.571 INFO:journalctl@ceph.mon.c.smithi069.stdout: File "/var/lib/ceph/dc326fc2-fa65-11ee-bc8e-c7b262605968/cephadm.de2a8faf91349910526b51da93d8bd42d57efdd2e93c32cd09ae64ecc1748ae9/__main__.py", line 3096, in command_deploy_from 2024-04-14T14:13:55.571 INFO:journalctl@ceph.mon.c.smithi069.stdout: File "/var/lib/ceph/dc326fc2-fa65-11ee-bc8e-c7b262605968/cephadm.de2a8faf91349910526b51da93d8bd42d57efdd2e93c32cd09ae64ecc1748ae9/__main__.py", line 3131, in _common_deploy 2024-04-14T14:13:55.571 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:13:55 smithi069 ceph-mon[24818]: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi069.yuumtr ... 2024-04-14T14:13:55.571 INFO:journalctl@ceph.mon.a.smithi069.stdout: Creating ceph-iscsi config... 2024-04-14T14:13:55.571 INFO:journalctl@ceph.mon.a.smithi069.stdout: Write file: /var/lib/ceph/dc326fc2-fa65-11ee-bc8e-c7b262605968/iscsi.foo.smithi069.yuumtr/iscsi-gateway.cfg 2024-04-14T14:13:55.571 INFO:journalctl@ceph.mon.a.smithi069.stdout: Write file: /var/lib/ceph/dc326fc2-fa65-11ee-bc8e-c7b262605968/iscsi.foo.smithi069.yuumtr/tcmu-runner-entrypoint.sh 2024-04-14T14:13:55.571 INFO:journalctl@ceph.mon.a.smithi069.stdout: Non-zero exit code 1 from systemctl restart ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968@iscsi.foo.smithi069.yuumtr 2024-04-14T14:13:55.571 INFO:journalctl@ceph.mon.a.smithi069.stdout: systemctl: stderr Job for ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968@iscsi.foo.smithi069.yuumtr.service failed because the control process exited with error code. 2024-04-14T14:13:55.572 INFO:journalctl@ceph.mon.a.smithi069.stdout: systemctl: stderr See "systemctl status ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968@iscsi.foo.smithi069.yuumtr.service" and "journalctl -xeu ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968@iscsi.foo.smithi069.yuumtr.service" for details. 2024-04-14T14:13:55.572 INFO:journalctl@ceph.mon.a.smithi069.stdout: Traceback (most recent call last): 2024-04-14T14:13:55.572 INFO:journalctl@ceph.mon.a.smithi069.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-04-14T14:13:55.572 INFO:journalctl@ceph.mon.a.smithi069.stdout: return _run_code(code, main_globals, None, 2024-04-14T14:13:55.572 INFO:journalctl@ceph.mon.a.smithi069.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-04-14T14:13:55.572 INFO:journalctl@ceph.mon.a.smithi069.stdout: exec(code, run_globals) 2024-04-14T14:13:55.572 INFO:journalctl@ceph.mon.a.smithi069.stdout: File "/var/lib/ceph/dc326fc2-fa65-11ee-bc8e-c7b262605968/cephadm.de2a8faf91349910526b51da93d8bd42d57efdd2e93c32cd09ae64ecc1748ae9/__main__.py", line 5608, in 2024-04-14T14:13:55.572 INFO:journalctl@ceph.mon.a.smithi069.stdout: File "/var/lib/ceph/dc326fc2-fa65-11ee-bc8e-c7b262605968/cephadm.de2a8faf91349910526b51da93d8bd42d57efdd2e93c32cd09ae64ecc1748ae9/__main__.py", line 5596, in main 2024-04-14T14:13:55.572 INFO:journalctl@ceph.mon.a.smithi069.stdout: File "/var/lib/ceph/dc326fc2-fa65-11ee-bc8e-c7b262605968/cephadm.de2a8faf91349910526b51da93d8bd42d57efdd2e93c32cd09ae64ecc1748ae9/__main__.py", line 3096, in command_deploy_from 2024-04-14T14:13:55.572 INFO:journalctl@ceph.mon.a.smithi069.stdout: File "/var/lib/ceph/dc326fc2-fa65-11ee-bc8e-c7b262605968/cephadm.de2a8faf91349910526b51da93d8bd42d57efdd2e93c32cd09ae64ecc1748ae9/__main__.py", line 3131, in _common_deploy 2024-04-14T14:13:55.572 INFO:journalctl@ceph.mon.a.smithi069.stdout: File "/var/lib/ceph/dc326fc2-fa65-11ee-bc8e-c7b262605968/cephadm.de2a8faf91349910526b51da93d8bd42d57efdd2e93c32cd09ae64ecc1748ae9/__main__.py", line 3151, in _deploy_daemon_container 2024-04-14T14:13:55.572 INFO:journalctl@ceph.mon.a.smithi069.stdout: File "/var/lib/ceph/dc326fc2-fa65-11ee-bc8e-c7b262605968/cephadm.de2a8faf91349910526b51da93d8bd42d57efdd2e93c32cd09ae64ecc1748ae9/__main__.py", line 1123, in deploy_daemon 2024-04-14T14:13:55.572 INFO:journalctl@ceph.mon.a.smithi069.stdout: File "/var/lib/ceph/dc326fc2-fa65-11ee-bc8e-c7b262605968/cephadm.de2a8faf91349910526b51da93d8bd42d57efdd2e93c32cd09ae64ecc1748ae9/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-04-14T14:13:55.572 INFO:journalctl@ceph.mon.a.smithi069.stdout: RuntimeError: Failed command: systemctl restart ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968@iscsi.foo.smithi069.yuumtr: Job for ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968@iscsi.foo.smithi069.yuumtr.service failed because the control process exited with error code. 2024-04-14T14:13:55.573 INFO:journalctl@ceph.mon.a.smithi069.stdout: See "systemctl status ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968@iscsi.foo.smithi069.yuumtr.service" and "journalctl -xeu ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968@iscsi.foo.smithi069.yuumtr.service" for details. 2024-04-14T14:13:55.573 INFO:journalctl@ceph.mon.a.smithi069.stdout: Traceback (most recent call last): 2024-04-14T14:13:55.573 INFO:journalctl@ceph.mon.a.smithi069.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1114, in _check_daemons 2024-04-14T14:13:55.573 INFO:journalctl@ceph.mon.a.smithi069.stdout: self.mgr._daemon_action(daemon_spec, action=action) 2024-04-14T14:13:55.573 INFO:journalctl@ceph.mon.a.smithi069.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 2424, in _daemon_action 2024-04-14T14:13:55.573 INFO:journalctl@ceph.mon.a.smithi069.stdout: CephadmServe(self)._create_daemon(daemon_spec, reconfig=(action == 'reconfig'))) 2024-04-14T14:13:55.573 INFO:journalctl@ceph.mon.a.smithi069.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 779, in wait_async 2024-04-14T14:13:55.573 INFO:journalctl@ceph.mon.a.smithi069.stdout: return self.event_loop.get_result(coro, timeout) 2024-04-14T14:13:55.573 INFO:journalctl@ceph.mon.a.smithi069.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 136, in get_result 2024-04-14T14:13:55.573 INFO:journalctl@ceph.mon.a.smithi069.stdout: return future.result(timeout) 2024-04-14T14:13:55.573 INFO:journalctl@ceph.mon.a.smithi069.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2024-04-14T14:13:55.573 INFO:journalctl@ceph.mon.a.smithi069.stdout: return self.__get_result() 2024-04-14T14:13:55.573 INFO:journalctl@ceph.mon.a.smithi069.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2024-04-14T14:13:55.573 INFO:journalctl@ceph.mon.a.smithi069.stdout: raise self._exception 2024-04-14T14:13:55.574 INFO:journalctl@ceph.mon.a.smithi069.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1374, in _create_daemon 2024-04-14T14:13:55.574 INFO:journalctl@ceph.mon.a.smithi069.stdout: config_blobs=daemon_spec.final_config, 2024-04-14T14:13:55.574 INFO:journalctl@ceph.mon.a.smithi069.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1685, in _run_cephadm 2024-04-14T14:13:55.574 INFO:journalctl@ceph.mon.a.smithi069.stdout: f'cephadm exited with an error code: {code}, stderr: {err}') 2024-04-14T14:13:55.574 INFO:journalctl@ceph.mon.a.smithi069.stdout: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi069.yuumtr ... 2024-04-14T14:13:55.574 INFO:journalctl@ceph.mon.a.smithi069.stdout: Creating ceph-iscsi config... 2024-04-14T14:13:55.574 INFO:journalctl@ceph.mon.a.smithi069.stdout: Write file: /var/lib/ceph/dc326fc2-fa65-11ee-bc8e-c7b262605968/iscsi.foo.smithi069.yuumtr/iscsi-gateway.cfg 2024-04-14T14:13:55.574 INFO:journalctl@ceph.mon.a.smithi069.stdout: Write file: /var/lib/ceph/dc326fc2-fa65-11ee-bc8e-c7b262605968/iscsi.foo.smithi069.yuumtr/tcmu-runner-entrypoint.sh 2024-04-14T14:13:55.574 INFO:journalctl@ceph.mon.a.smithi069.stdout: Non-zero exit code 1 from systemctl restart ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968@iscsi.foo.smithi069.yuumtr 2024-04-14T14:13:55.574 INFO:journalctl@ceph.mon.a.smithi069.stdout: systemctl: stderr Job for ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968@iscsi.foo.smithi069.yuumtr.service failed because the control process exited with error code. 2024-04-14T14:13:55.574 INFO:journalctl@ceph.mon.a.smithi069.stdout: systemctl: stderr See "systemctl status ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968@iscsi.foo.smithi069.yuumtr.service" and "journalctl -xeu ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968@iscsi.foo.smithi069.yuumtr.service" for details. 2024-04-14T14:13:55.574 INFO:journalctl@ceph.mon.a.smithi069.stdout: Traceback (most recent call last): 2024-04-14T14:13:55.575 INFO:journalctl@ceph.mon.a.smithi069.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-04-14T14:13:55.575 INFO:journalctl@ceph.mon.a.smithi069.stdout: return _run_code(code, main_globals, None, 2024-04-14T14:13:55.575 INFO:journalctl@ceph.mon.a.smithi069.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-04-14T14:13:55.575 INFO:journalctl@ceph.mon.a.smithi069.stdout: exec(code, run_globals) 2024-04-14T14:13:55.575 INFO:journalctl@ceph.mon.a.smithi069.stdout: File "/var/lib/ceph/dc326fc2-fa65-11ee-bc8e-c7b262605968/cephadm.de2a8faf91349910526b51da93d8bd42d57efdd2e93c32cd09ae64ecc1748ae9/__main__.py", line 5608, in 2024-04-14T14:13:55.575 INFO:journalctl@ceph.mon.a.smithi069.stdout: File "/var/lib/ceph/dc326fc2-fa65-11ee-bc8e-c7b262605968/cephadm.de2a8faf91349910526b51da93d8bd42d57efdd2e93c32cd09ae64ecc1748ae9/__main__.py", line 5596, in main 2024-04-14T14:13:55.575 INFO:journalctl@ceph.mon.a.smithi069.stdout: File "/var/lib/ceph/dc326fc2-fa65-11ee-bc8e-c7b262605968/cephadm.de2a8faf91349910526b51da93d8bd42d57efdd2e93c32cd09ae64ecc1748ae9/__main__.py", line 3096, in command_deploy_from 2024-04-14T14:13:55.575 INFO:journalctl@ceph.mon.a.smithi069.stdout: File "/var/lib/ceph/dc326fc2-fa65-11ee-bc8e-c7b262605968/cephadm.de2a8faf91349910526b51da93d8bd42d57efdd2e93c32cd09ae64ecc1748ae9/__main__.py", line 3131, in _common_deploy 2024-04-14T14:13:55.575 INFO:journalctl@ceph.mon.a.smithi069.stdout: File "/var/lib/ceph/dc326fc2-fa65-11ee-bc8e-c7b262605968/cephadm.de2a8faf91349910526b51da93d8bd42d57efdd2e93c32cd09ae64ecc1748ae9/__main__.py", line 3151, in _deploy_daemon_container 2024-04-14T14:13:55.575 INFO:journalctl@ceph.mon.a.smithi069.stdout: File "/var/lib/ceph/dc326fc2-fa65-11ee-bc8e-c7b262605968/cephadm.de2a8faf91349910526b51da93d8bd42d57efdd2e93c32cd09ae64ecc1748ae9/__main__.py", line 1123, in deploy_daemon 2024-04-14T14:13:55.575 INFO:journalctl@ceph.mon.a.smithi069.stdout: File "/var/lib/ceph/dc326fc2-fa65-11ee-bc8e-c7b262605968/cephadm.de2a8faf91349910526b51da93d8bd42d57efdd2e93c32cd09ae64ecc1748ae9/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-04-14T14:13:55.575 INFO:journalctl@ceph.mon.a.smithi069.stdout: RuntimeError: Failed command: systemctl restart ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968@iscsi.foo.smithi069.yuumtr: Job for ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968@iscsi.foo.smithi069.yuumtr.service failed because the control process exited with error code. 2024-04-14T14:13:55.575 INFO:journalctl@ceph.mon.a.smithi069.stdout: See "systemctl status ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968@iscsi.foo.smithi069.yuumtr.service" and "journalctl -xeu ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968@iscsi.foo.smithi069.yuumtr.service" for details. 2024-04-14T14:13:55.576 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:13:55 smithi069 ceph-mon[24818]: Reconfiguring prometheus.a (dependencies changed)... 2024-04-14T14:13:55.576 INFO:journalctl@ceph.mon.c.smithi069.stdout: File "/var/lib/ceph/dc326fc2-fa65-11ee-bc8e-c7b262605968/cephadm.de2a8faf91349910526b51da93d8bd42d57efdd2e93c32cd09ae64ecc1748ae9/__main__.py", line 3151, in _deploy_daemon_container 2024-04-14T14:13:55.576 INFO:journalctl@ceph.mon.c.smithi069.stdout: File "/var/lib/ceph/dc326fc2-fa65-11ee-bc8e-c7b262605968/cephadm.de2a8faf91349910526b51da93d8bd42d57efdd2e93c32cd09ae64ecc1748ae9/__main__.py", line 1123, in deploy_daemon 2024-04-14T14:13:55.576 INFO:journalctl@ceph.mon.c.smithi069.stdout: File "/var/lib/ceph/dc326fc2-fa65-11ee-bc8e-c7b262605968/cephadm.de2a8faf91349910526b51da93d8bd42d57efdd2e93c32cd09ae64ecc1748ae9/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-04-14T14:13:55.576 INFO:journalctl@ceph.mon.c.smithi069.stdout: RuntimeError: Failed command: systemctl restart ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968@iscsi.foo.smithi069.yuumtr: Job for ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968@iscsi.foo.smithi069.yuumtr.service failed because the control process exited with error code. 2024-04-14T14:13:55.576 INFO:journalctl@ceph.mon.c.smithi069.stdout: See "systemctl status ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968@iscsi.foo.smithi069.yuumtr.service" and "journalctl -xeu ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968@iscsi.foo.smithi069.yuumtr.service" for details. 2024-04-14T14:13:55.576 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:13:55 smithi069 ceph-mon[29472]: Reconfiguring prometheus.a (dependencies changed)... 2024-04-14T14:13:55.582 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:13:55 smithi122 ceph-mon[25721]: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi069.yuumtr ... 2024-04-14T14:13:55.583 INFO:journalctl@ceph.mon.b.smithi122.stdout: Creating ceph-iscsi config... 2024-04-14T14:13:55.583 INFO:journalctl@ceph.mon.b.smithi122.stdout: Write file: /var/lib/ceph/dc326fc2-fa65-11ee-bc8e-c7b262605968/iscsi.foo.smithi069.yuumtr/iscsi-gateway.cfg 2024-04-14T14:13:55.583 INFO:journalctl@ceph.mon.b.smithi122.stdout: Write file: /var/lib/ceph/dc326fc2-fa65-11ee-bc8e-c7b262605968/iscsi.foo.smithi069.yuumtr/tcmu-runner-entrypoint.sh 2024-04-14T14:13:55.583 INFO:journalctl@ceph.mon.b.smithi122.stdout: Non-zero exit code 1 from systemctl restart ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968@iscsi.foo.smithi069.yuumtr 2024-04-14T14:13:55.583 INFO:journalctl@ceph.mon.b.smithi122.stdout: systemctl: stderr Job for ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968@iscsi.foo.smithi069.yuumtr.service failed because the control process exited with error code. 2024-04-14T14:13:55.583 INFO:journalctl@ceph.mon.b.smithi122.stdout: systemctl: stderr See "systemctl status ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968@iscsi.foo.smithi069.yuumtr.service" and "journalctl -xeu ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968@iscsi.foo.smithi069.yuumtr.service" for details. 2024-04-14T14:13:55.583 INFO:journalctl@ceph.mon.b.smithi122.stdout: Traceback (most recent call last): 2024-04-14T14:13:55.583 INFO:journalctl@ceph.mon.b.smithi122.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-04-14T14:13:55.583 INFO:journalctl@ceph.mon.b.smithi122.stdout: return _run_code(code, main_globals, None, 2024-04-14T14:13:55.583 INFO:journalctl@ceph.mon.b.smithi122.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-04-14T14:13:55.583 INFO:journalctl@ceph.mon.b.smithi122.stdout: exec(code, run_globals) 2024-04-14T14:13:55.583 INFO:journalctl@ceph.mon.b.smithi122.stdout: File "/var/lib/ceph/dc326fc2-fa65-11ee-bc8e-c7b262605968/cephadm.de2a8faf91349910526b51da93d8bd42d57efdd2e93c32cd09ae64ecc1748ae9/__main__.py", line 5608, in 2024-04-14T14:13:55.583 INFO:journalctl@ceph.mon.b.smithi122.stdout: File "/var/lib/ceph/dc326fc2-fa65-11ee-bc8e-c7b262605968/cephadm.de2a8faf91349910526b51da93d8bd42d57efdd2e93c32cd09ae64ecc1748ae9/__main__.py", line 5596, in main 2024-04-14T14:13:55.583 INFO:journalctl@ceph.mon.b.smithi122.stdout: File "/var/lib/ceph/dc326fc2-fa65-11ee-bc8e-c7b262605968/cephadm.de2a8faf91349910526b51da93d8bd42d57efdd2e93c32cd09ae64ecc1748ae9/__main__.py", line 3096, in command_deploy_from 2024-04-14T14:13:55.584 INFO:journalctl@ceph.mon.b.smithi122.stdout: File "/var/lib/ceph/dc326fc2-fa65-11ee-bc8e-c7b262605968/cephadm.de2a8faf91349910526b51da93d8bd42d57efdd2e93c32cd09ae64ecc1748ae9/__main__.py", line 3131, in _common_deploy 2024-04-14T14:13:55.584 INFO:journalctl@ceph.mon.b.smithi122.stdout: File "/var/lib/ceph/dc326fc2-fa65-11ee-bc8e-c7b262605968/cephadm.de2a8faf91349910526b51da93d8bd42d57efdd2e93c32cd09ae64ecc1748ae9/__main__.py", line 3151, in _deploy_daemon_container 2024-04-14T14:13:55.584 INFO:journalctl@ceph.mon.b.smithi122.stdout: File "/var/lib/ceph/dc326fc2-fa65-11ee-bc8e-c7b262605968/cephadm.de2a8faf91349910526b51da93d8bd42d57efdd2e93c32cd09ae64ecc1748ae9/__main__.py", line 1123, in deploy_daemon 2024-04-14T14:13:55.584 INFO:journalctl@ceph.mon.b.smithi122.stdout: File "/var/lib/ceph/dc326fc2-fa65-11ee-bc8e-c7b262605968/cephadm.de2a8faf91349910526b51da93d8bd42d57efdd2e93c32cd09ae64ecc1748ae9/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-04-14T14:13:55.584 INFO:journalctl@ceph.mon.b.smithi122.stdout: RuntimeError: Failed command: systemctl restart ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968@iscsi.foo.smithi069.yuumtr: Job for ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968@iscsi.foo.smithi069.yuumtr.service failed because the control process exited with error code. 2024-04-14T14:13:55.584 INFO:journalctl@ceph.mon.b.smithi122.stdout: See "systemctl status ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968@iscsi.foo.smithi069.yuumtr.service" and "journalctl -xeu ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968@iscsi.foo.smithi069.yuumtr.service" for details. 2024-04-14T14:13:55.584 INFO:journalctl@ceph.mon.b.smithi122.stdout: Traceback (most recent call last): 2024-04-14T14:13:55.584 INFO:journalctl@ceph.mon.b.smithi122.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1114, in _check_daemons 2024-04-14T14:13:55.584 INFO:journalctl@ceph.mon.b.smithi122.stdout: self.mgr._daemon_action(daemon_spec, action=action) 2024-04-14T14:13:55.584 INFO:journalctl@ceph.mon.b.smithi122.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 2424, in _daemon_action 2024-04-14T14:13:55.584 INFO:journalctl@ceph.mon.b.smithi122.stdout: CephadmServe(self)._create_daemon(daemon_spec, reconfig=(action == 'reconfig'))) 2024-04-14T14:13:55.584 INFO:journalctl@ceph.mon.b.smithi122.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 779, in wait_async 2024-04-14T14:13:55.584 INFO:journalctl@ceph.mon.b.smithi122.stdout: return self.event_loop.get_result(coro, timeout) 2024-04-14T14:13:55.584 INFO:journalctl@ceph.mon.b.smithi122.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 136, in get_result 2024-04-14T14:13:55.585 INFO:journalctl@ceph.mon.b.smithi122.stdout: return future.result(timeout) 2024-04-14T14:13:55.585 INFO:journalctl@ceph.mon.b.smithi122.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2024-04-14T14:13:55.585 INFO:journalctl@ceph.mon.b.smithi122.stdout: return self.__get_result() 2024-04-14T14:13:55.585 INFO:journalctl@ceph.mon.b.smithi122.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2024-04-14T14:13:55.585 INFO:journalctl@ceph.mon.b.smithi122.stdout: raise self._exception 2024-04-14T14:13:55.585 INFO:journalctl@ceph.mon.b.smithi122.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1374, in _create_daemon 2024-04-14T14:13:55.585 INFO:journalctl@ceph.mon.b.smithi122.stdout: config_blobs=daemon_spec.final_config, 2024-04-14T14:13:55.585 INFO:journalctl@ceph.mon.b.smithi122.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1685, in _run_cephadm 2024-04-14T14:13:55.585 INFO:journalctl@ceph.mon.b.smithi122.stdout: f'cephadm exited with an error code: {code}, stderr: {err}') 2024-04-14T14:13:55.585 INFO:journalctl@ceph.mon.b.smithi122.stdout: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi069.yuumtr ... 2024-04-14T14:13:55.585 INFO:journalctl@ceph.mon.b.smithi122.stdout: Creating ceph-iscsi config... 2024-04-14T14:13:55.585 INFO:journalctl@ceph.mon.b.smithi122.stdout: Write file: /var/lib/ceph/dc326fc2-fa65-11ee-bc8e-c7b262605968/iscsi.foo.smithi069.yuumtr/iscsi-gateway.cfg 2024-04-14T14:13:55.585 INFO:journalctl@ceph.mon.b.smithi122.stdout: Write file: /var/lib/ceph/dc326fc2-fa65-11ee-bc8e-c7b262605968/iscsi.foo.smithi069.yuumtr/tcmu-runner-entrypoint.sh 2024-04-14T14:13:55.585 INFO:journalctl@ceph.mon.b.smithi122.stdout: Non-zero exit code 1 from systemctl restart ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968@iscsi.foo.smithi069.yuumtr 2024-04-14T14:13:55.586 INFO:journalctl@ceph.mon.b.smithi122.stdout: systemctl: stderr Job for ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968@iscsi.foo.smithi069.yuumtr.service failed because the control process exited with error code. 2024-04-14T14:13:55.586 INFO:journalctl@ceph.mon.b.smithi122.stdout: systemctl: stderr See "systemctl status ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968@iscsi.foo.smithi069.yuumtr.service" and "journalctl -xeu ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968@iscsi.foo.smithi069.yuumtr.service" for details. 2024-04-14T14:13:55.586 INFO:journalctl@ceph.mon.b.smithi122.stdout: Traceback (most recent call last): 2024-04-14T14:13:55.586 INFO:journalctl@ceph.mon.b.smithi122.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-04-14T14:13:55.586 INFO:journalctl@ceph.mon.b.smithi122.stdout: return _run_code(code, main_globals, None, 2024-04-14T14:13:55.586 INFO:journalctl@ceph.mon.b.smithi122.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-04-14T14:13:55.586 INFO:journalctl@ceph.mon.b.smithi122.stdout: exec(code, run_globals) 2024-04-14T14:13:55.586 INFO:journalctl@ceph.mon.b.smithi122.stdout: File "/var/lib/ceph/dc326fc2-fa65-11ee-bc8e-c7b262605968/cephadm.de2a8faf91349910526b51da93d8bd42d57efdd2e93c32cd09ae64ecc1748ae9/__main__.py", line 5608, in 2024-04-14T14:13:55.586 INFO:journalctl@ceph.mon.b.smithi122.stdout: File "/var/lib/ceph/dc326fc2-fa65-11ee-bc8e-c7b262605968/cephadm.de2a8faf91349910526b51da93d8bd42d57efdd2e93c32cd09ae64ecc1748ae9/__main__.py", line 5596, in main 2024-04-14T14:13:55.586 INFO:journalctl@ceph.mon.b.smithi122.stdout: File "/var/lib/ceph/dc326fc2-fa65-11ee-bc8e-c7b262605968/cephadm.de2a8faf91349910526b51da93d8bd42d57efdd2e93c32cd09ae64ecc1748ae9/__main__.py", line 3096, in command_deploy_from 2024-04-14T14:13:55.586 INFO:journalctl@ceph.mon.b.smithi122.stdout: File "/var/lib/ceph/dc326fc2-fa65-11ee-bc8e-c7b262605968/cephadm.de2a8faf91349910526b51da93d8bd42d57efdd2e93c32cd09ae64ecc1748ae9/__main__.py", line 3131, in _common_deploy 2024-04-14T14:13:55.586 INFO:journalctl@ceph.mon.b.smithi122.stdout: File "/var/lib/ceph/dc326fc2-fa65-11ee-bc8e-c7b262605968/cephadm.de2a8faf91349910526b51da93d8bd42d57efdd2e93c32cd09ae64ecc1748ae9/__main__.py", line 3151, in _deploy_daemon_container 2024-04-14T14:13:55.586 INFO:journalctl@ceph.mon.b.smithi122.stdout: File "/var/lib/ceph/dc326fc2-fa65-11ee-bc8e-c7b262605968/cephadm.de2a8faf91349910526b51da93d8bd42d57efdd2e93c32cd09ae64ecc1748ae9/__main__.py", line 1123, in deploy_daemon 2024-04-14T14:13:55.587 INFO:journalctl@ceph.mon.b.smithi122.stdout: File "/var/lib/ceph/dc326fc2-fa65-11ee-bc8e-c7b262605968/cephadm.de2a8faf91349910526b51da93d8bd42d57efdd2e93c32cd09ae64ecc1748ae9/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-04-14T14:13:55.587 INFO:journalctl@ceph.mon.b.smithi122.stdout: RuntimeError: Failed command: systemctl restart ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968@iscsi.foo.smithi069.yuumtr: Job for ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968@iscsi.foo.smithi069.yuumtr.service failed because the control process exited with error code. 2024-04-14T14:13:55.587 INFO:journalctl@ceph.mon.b.smithi122.stdout: See "systemctl status ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968@iscsi.foo.smithi069.yuumtr.service" and "journalctl -xeu ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968@iscsi.foo.smithi069.yuumtr.service" for details. 2024-04-14T14:13:55.587 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:13:55 smithi122 ceph-mon[25721]: Reconfiguring prometheus.a (dependencies changed)... 2024-04-14T14:13:56.787 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:13:56 smithi122 ceph-mon[25721]: Reconfiguring daemon prometheus.a on smithi122 2024-04-14T14:13:56.787 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:13:56 smithi122 ceph-mon[25721]: pgmap v10: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 20 KiB/s rd, 0 B/s wr, 11 op/s 2024-04-14T14:13:56.817 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:13:56 smithi069 ceph-mon[24818]: Reconfiguring daemon prometheus.a on smithi122 2024-04-14T14:13:56.817 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:13:56 smithi069 ceph-mon[24818]: pgmap v10: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 20 KiB/s rd, 0 B/s wr, 11 op/s 2024-04-14T14:13:56.817 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 14 14:13:56 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-y[63038]: ::ffff:172.21.15.122 - - [14/Apr/2024:14:13:56] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-04-14T14:13:56.817 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:13:56 smithi069 ceph-mon[29472]: Reconfiguring daemon prometheus.a on smithi122 2024-04-14T14:13:56.817 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:13:56 smithi069 ceph-mon[29472]: pgmap v10: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 20 KiB/s rd, 0 B/s wr, 11 op/s 2024-04-14T14:13:58.214 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Apr 14 14:13:58 smithi122 systemd[1]: Stopping Ceph prometheus.a for dc326fc2-fa65-11ee-bc8e-c7b262605968... 2024-04-14T14:13:58.529 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Apr 14 14:13:58 smithi122 bash[57909]: Error: no container with name or ID "ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-prometheus.a" found: no such container 2024-04-14T14:13:58.530 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Apr 14 14:13:58 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-prometheus-a[55061]: ts=2024-04-14T14:13:58.357Z caller=main.go:775 level=warn msg="Received SIGTERM, exiting gracefully..." 2024-04-14T14:13:58.530 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Apr 14 14:13:58 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-prometheus-a[55061]: ts=2024-04-14T14:13:58.357Z caller=main.go:798 level=info msg="Stopping scrape discovery manager..." 2024-04-14T14:13:58.530 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Apr 14 14:13:58 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-prometheus-a[55061]: ts=2024-04-14T14:13:58.357Z caller=main.go:812 level=info msg="Stopping notify discovery manager..." 2024-04-14T14:13:58.530 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Apr 14 14:13:58 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-prometheus-a[55061]: ts=2024-04-14T14:13:58.357Z caller=main.go:834 level=info msg="Stopping scrape manager..." 2024-04-14T14:13:58.530 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Apr 14 14:13:58 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-prometheus-a[55061]: ts=2024-04-14T14:13:58.357Z caller=main.go:794 level=info msg="Scrape discovery manager stopped" 2024-04-14T14:13:58.530 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Apr 14 14:13:58 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-prometheus-a[55061]: ts=2024-04-14T14:13:58.357Z caller=main.go:808 level=info msg="Notify discovery manager stopped" 2024-04-14T14:13:58.530 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Apr 14 14:13:58 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-prometheus-a[55061]: ts=2024-04-14T14:13:58.357Z caller=manager.go:945 level=info component="rule manager" msg="Stopping rule manager..." 2024-04-14T14:13:58.530 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Apr 14 14:13:58 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-prometheus-a[55061]: ts=2024-04-14T14:13:58.357Z caller=main.go:828 level=info msg="Scrape manager stopped" 2024-04-14T14:13:58.530 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Apr 14 14:13:58 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-prometheus-a[55061]: ts=2024-04-14T14:13:58.357Z caller=manager.go:955 level=info component="rule manager" msg="Rule manager stopped" 2024-04-14T14:13:58.530 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Apr 14 14:13:58 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-prometheus-a[55061]: ts=2024-04-14T14:13:58.358Z caller=notifier.go:600 level=info component=notifier msg="Stopping notification manager..." 2024-04-14T14:13:58.531 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Apr 14 14:13:58 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-prometheus-a[55061]: ts=2024-04-14T14:13:58.358Z caller=main.go:1054 level=info msg="Notifier manager stopped" 2024-04-14T14:13:58.531 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Apr 14 14:13:58 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-prometheus-a[55061]: ts=2024-04-14T14:13:58.358Z caller=main.go:1066 level=info msg="See you next time!" 2024-04-14T14:13:58.531 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Apr 14 14:13:58 smithi122 podman[57922]: 2024-04-14 14:13:58.367491394 +0000 UTC m=+0.115313049 container died 9ebe35247d1a39f019e49a526e82fee7cb72172eb0477901dbef3a479dc4f20c (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-prometheus-a, maintainer=The Prometheus Authors ) 2024-04-14T14:13:58.817 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:13:58 smithi069 ceph-mon[24818]: pgmap v11: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 20 KiB/s rd, 0 B/s wr, 11 op/s 2024-04-14T14:13:58.817 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:13:58 smithi069 ceph-mon[24818]: from='mgr.24938 172.21.15.122:0/87298362' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T14:13:58.817 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:13:58 smithi069 ceph-mon[29472]: pgmap v11: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 20 KiB/s rd, 0 B/s wr, 11 op/s 2024-04-14T14:13:58.817 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:13:58 smithi069 ceph-mon[29472]: from='mgr.24938 172.21.15.122:0/87298362' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T14:13:58.845 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Apr 14 14:13:58 smithi122 podman[57922]: 2024-04-14 14:13:58.689605015 +0000 UTC m=+0.437426669 container cleanup 9ebe35247d1a39f019e49a526e82fee7cb72172eb0477901dbef3a479dc4f20c (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-prometheus-a, maintainer=The Prometheus Authors ) 2024-04-14T14:13:58.845 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Apr 14 14:13:58 smithi122 bash[57922]: ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-prometheus-a 2024-04-14T14:13:58.846 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Apr 14 14:13:58 smithi122 bash[57950]: Error: no container with name or ID "ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-prometheus.a" found: no such container 2024-04-14T14:13:58.846 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:13:58 smithi122 ceph-mon[25721]: pgmap v11: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 20 KiB/s rd, 0 B/s wr, 11 op/s 2024-04-14T14:13:58.846 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:13:58 smithi122 ceph-mon[25721]: from='mgr.24938 172.21.15.122:0/87298362' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T14:13:59.249 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Apr 14 14:13:58 smithi122 podman[57936]: 2024-04-14 14:13:58.891343369 +0000 UTC m=+0.526123313 container remove 9ebe35247d1a39f019e49a526e82fee7cb72172eb0477901dbef3a479dc4f20c (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-prometheus-a, maintainer=The Prometheus Authors ) 2024-04-14T14:13:59.249 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Apr 14 14:13:58 smithi122 systemd[1]: ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968@prometheus.a.service: Deactivated successfully. 2024-04-14T14:13:59.249 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Apr 14 14:13:58 smithi122 systemd[1]: Stopped Ceph prometheus.a for dc326fc2-fa65-11ee-bc8e-c7b262605968. 2024-04-14T14:13:59.249 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Apr 14 14:13:58 smithi122 systemd[1]: ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968@prometheus.a.service: Consumed 1.422s CPU time. 2024-04-14T14:13:59.249 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Apr 14 14:13:58 smithi122 systemd[1]: Starting Ceph prometheus.a for dc326fc2-fa65-11ee-bc8e-c7b262605968... 2024-04-14T14:13:59.249 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Apr 14 14:13:59 smithi122 podman[58018]: 2024-04-14 14:13:59.024768003 +0000 UTC m=+0.018384070 image pull 514e6a882f6e74806a5856468489eeff8d7106095557578da96935e4d0ba4d9d quay.io/prometheus/prometheus:v2.33.4 2024-04-14T14:13:59.249 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Apr 14 14:13:59 smithi122 podman[58018]: 2024-04-14 14:13:59.125591785 +0000 UTC m=+0.119207886 container create 012f5f9d7df590c1b959233887ee191cf2e883f6b040d0129d635034a550d3a4 (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-prometheus-a, maintainer=The Prometheus Authors ) 2024-04-14T14:13:59.587 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Apr 14 14:13:59 smithi122 podman[58018]: 2024-04-14 14:13:59.352779075 +0000 UTC m=+0.346395133 container init 012f5f9d7df590c1b959233887ee191cf2e883f6b040d0129d635034a550d3a4 (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-prometheus-a, maintainer=The Prometheus Authors ) 2024-04-14T14:13:59.587 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Apr 14 14:13:59 smithi122 podman[58018]: 2024-04-14 14:13:59.355397123 +0000 UTC m=+0.349013186 container start 012f5f9d7df590c1b959233887ee191cf2e883f6b040d0129d635034a550d3a4 (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-prometheus-a, maintainer=The Prometheus Authors ) 2024-04-14T14:13:59.587 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Apr 14 14:13:59 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-prometheus-a[58033]: ts=2024-04-14T14:13:59.378Z caller=main.go:475 level=info msg="No time or size retention was set so using the default time retention" duration=15d 2024-04-14T14:13:59.588 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Apr 14 14:13:59 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-prometheus-a[58033]: ts=2024-04-14T14:13:59.378Z caller=main.go:512 level=info msg="Starting Prometheus" version="(version=2.33.4, branch=HEAD, revision=83032011a5d3e6102624fe58241a374a7201fee8)" 2024-04-14T14:13:59.588 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Apr 14 14:13:59 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-prometheus-a[58033]: ts=2024-04-14T14:13:59.378Z caller=main.go:517 level=info build_context="(go=go1.17.7, user=root@d13bf69e7be8, date=20220222-16:51:28)" 2024-04-14T14:13:59.588 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Apr 14 14:13:59 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-prometheus-a[58033]: ts=2024-04-14T14:13:59.378Z caller=main.go:518 level=info host_details="(Linux 5.14.0-435.el9.x86_64 #1 SMP PREEMPT_DYNAMIC Fri Mar 29 18:46:37 UTC 2024 x86_64 smithi122 (none))" 2024-04-14T14:13:59.588 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Apr 14 14:13:59 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-prometheus-a[58033]: ts=2024-04-14T14:13:59.378Z caller=main.go:519 level=info fd_limits="(soft=1048576, hard=1048576)" 2024-04-14T14:13:59.588 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Apr 14 14:13:59 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-prometheus-a[58033]: ts=2024-04-14T14:13:59.378Z caller=main.go:520 level=info vm_limits="(soft=unlimited, hard=unlimited)" 2024-04-14T14:13:59.588 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Apr 14 14:13:59 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-prometheus-a[58033]: ts=2024-04-14T14:13:59.380Z caller=web.go:570 level=info component=web msg="Start listening for connections" address=:9095 2024-04-14T14:13:59.588 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Apr 14 14:13:59 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-prometheus-a[58033]: ts=2024-04-14T14:13:59.381Z caller=main.go:923 level=info msg="Starting TSDB ..." 2024-04-14T14:13:59.588 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Apr 14 14:13:59 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-prometheus-a[58033]: ts=2024-04-14T14:13:59.382Z caller=tls_config.go:195 level=info component=web msg="TLS is disabled." http2=false 2024-04-14T14:13:59.588 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Apr 14 14:13:59 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-prometheus-a[58033]: ts=2024-04-14T14:13:59.383Z caller=head.go:493 level=info component=tsdb msg="Replaying on-disk memory mappable chunks if any" 2024-04-14T14:13:59.588 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Apr 14 14:13:59 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-prometheus-a[58033]: ts=2024-04-14T14:13:59.385Z caller=head.go:527 level=info component=tsdb msg="On-disk memory mappable chunks replay completed" duration=1.488048ms 2024-04-14T14:13:59.588 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Apr 14 14:13:59 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-prometheus-a[58033]: ts=2024-04-14T14:13:59.385Z caller=head.go:533 level=info component=tsdb msg="Replaying WAL, this may take a while" 2024-04-14T14:13:59.589 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Apr 14 14:13:59 smithi122 bash[58018]: 012f5f9d7df590c1b959233887ee191cf2e883f6b040d0129d635034a550d3a4 2024-04-14T14:13:59.589 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Apr 14 14:13:59 smithi122 systemd[1]: Started Ceph prometheus.a for dc326fc2-fa65-11ee-bc8e-c7b262605968. 2024-04-14T14:13:59.589 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Apr 14 14:13:59 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-prometheus-a[58033]: ts=2024-04-14T14:13:59.585Z caller=head.go:604 level=info component=tsdb msg="WAL segment loaded" segment=0 maxSegment=4 2024-04-14T14:13:59.845 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Apr 14 14:13:59 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-prometheus-a[58033]: ts=2024-04-14T14:13:59.833Z caller=head.go:604 level=info component=tsdb msg="WAL segment loaded" segment=1 maxSegment=4 2024-04-14T14:14:00.345 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Apr 14 14:14:00 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-prometheus-a[58033]: ts=2024-04-14T14:14:00.058Z caller=head.go:604 level=info component=tsdb msg="WAL segment loaded" segment=2 maxSegment=4 2024-04-14T14:14:00.345 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Apr 14 14:14:00 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-prometheus-a[58033]: ts=2024-04-14T14:14:00.109Z caller=head.go:604 level=info component=tsdb msg="WAL segment loaded" segment=3 maxSegment=4 2024-04-14T14:14:00.345 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Apr 14 14:14:00 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-prometheus-a[58033]: ts=2024-04-14T14:14:00.109Z caller=head.go:604 level=info component=tsdb msg="WAL segment loaded" segment=4 maxSegment=4 2024-04-14T14:14:00.345 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Apr 14 14:14:00 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-prometheus-a[58033]: ts=2024-04-14T14:14:00.109Z caller=head.go:610 level=info component=tsdb msg="WAL replay completed" checkpoint_replay_duration=44.567µs wal_replay_duration=724.566556ms total_replay_duration=726.120811ms 2024-04-14T14:14:00.345 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Apr 14 14:14:00 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-prometheus-a[58033]: ts=2024-04-14T14:14:00.114Z caller=main.go:944 level=info fs_type=XFS_SUPER_MAGIC 2024-04-14T14:14:00.345 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Apr 14 14:14:00 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-prometheus-a[58033]: ts=2024-04-14T14:14:00.114Z caller=main.go:947 level=info msg="TSDB started" 2024-04-14T14:14:00.346 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Apr 14 14:14:00 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-prometheus-a[58033]: ts=2024-04-14T14:14:00.114Z caller=main.go:1128 level=info msg="Loading configuration file" filename=/etc/prometheus/prometheus.yml 2024-04-14T14:14:00.346 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Apr 14 14:14:00 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-prometheus-a[58033]: ts=2024-04-14T14:14:00.137Z caller=main.go:1165 level=info msg="Completed loading of configuration file" filename=/etc/prometheus/prometheus.yml totalDuration=22.770553ms db_storage=710ns remote_storage=1.299µs web_handler=378ns query_engine=1.009µs scrape=239.766µs scrape_sd=274.262µs notify=17.984µs notify_sd=14.315µs rules=21.9192ms 2024-04-14T14:14:00.346 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Apr 14 14:14:00 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-prometheus-a[58033]: ts=2024-04-14T14:14:00.137Z caller=main.go:896 level=info msg="Server is ready to receive web requests." 2024-04-14T14:14:00.817 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:14:00 smithi069 ceph-mon[24818]: pgmap v12: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 20 KiB/s rd, 0 B/s wr, 11 op/s 2024-04-14T14:14:00.817 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:14:00 smithi069 ceph-mon[24818]: from='mgr.24938 ' entity='mgr.x' 2024-04-14T14:14:00.817 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:14:00 smithi069 ceph-mon[24818]: from='mgr.24938 ' entity='mgr.x' 2024-04-14T14:14:00.817 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:14:00 smithi069 ceph-mon[24818]: from='mgr.24938 172.21.15.122:0/87298362' entity='mgr.x' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-04-14T14:14:00.817 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:14:00 smithi069 ceph-mon[24818]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-04-14T14:14:00.817 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:14:00 smithi069 ceph-mon[29472]: pgmap v12: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 20 KiB/s rd, 0 B/s wr, 11 op/s 2024-04-14T14:14:00.817 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:14:00 smithi069 ceph-mon[29472]: from='mgr.24938 ' entity='mgr.x' 2024-04-14T14:14:00.818 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:14:00 smithi069 ceph-mon[29472]: from='mgr.24938 ' entity='mgr.x' 2024-04-14T14:14:00.818 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:14:00 smithi069 ceph-mon[29472]: from='mgr.24938 172.21.15.122:0/87298362' entity='mgr.x' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-04-14T14:14:00.818 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:14:00 smithi069 ceph-mon[29472]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-04-14T14:14:00.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:14:00 smithi122 ceph-mon[25721]: pgmap v12: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 20 KiB/s rd, 0 B/s wr, 11 op/s 2024-04-14T14:14:00.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:14:00 smithi122 ceph-mon[25721]: from='mgr.24938 ' entity='mgr.x' 2024-04-14T14:14:00.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:14:00 smithi122 ceph-mon[25721]: from='mgr.24938 ' entity='mgr.x' 2024-04-14T14:14:00.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:14:00 smithi122 ceph-mon[25721]: from='mgr.24938 172.21.15.122:0/87298362' entity='mgr.x' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-04-14T14:14:00.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:14:00 smithi122 ceph-mon[25721]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-04-14T14:14:02.809 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:14:02 smithi122 ceph-mon[25721]: pgmap v13: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 482 B/s rd, 0 op/s 2024-04-14T14:14:02.817 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:14:02 smithi069 ceph-mon[24818]: pgmap v13: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 482 B/s rd, 0 op/s 2024-04-14T14:14:02.817 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:14:02 smithi069 ceph-mon[29472]: pgmap v13: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 482 B/s rd, 0 op/s 2024-04-14T14:14:04.583 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:14:04 smithi069 ceph-mon[24818]: pgmap v14: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 426 B/s rd, 0 op/s 2024-04-14T14:14:04.584 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:14:04 smithi069 ceph-mon[24818]: from='mgr.24938 ' entity='mgr.x' 2024-04-14T14:14:04.584 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:14:04 smithi069 ceph-mon[24818]: from='mgr.24938 ' entity='mgr.x' 2024-04-14T14:14:04.584 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:14:04 smithi069 ceph-mon[24818]: from='mgr.24938 172.21.15.122:0/87298362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T14:14:04.584 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:14:04 smithi069 ceph-mon[24818]: from='mgr.24938 172.21.15.122:0/87298362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-14T14:14:04.584 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:14:04 smithi069 ceph-mon[24818]: pgmap v15: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 506 B/s rd, 0 op/s 2024-04-14T14:14:04.584 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:14:04 smithi069 ceph-mon[24818]: pgmap v16: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail 2024-04-14T14:14:04.584 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:14:04 smithi069 ceph-mon[24818]: from='mgr.24938 ' entity='mgr.x' 2024-04-14T14:14:04.584 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:14:04 smithi069 ceph-mon[24818]: Reconfiguring iscsi.foo.smithi069.yuumtr (dependencies changed)... 2024-04-14T14:14:04.584 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:14:04 smithi069 ceph-mon[24818]: from='mgr.24938 172.21.15.122:0/87298362' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi069.yuumtr", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-04-14T14:14:04.584 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:14:04 smithi069 ceph-mon[24818]: from='mgr.24938 ' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi069.yuumtr", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-04-14T14:14:04.584 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:14:04 smithi069 ceph-mon[24818]: from='mgr.24938 172.21.15.122:0/87298362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T14:14:04.585 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:14:04 smithi069 ceph-mon[24818]: Reconfiguring daemon iscsi.foo.smithi069.yuumtr on smithi069 2024-04-14T14:14:04.585 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:14:04 smithi069 ceph-mon[29472]: pgmap v14: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 426 B/s rd, 0 op/s 2024-04-14T14:14:04.585 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:14:04 smithi069 ceph-mon[29472]: from='mgr.24938 ' entity='mgr.x' 2024-04-14T14:14:04.585 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:14:04 smithi069 ceph-mon[29472]: from='mgr.24938 ' entity='mgr.x' 2024-04-14T14:14:04.585 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:14:04 smithi069 ceph-mon[29472]: from='mgr.24938 172.21.15.122:0/87298362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T14:14:04.585 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:14:04 smithi069 ceph-mon[29472]: from='mgr.24938 172.21.15.122:0/87298362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-14T14:14:04.585 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:14:04 smithi069 ceph-mon[29472]: pgmap v15: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 506 B/s rd, 0 op/s 2024-04-14T14:14:04.585 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:14:04 smithi069 ceph-mon[29472]: pgmap v16: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail 2024-04-14T14:14:04.585 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:14:04 smithi069 ceph-mon[29472]: from='mgr.24938 ' entity='mgr.x' 2024-04-14T14:14:04.585 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:14:04 smithi069 ceph-mon[29472]: Reconfiguring iscsi.foo.smithi069.yuumtr (dependencies changed)... 2024-04-14T14:14:04.585 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:14:04 smithi069 ceph-mon[29472]: from='mgr.24938 172.21.15.122:0/87298362' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi069.yuumtr", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-04-14T14:14:04.585 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:14:04 smithi069 ceph-mon[29472]: from='mgr.24938 ' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi069.yuumtr", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-04-14T14:14:04.586 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:14:04 smithi069 ceph-mon[29472]: from='mgr.24938 172.21.15.122:0/87298362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T14:14:04.586 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:14:04 smithi069 ceph-mon[29472]: Reconfiguring daemon iscsi.foo.smithi069.yuumtr on smithi069 2024-04-14T14:14:04.595 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:14:04 smithi122 ceph-mon[25721]: pgmap v14: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 426 B/s rd, 0 op/s 2024-04-14T14:14:04.595 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:14:04 smithi122 ceph-mon[25721]: from='mgr.24938 ' entity='mgr.x' 2024-04-14T14:14:04.595 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:14:04 smithi122 ceph-mon[25721]: from='mgr.24938 ' entity='mgr.x' 2024-04-14T14:14:04.595 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:14:04 smithi122 ceph-mon[25721]: from='mgr.24938 172.21.15.122:0/87298362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T14:14:04.595 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:14:04 smithi122 ceph-mon[25721]: from='mgr.24938 172.21.15.122:0/87298362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-14T14:14:04.595 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:14:04 smithi122 ceph-mon[25721]: pgmap v15: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 506 B/s rd, 0 op/s 2024-04-14T14:14:04.596 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:14:04 smithi122 ceph-mon[25721]: pgmap v16: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail 2024-04-14T14:14:04.596 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:14:04 smithi122 ceph-mon[25721]: from='mgr.24938 ' entity='mgr.x' 2024-04-14T14:14:04.596 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:14:04 smithi122 ceph-mon[25721]: Reconfiguring iscsi.foo.smithi069.yuumtr (dependencies changed)... 2024-04-14T14:14:04.596 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:14:04 smithi122 ceph-mon[25721]: from='mgr.24938 172.21.15.122:0/87298362' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi069.yuumtr", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-04-14T14:14:04.596 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:14:04 smithi122 ceph-mon[25721]: from='mgr.24938 ' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi069.yuumtr", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-04-14T14:14:04.596 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:14:04 smithi122 ceph-mon[25721]: from='mgr.24938 172.21.15.122:0/87298362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T14:14:04.596 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:14:04 smithi122 ceph-mon[25721]: Reconfiguring daemon iscsi.foo.smithi069.yuumtr on smithi069 2024-04-14T14:14:06.638 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:14:06 smithi069 ceph-mon[24818]: pgmap v17: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail 2024-04-14T14:14:06.639 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:14:06 smithi069 ceph-mon[29472]: pgmap v17: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail 2024-04-14T14:14:06.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:14:06 smithi122 ceph-mon[25721]: pgmap v17: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail 2024-04-14T14:14:08.817 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:14:08 smithi069 ceph-mon[29472]: pgmap v18: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail 2024-04-14T14:14:08.817 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:14:08 smithi069 ceph-mon[29472]: from='mgr.24938 ' entity='mgr.x' 2024-04-14T14:14:08.817 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:14:08 smithi069 ceph-mon[29472]: from='mgr.24938 ' entity='mgr.x' 2024-04-14T14:14:08.817 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:14:08 smithi069 ceph-mon[29472]: from='mgr.24938 172.21.15.122:0/87298362' entity='mgr.x' cmd=[{"prefix": "dashboard iscsi-gateway-list"}]: dispatch 2024-04-14T14:14:08.817 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:14:08 smithi069 ceph-mon[29472]: from='mgr.24938 172.21.15.122:0/87298362' entity='mgr.x' cmd=[{"prefix": "dashboard set-iscsi-api-ssl-verification", "value": "true"}]: dispatch 2024-04-14T14:14:08.817 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:14:08 smithi069 ceph-mon[24818]: pgmap v18: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail 2024-04-14T14:14:08.818 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:14:08 smithi069 ceph-mon[24818]: from='mgr.24938 ' entity='mgr.x' 2024-04-14T14:14:08.818 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:14:08 smithi069 ceph-mon[24818]: from='mgr.24938 ' entity='mgr.x' 2024-04-14T14:14:08.818 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:14:08 smithi069 ceph-mon[24818]: from='mgr.24938 172.21.15.122:0/87298362' entity='mgr.x' cmd=[{"prefix": "dashboard iscsi-gateway-list"}]: dispatch 2024-04-14T14:14:08.818 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:14:08 smithi069 ceph-mon[24818]: from='mgr.24938 172.21.15.122:0/87298362' entity='mgr.x' cmd=[{"prefix": "dashboard set-iscsi-api-ssl-verification", "value": "true"}]: dispatch 2024-04-14T14:14:08.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:14:08 smithi122 ceph-mon[25721]: pgmap v18: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail 2024-04-14T14:14:08.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:14:08 smithi122 ceph-mon[25721]: from='mgr.24938 ' entity='mgr.x' 2024-04-14T14:14:08.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:14:08 smithi122 ceph-mon[25721]: from='mgr.24938 ' entity='mgr.x' 2024-04-14T14:14:08.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:14:08 smithi122 ceph-mon[25721]: from='mgr.24938 172.21.15.122:0/87298362' entity='mgr.x' cmd=[{"prefix": "dashboard iscsi-gateway-list"}]: dispatch 2024-04-14T14:14:08.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:14:08 smithi122 ceph-mon[25721]: from='mgr.24938 172.21.15.122:0/87298362' entity='mgr.x' cmd=[{"prefix": "dashboard set-iscsi-api-ssl-verification", "value": "true"}]: dispatch 2024-04-14T14:14:09.817 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:14:09 smithi069 ceph-mon[24818]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard iscsi-gateway-list"}]: dispatch 2024-04-14T14:14:09.817 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:14:09 smithi069 ceph-mon[24818]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-iscsi-api-ssl-verification", "value": "true"}]: dispatch 2024-04-14T14:14:09.817 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:14:09 smithi069 ceph-mon[24818]: from='client.? 172.21.15.69:0/495909408' entity='client.iscsi.foo.smithi069.yuumtr' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-04-14T14:14:09.817 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:14:09 smithi069 ceph-mon[24818]: from='client.? 172.21.15.69:0/4100809802' entity='client.iscsi.foo.smithi069.yuumtr' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.69:6800/281907388"}]: dispatch 2024-04-14T14:14:09.817 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:14:09 smithi069 ceph-mon[29472]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard iscsi-gateway-list"}]: dispatch 2024-04-14T14:14:09.818 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:14:09 smithi069 ceph-mon[29472]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-iscsi-api-ssl-verification", "value": "true"}]: dispatch 2024-04-14T14:14:09.818 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:14:09 smithi069 ceph-mon[29472]: from='client.? 172.21.15.69:0/495909408' entity='client.iscsi.foo.smithi069.yuumtr' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-04-14T14:14:09.818 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:14:09 smithi069 ceph-mon[29472]: from='client.? 172.21.15.69:0/4100809802' entity='client.iscsi.foo.smithi069.yuumtr' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.69:6800/281907388"}]: dispatch 2024-04-14T14:14:09.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:14:09 smithi122 ceph-mon[25721]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard iscsi-gateway-list"}]: dispatch 2024-04-14T14:14:09.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:14:09 smithi122 ceph-mon[25721]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-iscsi-api-ssl-verification", "value": "true"}]: dispatch 2024-04-14T14:14:09.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:14:09 smithi122 ceph-mon[25721]: from='client.? 172.21.15.69:0/495909408' entity='client.iscsi.foo.smithi069.yuumtr' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-04-14T14:14:09.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:14:09 smithi122 ceph-mon[25721]: from='client.? 172.21.15.69:0/4100809802' entity='client.iscsi.foo.smithi069.yuumtr' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.69:6800/281907388"}]: dispatch 2024-04-14T14:14:10.817 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:14:10 smithi069 ceph-mon[24818]: pgmap v19: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 126 B/s rd, 0 op/s 2024-04-14T14:14:10.817 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:14:10 smithi069 ceph-mon[24818]: from='client.? 172.21.15.69:0/4100809802' entity='client.iscsi.foo.smithi069.yuumtr' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.69:6800/281907388"}]': finished 2024-04-14T14:14:10.817 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:14:10 smithi069 ceph-mon[24818]: osdmap e104: 8 total, 8 up, 8 in 2024-04-14T14:14:10.817 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:14:10 smithi069 ceph-mon[24818]: from='client.? 172.21.15.69:0/3333766070' entity='client.iscsi.foo.smithi069.yuumtr' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.69:0/2364622569"}]: dispatch 2024-04-14T14:14:10.817 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:14:10 smithi069 ceph-mon[29472]: pgmap v19: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 126 B/s rd, 0 op/s 2024-04-14T14:14:10.817 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:14:10 smithi069 ceph-mon[29472]: from='client.? 172.21.15.69:0/4100809802' entity='client.iscsi.foo.smithi069.yuumtr' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.69:6800/281907388"}]': finished 2024-04-14T14:14:10.817 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:14:10 smithi069 ceph-mon[29472]: osdmap e104: 8 total, 8 up, 8 in 2024-04-14T14:14:10.817 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:14:10 smithi069 ceph-mon[29472]: from='client.? 172.21.15.69:0/3333766070' entity='client.iscsi.foo.smithi069.yuumtr' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.69:0/2364622569"}]: dispatch 2024-04-14T14:14:10.844 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:14:10 smithi122 ceph-mon[25721]: pgmap v19: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 126 B/s rd, 0 op/s 2024-04-14T14:14:10.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:14:10 smithi122 ceph-mon[25721]: from='client.? 172.21.15.69:0/4100809802' entity='client.iscsi.foo.smithi069.yuumtr' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.69:6800/281907388"}]': finished 2024-04-14T14:14:10.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:14:10 smithi122 ceph-mon[25721]: osdmap e104: 8 total, 8 up, 8 in 2024-04-14T14:14:10.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:14:10 smithi122 ceph-mon[25721]: from='client.? 172.21.15.69:0/3333766070' entity='client.iscsi.foo.smithi069.yuumtr' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.69:0/2364622569"}]: dispatch 2024-04-14T14:14:11.817 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:14:11 smithi069 ceph-mon[24818]: from='client.? 172.21.15.69:0/3333766070' entity='client.iscsi.foo.smithi069.yuumtr' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.69:0/2364622569"}]': finished 2024-04-14T14:14:11.817 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:14:11 smithi069 ceph-mon[24818]: osdmap e105: 8 total, 8 up, 8 in 2024-04-14T14:14:11.817 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:14:11 smithi069 ceph-mon[24818]: from='client.? 172.21.15.69:0/3816768936' entity='client.iscsi.foo.smithi069.yuumtr' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.69:0/2078564664"}]: dispatch 2024-04-14T14:14:11.817 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:14:11 smithi069 ceph-mon[29472]: from='client.? 172.21.15.69:0/3333766070' entity='client.iscsi.foo.smithi069.yuumtr' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.69:0/2364622569"}]': finished 2024-04-14T14:14:11.817 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:14:11 smithi069 ceph-mon[29472]: osdmap e105: 8 total, 8 up, 8 in 2024-04-14T14:14:11.817 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:14:11 smithi069 ceph-mon[29472]: from='client.? 172.21.15.69:0/3816768936' entity='client.iscsi.foo.smithi069.yuumtr' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.69:0/2078564664"}]: dispatch 2024-04-14T14:14:11.844 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:14:11 smithi122 ceph-mon[25721]: from='client.? 172.21.15.69:0/3333766070' entity='client.iscsi.foo.smithi069.yuumtr' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.69:0/2364622569"}]': finished 2024-04-14T14:14:11.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:14:11 smithi122 ceph-mon[25721]: osdmap e105: 8 total, 8 up, 8 in 2024-04-14T14:14:11.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:14:11 smithi122 ceph-mon[25721]: from='client.? 172.21.15.69:0/3816768936' entity='client.iscsi.foo.smithi069.yuumtr' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.69:0/2078564664"}]: dispatch 2024-04-14T14:14:12.817 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:14:12 smithi069 ceph-mon[24818]: pgmap v22: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 127 B/s rd, 0 op/s 2024-04-14T14:14:12.817 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:14:12 smithi069 ceph-mon[24818]: from='client.? 172.21.15.69:0/3816768936' entity='client.iscsi.foo.smithi069.yuumtr' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.69:0/2078564664"}]': finished 2024-04-14T14:14:12.817 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:14:12 smithi069 ceph-mon[24818]: osdmap e106: 8 total, 8 up, 8 in 2024-04-14T14:14:12.817 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:14:12 smithi069 ceph-mon[24818]: from='client.? 172.21.15.69:0/1617261988' entity='client.iscsi.foo.smithi069.yuumtr' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.69:0/723938454"}]: dispatch 2024-04-14T14:14:12.817 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:14:12 smithi069 ceph-mon[29472]: pgmap v22: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 127 B/s rd, 0 op/s 2024-04-14T14:14:12.817 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:14:12 smithi069 ceph-mon[29472]: from='client.? 172.21.15.69:0/3816768936' entity='client.iscsi.foo.smithi069.yuumtr' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.69:0/2078564664"}]': finished 2024-04-14T14:14:12.818 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:14:12 smithi069 ceph-mon[29472]: osdmap e106: 8 total, 8 up, 8 in 2024-04-14T14:14:12.818 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:14:12 smithi069 ceph-mon[29472]: from='client.? 172.21.15.69:0/1617261988' entity='client.iscsi.foo.smithi069.yuumtr' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.69:0/723938454"}]: dispatch 2024-04-14T14:14:12.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:14:12 smithi122 ceph-mon[25721]: pgmap v22: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 127 B/s rd, 0 op/s 2024-04-14T14:14:12.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:14:12 smithi122 ceph-mon[25721]: from='client.? 172.21.15.69:0/3816768936' entity='client.iscsi.foo.smithi069.yuumtr' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.69:0/2078564664"}]': finished 2024-04-14T14:14:12.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:14:12 smithi122 ceph-mon[25721]: osdmap e106: 8 total, 8 up, 8 in 2024-04-14T14:14:12.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:14:12 smithi122 ceph-mon[25721]: from='client.? 172.21.15.69:0/1617261988' entity='client.iscsi.foo.smithi069.yuumtr' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.69:0/723938454"}]: dispatch 2024-04-14T14:14:12.845 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Apr 14 14:14:12 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-x[45487]: ::ffff:172.21.15.122 - - [14/Apr/2024:14:14:12] "GET /metrics HTTP/1.1" 200 37981 "" "Prometheus/2.33.4" 2024-04-14T14:14:13.817 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:14:13 smithi069 ceph-mon[24818]: from='client.? 172.21.15.69:0/1617261988' entity='client.iscsi.foo.smithi069.yuumtr' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.69:0/723938454"}]': finished 2024-04-14T14:14:13.817 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:14:13 smithi069 ceph-mon[24818]: osdmap e107: 8 total, 8 up, 8 in 2024-04-14T14:14:13.817 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:14:13 smithi069 ceph-mon[24818]: from='mgr.24938 172.21.15.122:0/87298362' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T14:14:13.817 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:14:13 smithi069 ceph-mon[24818]: from='mgr.24938 ' entity='mgr.x' 2024-04-14T14:14:13.817 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:14:13 smithi069 ceph-mon[24818]: from='mgr.24938 ' entity='mgr.x' 2024-04-14T14:14:13.817 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:14:13 smithi069 ceph-mon[24818]: from='mgr.24938 172.21.15.122:0/87298362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T14:14:13.818 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:14:13 smithi069 ceph-mon[24818]: from='mgr.24938 172.21.15.122:0/87298362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-14T14:14:13.818 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:14:13 smithi069 ceph-mon[24818]: from='mgr.24938 ' entity='mgr.x' 2024-04-14T14:14:13.818 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:14:13 smithi069 ceph-mon[24818]: from='client.? 172.21.15.69:0/2390200620' entity='client.iscsi.foo.smithi069.yuumtr' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.69:0/1654773101"}]: dispatch 2024-04-14T14:14:13.818 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:14:13 smithi069 ceph-mon[24818]: from='client.? ' entity='client.iscsi.foo.smithi069.yuumtr' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.69:0/1654773101"}]: dispatch 2024-04-14T14:14:13.818 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:14:13 smithi069 ceph-mon[29472]: from='client.? 172.21.15.69:0/1617261988' entity='client.iscsi.foo.smithi069.yuumtr' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.69:0/723938454"}]': finished 2024-04-14T14:14:13.818 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:14:13 smithi069 ceph-mon[29472]: osdmap e107: 8 total, 8 up, 8 in 2024-04-14T14:14:13.818 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:14:13 smithi069 ceph-mon[29472]: from='mgr.24938 172.21.15.122:0/87298362' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T14:14:13.818 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:14:13 smithi069 ceph-mon[29472]: from='mgr.24938 ' entity='mgr.x' 2024-04-14T14:14:13.818 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:14:13 smithi069 ceph-mon[29472]: from='mgr.24938 ' entity='mgr.x' 2024-04-14T14:14:13.819 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:14:13 smithi069 ceph-mon[29472]: from='mgr.24938 172.21.15.122:0/87298362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T14:14:13.819 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:14:13 smithi069 ceph-mon[29472]: from='mgr.24938 172.21.15.122:0/87298362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-14T14:14:13.819 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:14:13 smithi069 ceph-mon[29472]: from='mgr.24938 ' entity='mgr.x' 2024-04-14T14:14:13.819 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:14:13 smithi069 ceph-mon[29472]: from='client.? 172.21.15.69:0/2390200620' entity='client.iscsi.foo.smithi069.yuumtr' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.69:0/1654773101"}]: dispatch 2024-04-14T14:14:13.819 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:14:13 smithi069 ceph-mon[29472]: from='client.? ' entity='client.iscsi.foo.smithi069.yuumtr' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.69:0/1654773101"}]: dispatch 2024-04-14T14:14:13.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:14:13 smithi122 ceph-mon[25721]: from='client.? 172.21.15.69:0/1617261988' entity='client.iscsi.foo.smithi069.yuumtr' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.69:0/723938454"}]': finished 2024-04-14T14:14:13.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:14:13 smithi122 ceph-mon[25721]: osdmap e107: 8 total, 8 up, 8 in 2024-04-14T14:14:13.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:14:13 smithi122 ceph-mon[25721]: from='mgr.24938 172.21.15.122:0/87298362' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T14:14:13.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:14:13 smithi122 ceph-mon[25721]: from='mgr.24938 ' entity='mgr.x' 2024-04-14T14:14:13.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:14:13 smithi122 ceph-mon[25721]: from='mgr.24938 ' entity='mgr.x' 2024-04-14T14:14:13.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:14:13 smithi122 ceph-mon[25721]: from='mgr.24938 172.21.15.122:0/87298362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T14:14:13.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:14:13 smithi122 ceph-mon[25721]: from='mgr.24938 172.21.15.122:0/87298362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-14T14:14:13.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:14:13 smithi122 ceph-mon[25721]: from='mgr.24938 ' entity='mgr.x' 2024-04-14T14:14:13.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:14:13 smithi122 ceph-mon[25721]: from='client.? 172.21.15.69:0/2390200620' entity='client.iscsi.foo.smithi069.yuumtr' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.69:0/1654773101"}]: dispatch 2024-04-14T14:14:13.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:14:13 smithi122 ceph-mon[25721]: from='client.? ' entity='client.iscsi.foo.smithi069.yuumtr' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.69:0/1654773101"}]: dispatch 2024-04-14T14:14:14.817 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:14:14 smithi069 ceph-mon[29472]: pgmap v25: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail 2024-04-14T14:14:14.817 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:14:14 smithi069 ceph-mon[29472]: pgmap v26: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail 2024-04-14T14:14:14.817 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:14:14 smithi069 ceph-mon[29472]: from='client.? ' entity='client.iscsi.foo.smithi069.yuumtr' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.69:0/1654773101"}]': finished 2024-04-14T14:14:14.817 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:14:14 smithi069 ceph-mon[29472]: osdmap e108: 8 total, 8 up, 8 in 2024-04-14T14:14:14.817 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:14:14 smithi069 ceph-mon[29472]: from='client.? 172.21.15.69:0/1065516435' entity='client.iscsi.foo.smithi069.yuumtr' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.69:6801/281907388"}]: dispatch 2024-04-14T14:14:14.817 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:14:14 smithi069 ceph-mon[29472]: from='client.? ' entity='client.iscsi.foo.smithi069.yuumtr' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.69:6801/281907388"}]: dispatch 2024-04-14T14:14:14.817 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:14:14 smithi069 ceph-mon[24818]: pgmap v25: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail 2024-04-14T14:14:14.818 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:14:14 smithi069 ceph-mon[24818]: pgmap v26: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail 2024-04-14T14:14:14.818 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:14:14 smithi069 ceph-mon[24818]: from='client.? ' entity='client.iscsi.foo.smithi069.yuumtr' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.69:0/1654773101"}]': finished 2024-04-14T14:14:14.818 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:14:14 smithi069 ceph-mon[24818]: osdmap e108: 8 total, 8 up, 8 in 2024-04-14T14:14:14.818 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:14:14 smithi069 ceph-mon[24818]: from='client.? 172.21.15.69:0/1065516435' entity='client.iscsi.foo.smithi069.yuumtr' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.69:6801/281907388"}]: dispatch 2024-04-14T14:14:14.818 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:14:14 smithi069 ceph-mon[24818]: from='client.? ' entity='client.iscsi.foo.smithi069.yuumtr' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.69:6801/281907388"}]: dispatch 2024-04-14T14:14:14.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:14:14 smithi122 ceph-mon[25721]: pgmap v25: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail 2024-04-14T14:14:14.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:14:14 smithi122 ceph-mon[25721]: pgmap v26: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail 2024-04-14T14:14:14.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:14:14 smithi122 ceph-mon[25721]: from='client.? ' entity='client.iscsi.foo.smithi069.yuumtr' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.69:0/1654773101"}]': finished 2024-04-14T14:14:14.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:14:14 smithi122 ceph-mon[25721]: osdmap e108: 8 total, 8 up, 8 in 2024-04-14T14:14:14.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:14:14 smithi122 ceph-mon[25721]: from='client.? 172.21.15.69:0/1065516435' entity='client.iscsi.foo.smithi069.yuumtr' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.69:6801/281907388"}]: dispatch 2024-04-14T14:14:14.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:14:14 smithi122 ceph-mon[25721]: from='client.? ' entity='client.iscsi.foo.smithi069.yuumtr' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.69:6801/281907388"}]: dispatch 2024-04-14T14:14:16.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:14:15 smithi122 ceph-mon[25721]: pgmap v28: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s 2024-04-14T14:14:16.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:14:15 smithi122 ceph-mon[25721]: from='client.? ' entity='client.iscsi.foo.smithi069.yuumtr' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.69:6801/281907388"}]': finished 2024-04-14T14:14:16.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:14:15 smithi122 ceph-mon[25721]: osdmap e109: 8 total, 8 up, 8 in 2024-04-14T14:14:16.317 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:14:15 smithi069 ceph-mon[24818]: pgmap v28: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s 2024-04-14T14:14:16.317 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:14:15 smithi069 ceph-mon[24818]: from='client.? ' entity='client.iscsi.foo.smithi069.yuumtr' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.69:6801/281907388"}]': finished 2024-04-14T14:14:16.317 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:14:15 smithi069 ceph-mon[24818]: osdmap e109: 8 total, 8 up, 8 in 2024-04-14T14:14:16.317 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:14:15 smithi069 ceph-mon[29472]: pgmap v28: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s 2024-04-14T14:14:16.317 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:14:15 smithi069 ceph-mon[29472]: from='client.? ' entity='client.iscsi.foo.smithi069.yuumtr' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.69:6801/281907388"}]': finished 2024-04-14T14:14:16.317 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:14:15 smithi069 ceph-mon[29472]: osdmap e109: 8 total, 8 up, 8 in 2024-04-14T14:14:18.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:14:17 smithi122 ceph-mon[25721]: pgmap v30: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:14:18.317 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:14:17 smithi069 ceph-mon[24818]: pgmap v30: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:14:18.317 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:14:17 smithi069 ceph-mon[29472]: pgmap v30: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:14:19.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:14:18 smithi122 ceph-mon[25721]: from='client.25273 -' entity='client.iscsi.foo.smithi069.yuumtr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T14:14:19.317 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:14:18 smithi069 ceph-mon[24818]: from='client.25273 -' entity='client.iscsi.foo.smithi069.yuumtr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T14:14:19.317 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:14:18 smithi069 ceph-mon[29472]: from='client.25273 -' entity='client.iscsi.foo.smithi069.yuumtr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T14:14:20.317 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:14:19 smithi069 ceph-mon[24818]: pgmap v31: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-04-14T14:14:20.317 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:14:19 smithi069 ceph-mon[29472]: pgmap v31: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-04-14T14:14:20.345 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:14:19 smithi122 ceph-mon[25721]: pgmap v31: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-04-14T14:14:22.802 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:14:22 smithi122 ceph-mon[25721]: pgmap v32: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:14:22.817 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:14:22 smithi069 ceph-mon[24818]: pgmap v32: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:14:22.817 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:14:22 smithi069 ceph-mon[29472]: pgmap v32: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:14:23.095 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Apr 14 14:14:22 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-x[45487]: ::ffff:172.21.15.122 - - [14/Apr/2024:14:14:22] "GET /metrics HTTP/1.1" 200 37981 "" "Prometheus/2.33.4" 2024-04-14T14:14:24.817 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:14:24 smithi069 ceph-mon[24818]: pgmap v33: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 569 B/s rd, 0 op/s 2024-04-14T14:14:24.817 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:14:24 smithi069 ceph-mon[29472]: pgmap v33: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 569 B/s rd, 0 op/s 2024-04-14T14:14:24.844 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:14:24 smithi122 ceph-mon[25721]: pgmap v33: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 569 B/s rd, 0 op/s 2024-04-14T14:14:26.817 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:14:26 smithi069 ceph-mon[24818]: pgmap v34: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-14T14:14:26.817 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:14:26 smithi069 ceph-mon[29472]: pgmap v34: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-14T14:14:26.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:14:26 smithi122 ceph-mon[25721]: pgmap v34: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-14T14:14:28.817 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:14:28 smithi069 ceph-mon[24818]: pgmap v35: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:14:28.817 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:14:28 smithi069 ceph-mon[24818]: from='mgr.24938 172.21.15.122:0/87298362' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T14:14:28.817 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:14:28 smithi069 ceph-mon[29472]: pgmap v35: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:14:28.817 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:14:28 smithi069 ceph-mon[29472]: from='mgr.24938 172.21.15.122:0/87298362' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T14:14:28.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:14:28 smithi122 ceph-mon[25721]: pgmap v35: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:14:28.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:14:28 smithi122 ceph-mon[25721]: from='mgr.24938 172.21.15.122:0/87298362' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T14:14:29.817 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:14:29 smithi069 ceph-mon[24818]: from='client.25273 -' entity='client.iscsi.foo.smithi069.yuumtr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T14:14:29.817 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:14:29 smithi069 ceph-mon[29472]: from='client.25273 -' entity='client.iscsi.foo.smithi069.yuumtr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T14:14:29.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:14:29 smithi122 ceph-mon[25721]: from='client.25273 -' entity='client.iscsi.foo.smithi069.yuumtr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T14:14:30.817 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:14:30 smithi069 ceph-mon[24818]: pgmap v36: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:14:30.817 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:14:30 smithi069 ceph-mon[29472]: pgmap v36: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:14:30.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:14:30 smithi122 ceph-mon[25721]: pgmap v36: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:14:32.817 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:14:32 smithi069 ceph-mon[24818]: pgmap v37: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:14:32.817 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:14:32 smithi069 ceph-mon[29472]: pgmap v37: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:14:32.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:14:32 smithi122 ceph-mon[25721]: pgmap v37: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:14:32.845 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Apr 14 14:14:32 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-x[45487]: ::ffff:172.21.15.122 - - [14/Apr/2024:14:14:32] "GET /metrics HTTP/1.1" 200 37983 "" "Prometheus/2.33.4" 2024-04-14T14:14:34.817 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:14:34 smithi069 ceph-mon[24818]: pgmap v38: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:14:34.817 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:14:34 smithi069 ceph-mon[29472]: pgmap v38: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:14:34.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:14:34 smithi122 ceph-mon[25721]: pgmap v38: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:14:36.817 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:14:36 smithi069 ceph-mon[24818]: pgmap v39: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:14:36.817 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:14:36 smithi069 ceph-mon[29472]: pgmap v39: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:14:36.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:14:36 smithi122 ceph-mon[25721]: pgmap v39: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:14:38.817 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:14:38 smithi069 ceph-mon[24818]: pgmap v40: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:14:38.817 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:14:38 smithi069 ceph-mon[29472]: pgmap v40: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:14:38.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:14:38 smithi122 ceph-mon[25721]: pgmap v40: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:14:39.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:14:39 smithi122 ceph-mon[25721]: from='client.25273 -' entity='client.iscsi.foo.smithi069.yuumtr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T14:14:40.067 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:14:39 smithi069 ceph-mon[24818]: from='client.25273 -' entity='client.iscsi.foo.smithi069.yuumtr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T14:14:40.067 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:14:39 smithi069 ceph-mon[29472]: from='client.25273 -' entity='client.iscsi.foo.smithi069.yuumtr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T14:14:40.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:14:40 smithi122 ceph-mon[25721]: pgmap v41: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:14:41.067 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:14:40 smithi069 ceph-mon[24818]: pgmap v41: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:14:41.067 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:14:40 smithi069 ceph-mon[29472]: pgmap v41: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:14:42.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:14:42 smithi122 ceph-mon[25721]: pgmap v42: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:14:42.845 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Apr 14 14:14:42 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-x[45487]: ::ffff:172.21.15.122 - - [14/Apr/2024:14:14:42] "GET /metrics HTTP/1.1" 200 37982 "" "Prometheus/2.33.4" 2024-04-14T14:14:43.067 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:14:42 smithi069 ceph-mon[24818]: pgmap v42: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:14:43.067 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:14:42 smithi069 ceph-mon[29472]: pgmap v42: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:14:43.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:14:43 smithi122 ceph-mon[25721]: from='mgr.24938 172.21.15.122:0/87298362' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T14:14:44.067 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:14:43 smithi069 ceph-mon[24818]: from='mgr.24938 172.21.15.122:0/87298362' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T14:14:44.067 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:14:43 smithi069 ceph-mon[29472]: from='mgr.24938 172.21.15.122:0/87298362' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T14:14:44.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:14:44 smithi122 ceph-mon[25721]: pgmap v43: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:14:45.067 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:14:44 smithi069 ceph-mon[24818]: pgmap v43: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:14:45.067 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:14:44 smithi069 ceph-mon[29472]: pgmap v43: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:14:46.817 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:14:46 smithi069 ceph-mon[24818]: pgmap v44: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:14:46.817 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:14:46 smithi069 ceph-mon[29472]: pgmap v44: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:14:46.844 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:14:46 smithi122 ceph-mon[25721]: pgmap v44: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:14:48.817 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:14:48 smithi069 ceph-mon[24818]: pgmap v45: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:14:48.817 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:14:48 smithi069 ceph-mon[29472]: pgmap v45: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:14:48.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:14:48 smithi122 ceph-mon[25721]: pgmap v45: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:14:49.817 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:14:49 smithi069 ceph-mon[24818]: from='client.25273 -' entity='client.iscsi.foo.smithi069.yuumtr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T14:14:49.817 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:14:49 smithi069 ceph-mon[29472]: from='client.25273 -' entity='client.iscsi.foo.smithi069.yuumtr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T14:14:49.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:14:49 smithi122 ceph-mon[25721]: from='client.25273 -' entity='client.iscsi.foo.smithi069.yuumtr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T14:14:50.817 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:14:50 smithi069 ceph-mon[24818]: pgmap v46: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:14:50.817 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:14:50 smithi069 ceph-mon[29472]: pgmap v46: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:14:50.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:14:50 smithi122 ceph-mon[25721]: pgmap v46: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:14:52.817 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:14:52 smithi069 ceph-mon[24818]: pgmap v47: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:14:52.817 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:14:52 smithi069 ceph-mon[29472]: pgmap v47: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:14:52.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:14:52 smithi122 ceph-mon[25721]: pgmap v47: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:14:52.845 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Apr 14 14:14:52 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-x[45487]: ::ffff:172.21.15.122 - - [14/Apr/2024:14:14:52] "GET /metrics HTTP/1.1" 200 37982 "" "Prometheus/2.33.4" 2024-04-14T14:14:54.817 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:14:54 smithi069 ceph-mon[24818]: pgmap v48: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:14:54.817 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:14:54 smithi069 ceph-mon[29472]: pgmap v48: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:14:54.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:14:54 smithi122 ceph-mon[25721]: pgmap v48: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:14:56.817 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:14:56 smithi069 ceph-mon[24818]: pgmap v49: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:14:56.817 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:14:56 smithi069 ceph-mon[29472]: pgmap v49: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:14:56.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:14:56 smithi122 ceph-mon[25721]: pgmap v49: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:14:58.817 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:14:58 smithi069 ceph-mon[24818]: pgmap v50: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:14:58.817 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:14:58 smithi069 ceph-mon[24818]: from='mgr.24938 172.21.15.122:0/87298362' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T14:14:58.817 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:14:58 smithi069 ceph-mon[29472]: pgmap v50: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:14:58.817 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:14:58 smithi069 ceph-mon[29472]: from='mgr.24938 172.21.15.122:0/87298362' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T14:14:58.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:14:58 smithi122 ceph-mon[25721]: pgmap v50: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:14:58.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:14:58 smithi122 ceph-mon[25721]: from='mgr.24938 172.21.15.122:0/87298362' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T14:14:59.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:14:59 smithi122 ceph-mon[25721]: from='client.25273 -' entity='client.iscsi.foo.smithi069.yuumtr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T14:15:00.067 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:14:59 smithi069 ceph-mon[24818]: from='client.25273 -' entity='client.iscsi.foo.smithi069.yuumtr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T14:15:00.067 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:14:59 smithi069 ceph-mon[29472]: from='client.25273 -' entity='client.iscsi.foo.smithi069.yuumtr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T14:15:00.844 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:15:00 smithi122 ceph-mon[25721]: pgmap v51: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:15:01.067 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:15:00 smithi069 ceph-mon[24818]: pgmap v51: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:15:01.067 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:15:00 smithi069 ceph-mon[29472]: pgmap v51: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:15:02.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:15:02 smithi122 ceph-mon[25721]: pgmap v52: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:15:02.845 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Apr 14 14:15:02 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-x[45487]: ::ffff:172.21.15.122 - - [14/Apr/2024:14:15:02] "GET /metrics HTTP/1.1" 200 37981 "" "Prometheus/2.33.4" 2024-04-14T14:15:03.067 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:15:02 smithi069 ceph-mon[24818]: pgmap v52: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:15:03.067 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:15:02 smithi069 ceph-mon[29472]: pgmap v52: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:15:04.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:15:04 smithi122 ceph-mon[25721]: pgmap v53: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:15:05.067 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:15:04 smithi069 ceph-mon[24818]: pgmap v53: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:15:05.068 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:15:04 smithi069 ceph-mon[29472]: pgmap v53: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:15:06.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:15:06 smithi122 ceph-mon[25721]: pgmap v54: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:15:07.067 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:15:06 smithi069 ceph-mon[24818]: pgmap v54: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:15:07.067 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:15:06 smithi069 ceph-mon[29472]: pgmap v54: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:15:08.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:15:08 smithi122 ceph-mon[25721]: pgmap v55: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:15:09.067 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:15:08 smithi069 ceph-mon[24818]: pgmap v55: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:15:09.067 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:15:08 smithi069 ceph-mon[29472]: pgmap v55: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:15:09.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:15:09 smithi122 ceph-mon[25721]: from='client.25273 -' entity='client.iscsi.foo.smithi069.yuumtr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T14:15:10.067 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:15:09 smithi069 ceph-mon[24818]: from='client.25273 -' entity='client.iscsi.foo.smithi069.yuumtr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T14:15:10.067 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:15:09 smithi069 ceph-mon[29472]: from='client.25273 -' entity='client.iscsi.foo.smithi069.yuumtr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T14:15:10.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:15:10 smithi122 ceph-mon[25721]: pgmap v56: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:15:11.067 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:15:10 smithi069 ceph-mon[24818]: pgmap v56: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:15:11.067 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:15:10 smithi069 ceph-mon[29472]: pgmap v56: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:15:12.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:15:12 smithi122 ceph-mon[25721]: pgmap v57: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:15:12.845 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Apr 14 14:15:12 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-x[45487]: ::ffff:172.21.15.122 - - [14/Apr/2024:14:15:12] "GET /metrics HTTP/1.1" 200 37979 "" "Prometheus/2.33.4" 2024-04-14T14:15:13.067 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:15:12 smithi069 ceph-mon[24818]: pgmap v57: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:15:13.067 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:15:12 smithi069 ceph-mon[29472]: pgmap v57: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:15:14.070 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:15:13 smithi069 ceph-mon[24818]: from='mgr.24938 172.21.15.122:0/87298362' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T14:15:14.070 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:15:13 smithi069 ceph-mon[29472]: from='mgr.24938 172.21.15.122:0/87298362' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T14:15:14.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:15:13 smithi122 ceph-mon[25721]: from='mgr.24938 172.21.15.122:0/87298362' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T14:15:15.067 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:15:14 smithi069 ceph-mon[24818]: pgmap v58: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:15:15.067 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:15:14 smithi069 ceph-mon[24818]: from='mgr.24938 172.21.15.122:0/87298362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T14:15:15.067 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:15:14 smithi069 ceph-mon[24818]: from='mgr.24938 172.21.15.122:0/87298362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-14T14:15:15.067 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:15:14 smithi069 ceph-mon[24818]: pgmap v59: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 952 B/s rd, 0 op/s 2024-04-14T14:15:15.067 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:15:14 smithi069 ceph-mon[24818]: pgmap v60: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 584 B/s rd, 0 op/s 2024-04-14T14:15:15.067 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:15:14 smithi069 ceph-mon[24818]: from='mgr.24938 ' entity='mgr.x' 2024-04-14T14:15:15.068 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:15:14 smithi069 ceph-mon[29472]: pgmap v58: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:15:15.068 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:15:14 smithi069 ceph-mon[29472]: from='mgr.24938 172.21.15.122:0/87298362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T14:15:15.068 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:15:14 smithi069 ceph-mon[29472]: from='mgr.24938 172.21.15.122:0/87298362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-14T14:15:15.068 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:15:14 smithi069 ceph-mon[29472]: pgmap v59: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 952 B/s rd, 0 op/s 2024-04-14T14:15:15.068 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:15:14 smithi069 ceph-mon[29472]: pgmap v60: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 584 B/s rd, 0 op/s 2024-04-14T14:15:15.068 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:15:14 smithi069 ceph-mon[29472]: from='mgr.24938 ' entity='mgr.x' 2024-04-14T14:15:15.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:15:14 smithi122 ceph-mon[25721]: pgmap v58: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:15:15.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:15:14 smithi122 ceph-mon[25721]: from='mgr.24938 172.21.15.122:0/87298362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T14:15:15.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:15:14 smithi122 ceph-mon[25721]: from='mgr.24938 172.21.15.122:0/87298362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-14T14:15:15.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:15:14 smithi122 ceph-mon[25721]: pgmap v59: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 952 B/s rd, 0 op/s 2024-04-14T14:15:15.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:15:14 smithi122 ceph-mon[25721]: pgmap v60: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 584 B/s rd, 0 op/s 2024-04-14T14:15:15.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:15:14 smithi122 ceph-mon[25721]: from='mgr.24938 ' entity='mgr.x' 2024-04-14T14:15:17.817 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:15:17 smithi069 ceph-mon[24818]: pgmap v61: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-04-14T14:15:17.817 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:15:17 smithi069 ceph-mon[29472]: pgmap v61: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-04-14T14:15:17.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:15:17 smithi122 ceph-mon[25721]: pgmap v61: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-04-14T14:15:19.817 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:15:19 smithi069 ceph-mon[24818]: pgmap v62: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 584 B/s rd, 0 op/s 2024-04-14T14:15:19.817 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:15:19 smithi069 ceph-mon[24818]: from='client.25273 -' entity='client.iscsi.foo.smithi069.yuumtr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T14:15:19.817 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:15:19 smithi069 ceph-mon[29472]: pgmap v62: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 584 B/s rd, 0 op/s 2024-04-14T14:15:19.817 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:15:19 smithi069 ceph-mon[29472]: from='client.25273 -' entity='client.iscsi.foo.smithi069.yuumtr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T14:15:19.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:15:19 smithi122 ceph-mon[25721]: pgmap v62: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 584 B/s rd, 0 op/s 2024-04-14T14:15:19.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:15:19 smithi122 ceph-mon[25721]: from='client.25273 -' entity='client.iscsi.foo.smithi069.yuumtr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T14:15:21.817 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:15:21 smithi069 ceph-mon[24818]: pgmap v63: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-04-14T14:15:21.817 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:15:21 smithi069 ceph-mon[29472]: pgmap v63: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-04-14T14:15:21.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:15:21 smithi122 ceph-mon[25721]: pgmap v63: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-04-14T14:15:23.095 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Apr 14 14:15:22 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-x[45487]: ::ffff:172.21.15.122 - - [14/Apr/2024:14:15:22] "GET /metrics HTTP/1.1" 200 37979 "" "Prometheus/2.33.4" 2024-04-14T14:15:23.817 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:15:23 smithi069 ceph-mon[24818]: pgmap v64: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-04-14T14:15:23.817 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:15:23 smithi069 ceph-mon[29472]: pgmap v64: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-04-14T14:15:23.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:15:23 smithi122 ceph-mon[25721]: pgmap v64: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-04-14T14:15:25.817 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:15:25 smithi069 ceph-mon[24818]: pgmap v65: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-14T14:15:25.817 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:15:25 smithi069 ceph-mon[29472]: pgmap v65: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-14T14:15:25.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:15:25 smithi122 ceph-mon[25721]: pgmap v65: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-14T14:15:27.817 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:15:27 smithi069 ceph-mon[24818]: pgmap v66: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:15:27.817 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:15:27 smithi069 ceph-mon[29472]: pgmap v66: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:15:27.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:15:27 smithi122 ceph-mon[25721]: pgmap v66: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:15:28.817 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:15:28 smithi069 ceph-mon[24818]: from='mgr.24938 172.21.15.122:0/87298362' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T14:15:28.818 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:15:28 smithi069 ceph-mon[29472]: from='mgr.24938 172.21.15.122:0/87298362' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T14:15:28.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:15:28 smithi122 ceph-mon[25721]: from='mgr.24938 172.21.15.122:0/87298362' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T14:15:29.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:15:29 smithi122 ceph-mon[25721]: pgmap v67: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-04-14T14:15:29.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:15:29 smithi122 ceph-mon[25721]: from='client.25273 -' entity='client.iscsi.foo.smithi069.yuumtr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T14:15:30.067 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:15:29 smithi069 ceph-mon[24818]: pgmap v67: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-04-14T14:15:30.067 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:15:29 smithi069 ceph-mon[24818]: from='client.25273 -' entity='client.iscsi.foo.smithi069.yuumtr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T14:15:30.067 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:15:29 smithi069 ceph-mon[29472]: pgmap v67: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-04-14T14:15:30.067 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:15:29 smithi069 ceph-mon[29472]: from='client.25273 -' entity='client.iscsi.foo.smithi069.yuumtr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T14:15:31.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:15:31 smithi122 ceph-mon[25721]: pgmap v68: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:15:32.067 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:15:31 smithi069 ceph-mon[24818]: pgmap v68: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:15:32.067 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:15:31 smithi069 ceph-mon[29472]: pgmap v68: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:15:33.095 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Apr 14 14:15:32 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-x[45487]: ::ffff:172.21.15.122 - - [14/Apr/2024:14:15:32] "GET /metrics HTTP/1.1" 200 37981 "" "Prometheus/2.33.4" 2024-04-14T14:15:33.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:15:33 smithi122 ceph-mon[25721]: pgmap v69: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-04-14T14:15:34.067 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:15:33 smithi069 ceph-mon[24818]: pgmap v69: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-04-14T14:15:34.067 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:15:33 smithi069 ceph-mon[29472]: pgmap v69: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-04-14T14:15:35.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:15:35 smithi122 ceph-mon[25721]: pgmap v70: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-04-14T14:15:36.067 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:15:35 smithi069 ceph-mon[24818]: pgmap v70: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-04-14T14:15:36.067 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:15:35 smithi069 ceph-mon[29472]: pgmap v70: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-04-14T14:15:37.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:15:37 smithi122 ceph-mon[25721]: pgmap v71: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:15:38.067 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:15:37 smithi069 ceph-mon[24818]: pgmap v71: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:15:38.067 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:15:37 smithi069 ceph-mon[29472]: pgmap v71: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:15:39.846 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:15:39 smithi122 ceph-mon[25721]: pgmap v72: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:15:40.067 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:15:39 smithi069 ceph-mon[24818]: pgmap v72: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:15:40.067 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:15:39 smithi069 ceph-mon[29472]: pgmap v72: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:15:41.070 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:15:40 smithi069 ceph-mon[24818]: from='client.25273 -' entity='client.iscsi.foo.smithi069.yuumtr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T14:15:41.070 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:15:40 smithi069 ceph-mon[29472]: from='client.25273 -' entity='client.iscsi.foo.smithi069.yuumtr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T14:15:41.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:15:40 smithi122 ceph-mon[25721]: from='client.25273 -' entity='client.iscsi.foo.smithi069.yuumtr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T14:15:42.067 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:15:41 smithi069 ceph-mon[24818]: pgmap v73: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:15:42.067 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:15:41 smithi069 ceph-mon[29472]: pgmap v73: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:15:42.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:15:41 smithi122 ceph-mon[25721]: pgmap v73: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:15:43.067 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:15:42 smithi069 ceph-mon[24818]: pgmap v74: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:15:43.067 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:15:42 smithi069 ceph-mon[29472]: pgmap v74: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:15:43.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:15:42 smithi122 ceph-mon[25721]: pgmap v74: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:15:43.095 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Apr 14 14:15:42 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-x[45487]: ::ffff:172.21.15.122 - - [14/Apr/2024:14:15:42] "GET /metrics HTTP/1.1" 200 37983 "" "Prometheus/2.33.4" 2024-04-14T14:15:44.067 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:15:43 smithi069 ceph-mon[24818]: from='mgr.24938 172.21.15.122:0/87298362' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T14:15:44.067 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:15:43 smithi069 ceph-mon[29472]: from='mgr.24938 172.21.15.122:0/87298362' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T14:15:44.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:15:43 smithi122 ceph-mon[25721]: from='mgr.24938 172.21.15.122:0/87298362' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T14:15:45.067 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:15:44 smithi069 ceph-mon[24818]: pgmap v75: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:15:45.067 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:15:44 smithi069 ceph-mon[29472]: pgmap v75: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:15:45.094 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:15:44 smithi122 ceph-mon[25721]: pgmap v75: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:15:47.817 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:15:47 smithi069 ceph-mon[24818]: pgmap v76: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:15:47.817 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:15:47 smithi069 ceph-mon[29472]: pgmap v76: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:15:47.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:15:47 smithi122 ceph-mon[25721]: pgmap v76: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:15:49.817 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:15:49 smithi069 ceph-mon[24818]: pgmap v77: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:15:49.817 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:15:49 smithi069 ceph-mon[29472]: pgmap v77: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:15:49.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:15:49 smithi122 ceph-mon[25721]: pgmap v77: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:15:50.817 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:15:50 smithi069 ceph-mon[24818]: from='client.25273 -' entity='client.iscsi.foo.smithi069.yuumtr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T14:15:50.817 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:15:50 smithi069 ceph-mon[29472]: from='client.25273 -' entity='client.iscsi.foo.smithi069.yuumtr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T14:15:50.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:15:50 smithi122 ceph-mon[25721]: from='client.25273 -' entity='client.iscsi.foo.smithi069.yuumtr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T14:15:51.817 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:15:51 smithi069 ceph-mon[24818]: pgmap v78: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:15:51.817 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:15:51 smithi069 ceph-mon[29472]: pgmap v78: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:15:51.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:15:51 smithi122 ceph-mon[25721]: pgmap v78: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:15:53.095 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Apr 14 14:15:52 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-x[45487]: ::ffff:172.21.15.122 - - [14/Apr/2024:14:15:52] "GET /metrics HTTP/1.1" 200 37983 "" "Prometheus/2.33.4" 2024-04-14T14:15:53.817 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:15:53 smithi069 ceph-mon[24818]: pgmap v79: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:15:53.817 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:15:53 smithi069 ceph-mon[29472]: pgmap v79: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:15:53.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:15:53 smithi122 ceph-mon[25721]: pgmap v79: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:15:55.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:15:55 smithi122 ceph-mon[25721]: pgmap v80: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:15:56.067 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:15:55 smithi069 ceph-mon[24818]: pgmap v80: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:15:56.067 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:15:55 smithi069 ceph-mon[29472]: pgmap v80: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:15:57.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:15:57 smithi122 ceph-mon[25721]: pgmap v81: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:15:58.067 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:15:57 smithi069 ceph-mon[24818]: pgmap v81: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:15:58.067 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:15:57 smithi069 ceph-mon[29472]: pgmap v81: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:15:58.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:15:58 smithi122 ceph-mon[25721]: from='mgr.24938 172.21.15.122:0/87298362' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T14:15:59.067 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:15:58 smithi069 ceph-mon[24818]: from='mgr.24938 172.21.15.122:0/87298362' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T14:15:59.067 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:15:58 smithi069 ceph-mon[29472]: from='mgr.24938 172.21.15.122:0/87298362' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T14:15:59.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:15:59 smithi122 ceph-mon[25721]: pgmap v82: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:16:00.067 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:15:59 smithi069 ceph-mon[24818]: pgmap v82: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:16:00.067 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:15:59 smithi069 ceph-mon[29472]: pgmap v82: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:16:00.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:16:00 smithi122 ceph-mon[25721]: from='client.25273 -' entity='client.iscsi.foo.smithi069.yuumtr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T14:16:01.067 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:16:00 smithi069 ceph-mon[24818]: from='client.25273 -' entity='client.iscsi.foo.smithi069.yuumtr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T14:16:01.067 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:16:00 smithi069 ceph-mon[29472]: from='client.25273 -' entity='client.iscsi.foo.smithi069.yuumtr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T14:16:01.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:16:01 smithi122 ceph-mon[25721]: pgmap v83: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:16:02.067 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:16:01 smithi069 ceph-mon[24818]: pgmap v83: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:16:02.067 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:16:01 smithi069 ceph-mon[29472]: pgmap v83: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:16:03.095 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Apr 14 14:16:02 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-x[45487]: ::ffff:172.21.15.122 - - [14/Apr/2024:14:16:02] "GET /metrics HTTP/1.1" 200 37993 "" "Prometheus/2.33.4" 2024-04-14T14:16:03.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:16:03 smithi122 ceph-mon[25721]: pgmap v84: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:16:04.067 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:16:03 smithi069 ceph-mon[24818]: pgmap v84: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:16:04.067 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:16:03 smithi069 ceph-mon[29472]: pgmap v84: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:16:05.847 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:16:05 smithi122 ceph-mon[25721]: pgmap v85: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:16:06.067 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:16:05 smithi069 ceph-mon[24818]: pgmap v85: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:16:06.067 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:16:05 smithi069 ceph-mon[29472]: pgmap v85: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:16:07.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:16:07 smithi122 ceph-mon[25721]: pgmap v86: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:16:08.067 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:16:07 smithi069 ceph-mon[24818]: pgmap v86: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:16:08.067 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:16:07 smithi069 ceph-mon[29472]: pgmap v86: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:16:10.067 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:16:09 smithi069 ceph-mon[24818]: pgmap v87: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:16:10.067 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:16:09 smithi069 ceph-mon[29472]: pgmap v87: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:16:10.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:16:09 smithi122 ceph-mon[25721]: pgmap v87: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:16:11.067 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:16:10 smithi069 ceph-mon[24818]: from='client.25273 -' entity='client.iscsi.foo.smithi069.yuumtr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T14:16:11.067 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:16:10 smithi069 ceph-mon[29472]: from='client.25273 -' entity='client.iscsi.foo.smithi069.yuumtr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T14:16:11.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:16:10 smithi122 ceph-mon[25721]: from='client.25273 -' entity='client.iscsi.foo.smithi069.yuumtr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T14:16:12.067 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:16:11 smithi069 ceph-mon[24818]: pgmap v88: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:16:12.068 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:16:11 smithi069 ceph-mon[29472]: pgmap v88: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:16:12.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:16:11 smithi122 ceph-mon[25721]: pgmap v88: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:16:13.067 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:16:12 smithi069 ceph-mon[24818]: pgmap v89: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:16:13.067 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:16:12 smithi069 ceph-mon[29472]: pgmap v89: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:16:13.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:16:12 smithi122 ceph-mon[25721]: pgmap v89: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:16:13.095 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Apr 14 14:16:12 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-x[45487]: ::ffff:172.21.15.122 - - [14/Apr/2024:14:16:12] "GET /metrics HTTP/1.1" 200 37991 "" "Prometheus/2.33.4" 2024-04-14T14:16:14.067 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:16:13 smithi069 ceph-mon[24818]: from='mgr.24938 172.21.15.122:0/87298362' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T14:16:14.067 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:16:13 smithi069 ceph-mon[29472]: from='mgr.24938 172.21.15.122:0/87298362' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T14:16:14.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:16:13 smithi122 ceph-mon[25721]: from='mgr.24938 172.21.15.122:0/87298362' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T14:16:15.067 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:16:14 smithi069 ceph-mon[24818]: pgmap v90: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:16:15.067 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:16:14 smithi069 ceph-mon[24818]: from='mgr.24938 172.21.15.122:0/87298362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T14:16:15.067 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:16:14 smithi069 ceph-mon[24818]: from='mgr.24938 172.21.15.122:0/87298362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-14T14:16:15.067 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:16:14 smithi069 ceph-mon[24818]: pgmap v91: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s 2024-04-14T14:16:15.067 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:16:14 smithi069 ceph-mon[24818]: pgmap v92: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:16:15.067 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:16:14 smithi069 ceph-mon[24818]: from='mgr.24938 ' entity='mgr.x' 2024-04-14T14:16:15.067 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:16:14 smithi069 ceph-mon[29472]: pgmap v90: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:16:15.068 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:16:14 smithi069 ceph-mon[29472]: from='mgr.24938 172.21.15.122:0/87298362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T14:16:15.068 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:16:14 smithi069 ceph-mon[29472]: from='mgr.24938 172.21.15.122:0/87298362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-14T14:16:15.068 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:16:14 smithi069 ceph-mon[29472]: pgmap v91: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s 2024-04-14T14:16:15.068 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:16:14 smithi069 ceph-mon[29472]: pgmap v92: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:16:15.068 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:16:14 smithi069 ceph-mon[29472]: from='mgr.24938 ' entity='mgr.x' 2024-04-14T14:16:15.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:16:14 smithi122 ceph-mon[25721]: pgmap v90: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:16:15.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:16:14 smithi122 ceph-mon[25721]: from='mgr.24938 172.21.15.122:0/87298362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T14:16:15.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:16:14 smithi122 ceph-mon[25721]: from='mgr.24938 172.21.15.122:0/87298362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-14T14:16:15.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:16:14 smithi122 ceph-mon[25721]: pgmap v91: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s 2024-04-14T14:16:15.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:16:14 smithi122 ceph-mon[25721]: pgmap v92: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:16:15.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:16:14 smithi122 ceph-mon[25721]: from='mgr.24938 ' entity='mgr.x' 2024-04-14T14:16:17.817 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:16:17 smithi069 ceph-mon[24818]: pgmap v93: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:16:17.817 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:16:17 smithi069 ceph-mon[29472]: pgmap v93: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:16:17.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:16:17 smithi122 ceph-mon[25721]: pgmap v93: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:16:19.817 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:16:19 smithi069 ceph-mon[24818]: pgmap v94: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 590 B/s rd, 0 op/s 2024-04-14T14:16:19.817 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:16:19 smithi069 ceph-mon[29472]: pgmap v94: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 590 B/s rd, 0 op/s 2024-04-14T14:16:19.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:16:19 smithi122 ceph-mon[25721]: pgmap v94: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 590 B/s rd, 0 op/s 2024-04-14T14:16:20.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:16:20 smithi122 ceph-mon[25721]: from='client.25273 -' entity='client.iscsi.foo.smithi069.yuumtr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T14:16:21.067 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:16:20 smithi069 ceph-mon[24818]: from='client.25273 -' entity='client.iscsi.foo.smithi069.yuumtr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T14:16:21.067 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:16:20 smithi069 ceph-mon[29472]: from='client.25273 -' entity='client.iscsi.foo.smithi069.yuumtr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T14:16:21.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:16:21 smithi122 ceph-mon[25721]: pgmap v95: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:16:22.067 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:16:21 smithi069 ceph-mon[24818]: pgmap v95: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:16:22.067 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:16:21 smithi069 ceph-mon[29472]: pgmap v95: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:16:23.095 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Apr 14 14:16:22 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-x[45487]: ::ffff:172.21.15.122 - - [14/Apr/2024:14:16:22] "GET /metrics HTTP/1.1" 200 37991 "" "Prometheus/2.33.4" 2024-04-14T14:16:23.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:16:23 smithi122 ceph-mon[25721]: pgmap v96: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:16:24.067 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:16:23 smithi069 ceph-mon[24818]: pgmap v96: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:16:24.067 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:16:23 smithi069 ceph-mon[29472]: pgmap v96: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:16:25.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:16:25 smithi122 ceph-mon[25721]: pgmap v97: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-14T14:16:26.067 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:16:25 smithi069 ceph-mon[24818]: pgmap v97: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-14T14:16:26.067 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:16:25 smithi069 ceph-mon[29472]: pgmap v97: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-14T14:16:27.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:16:27 smithi122 ceph-mon[25721]: pgmap v98: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:16:28.067 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:16:27 smithi069 ceph-mon[24818]: pgmap v98: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:16:28.067 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:16:27 smithi069 ceph-mon[29472]: pgmap v98: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:16:28.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:16:28 smithi122 ceph-mon[25721]: from='mgr.24938 172.21.15.122:0/87298362' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T14:16:29.067 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:16:28 smithi069 ceph-mon[24818]: from='mgr.24938 172.21.15.122:0/87298362' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T14:16:29.067 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:16:28 smithi069 ceph-mon[29472]: from='mgr.24938 172.21.15.122:0/87298362' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T14:16:29.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:16:29 smithi122 ceph-mon[25721]: pgmap v99: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:16:30.067 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:16:29 smithi069 ceph-mon[24818]: pgmap v99: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:16:30.067 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:16:29 smithi069 ceph-mon[29472]: pgmap v99: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:16:31.067 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:16:30 smithi069 ceph-mon[24818]: from='client.25273 -' entity='client.iscsi.foo.smithi069.yuumtr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T14:16:31.067 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:16:30 smithi069 ceph-mon[29472]: from='client.25273 -' entity='client.iscsi.foo.smithi069.yuumtr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T14:16:31.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:16:30 smithi122 ceph-mon[25721]: from='client.25273 -' entity='client.iscsi.foo.smithi069.yuumtr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T14:16:32.067 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:16:31 smithi069 ceph-mon[24818]: pgmap v100: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:16:32.068 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:16:31 smithi069 ceph-mon[29472]: pgmap v100: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:16:32.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:16:31 smithi122 ceph-mon[25721]: pgmap v100: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:16:33.095 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Apr 14 14:16:32 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-x[45487]: ::ffff:172.21.15.122 - - [14/Apr/2024:14:16:32] "GET /metrics HTTP/1.1" 200 37993 "" "Prometheus/2.33.4" 2024-04-14T14:16:34.067 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:16:33 smithi069 ceph-mon[24818]: pgmap v101: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:16:34.067 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:16:33 smithi069 ceph-mon[29472]: pgmap v101: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:16:34.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:16:33 smithi122 ceph-mon[25721]: pgmap v101: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:16:36.067 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:16:35 smithi069 ceph-mon[24818]: pgmap v102: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:16:36.067 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:16:35 smithi069 ceph-mon[29472]: pgmap v102: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:16:36.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:16:35 smithi122 ceph-mon[25721]: pgmap v102: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:16:37.067 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:16:36 smithi069 ceph-mon[24818]: pgmap v103: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:16:37.067 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:16:36 smithi069 ceph-mon[29472]: pgmap v103: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:16:37.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:16:36 smithi122 ceph-mon[25721]: pgmap v103: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:16:39.817 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:16:39 smithi069 ceph-mon[24818]: pgmap v104: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:16:39.817 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:16:39 smithi069 ceph-mon[29472]: pgmap v104: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:16:39.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:16:39 smithi122 ceph-mon[25721]: pgmap v104: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:16:40.817 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:16:40 smithi069 ceph-mon[24818]: from='client.25273 -' entity='client.iscsi.foo.smithi069.yuumtr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T14:16:40.817 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:16:40 smithi069 ceph-mon[29472]: from='client.25273 -' entity='client.iscsi.foo.smithi069.yuumtr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T14:16:40.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:16:40 smithi122 ceph-mon[25721]: from='client.25273 -' entity='client.iscsi.foo.smithi069.yuumtr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T14:16:41.817 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:16:41 smithi069 ceph-mon[24818]: pgmap v105: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:16:41.817 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:16:41 smithi069 ceph-mon[29472]: pgmap v105: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:16:41.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:16:41 smithi122 ceph-mon[25721]: pgmap v105: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:16:43.095 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Apr 14 14:16:42 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-x[45487]: ::ffff:172.21.15.122 - - [14/Apr/2024:14:16:42] "GET /metrics HTTP/1.1" 200 37991 "" "Prometheus/2.33.4" 2024-04-14T14:16:43.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:16:43 smithi122 ceph-mon[25721]: pgmap v106: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:16:43.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:16:43 smithi122 ceph-mon[25721]: from='mgr.24938 172.21.15.122:0/87298362' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T14:16:44.067 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:16:43 smithi069 ceph-mon[24818]: pgmap v106: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:16:44.067 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:16:43 smithi069 ceph-mon[24818]: from='mgr.24938 172.21.15.122:0/87298362' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T14:16:44.067 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:16:43 smithi069 ceph-mon[29472]: pgmap v106: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:16:44.067 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:16:43 smithi069 ceph-mon[29472]: from='mgr.24938 172.21.15.122:0/87298362' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T14:16:45.700 DEBUG:teuthology.orchestra.run.smithi069:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid dc326fc2-fa65-11ee-bc8e-c7b262605968 -e sha1=a971fd5f5719038c70b817bafacd6d5400b1d888 -- bash -c 'ceph orch ps' 2024-04-14T14:16:45.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:16:45 smithi122 ceph-mon[25721]: pgmap v107: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:16:46.011 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:16:45 smithi069 ceph-mon[24818]: pgmap v107: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:16:46.012 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:16:45 smithi069 ceph-mon[29472]: pgmap v107: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:16:47.558 INFO:teuthology.orchestra.run.smithi069.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-14T14:16:47.558 INFO:teuthology.orchestra.run.smithi069.stdout:alertmanager.a smithi069 *:9093,9094 running (12m) 2m ago 20m 22.5M - 0.23.0 ba2b418f427c e300fc36e997 2024-04-14T14:16:47.558 INFO:teuthology.orchestra.run.smithi069.stdout:grafana.a smithi122 *:3000 error 2m ago 19m - - 2024-04-14T14:16:47.558 INFO:teuthology.orchestra.run.smithi069.stdout:iscsi.foo.smithi069.yuumtr smithi069 running (2m) 2m ago 19m 54.3M - 3.5 e1d6a67b021e 65932c077132 2024-04-14T14:16:47.558 INFO:teuthology.orchestra.run.smithi069.stdout:mgr.x smithi122 *:8443,9283 running (15m) 2m ago 24m 497M - 19.0.0-2778-ga971fd5f b96c509eef84 813744c2b615 2024-04-14T14:16:47.558 INFO:teuthology.orchestra.run.smithi069.stdout:mgr.y smithi069 *:8443,9283,8765 running (9m) 2m ago 26m 449M - 19.0.0-2778-ga971fd5f b96c509eef84 7af3d29d0e5f 2024-04-14T14:16:47.558 INFO:teuthology.orchestra.run.smithi069.stdout:mon.a smithi069 running (26m) 2m ago 26m 69.6M 2048M 17.2.0 e1d6a67b021e d3c553f91913 2024-04-14T14:16:47.558 INFO:teuthology.orchestra.run.smithi069.stdout:mon.b smithi122 running (24m) 2m ago 24m 55.6M 2048M 17.2.0 e1d6a67b021e 3b766bfce4be 2024-04-14T14:16:47.558 INFO:teuthology.orchestra.run.smithi069.stdout:mon.c smithi069 running (25m) 2m ago 25m 51.1M 2048M 17.2.0 e1d6a67b021e 23c078b9d1e5 2024-04-14T14:16:47.558 INFO:teuthology.orchestra.run.smithi069.stdout:node-exporter.a smithi069 *:9100 running (12m) 2m ago 20m 20.6M - 1.3.1 1dbe0e931976 ca074142ab73 2024-04-14T14:16:47.559 INFO:teuthology.orchestra.run.smithi069.stdout:node-exporter.b smithi122 *:9100 running (12m) 2m ago 20m 19.1M - 1.3.1 1dbe0e931976 996115353a0b 2024-04-14T14:16:47.559 INFO:teuthology.orchestra.run.smithi069.stdout:osd.0 smithi069 running (24m) 2m ago 24m 56.1M 2992M 17.2.0 e1d6a67b021e 46654654b6b6 2024-04-14T14:16:47.559 INFO:teuthology.orchestra.run.smithi069.stdout:osd.1 smithi069 running (24m) 2m ago 24m 53.0M 2992M 17.2.0 e1d6a67b021e cdba54a59609 2024-04-14T14:16:47.559 INFO:teuthology.orchestra.run.smithi069.stdout:osd.2 smithi069 running (23m) 2m ago 23m 50.3M 2992M 17.2.0 e1d6a67b021e bc37a1a9cba0 2024-04-14T14:16:47.559 INFO:teuthology.orchestra.run.smithi069.stdout:osd.3 smithi069 running (23m) 2m ago 23m 54.9M 2992M 17.2.0 e1d6a67b021e 0925ba3da801 2024-04-14T14:16:47.559 INFO:teuthology.orchestra.run.smithi069.stdout:osd.4 smithi122 running (22m) 2m ago 22m 53.1M 3248M 17.2.0 e1d6a67b021e c00975caf771 2024-04-14T14:16:47.559 INFO:teuthology.orchestra.run.smithi069.stdout:osd.5 smithi122 running (22m) 2m ago 22m 54.1M 3248M 17.2.0 e1d6a67b021e 5f52d217ceb8 2024-04-14T14:16:47.559 INFO:teuthology.orchestra.run.smithi069.stdout:osd.6 smithi122 running (22m) 2m ago 22m 49.8M 3248M 17.2.0 e1d6a67b021e a2acfec61475 2024-04-14T14:16:47.559 INFO:teuthology.orchestra.run.smithi069.stdout:osd.7 smithi122 running (21m) 2m ago 21m 55.3M 3248M 17.2.0 e1d6a67b021e 62411cd06801 2024-04-14T14:16:47.559 INFO:teuthology.orchestra.run.smithi069.stdout:prometheus.a smithi122 *:9095 running (2m) 2m ago 21m 42.2M - 2.33.4 514e6a882f6e 012f5f9d7df5 2024-04-14T14:16:47.559 INFO:teuthology.orchestra.run.smithi069.stdout:rgw.foo.smithi069.wesedz smithi069 *:8000 running (19m) 2m ago 19m 76.7M - 17.2.0 e1d6a67b021e 16de491440f3 2024-04-14T14:16:47.559 INFO:teuthology.orchestra.run.smithi069.stdout:rgw.foo.smithi122.jiobyv smithi122 *:8000 running (19m) 2m ago 19m 73.7M - 17.2.0 e1d6a67b021e 6d16de445dba 2024-04-14T14:16:47.817 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:16:47 smithi069 ceph-mon[24818]: pgmap v108: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:16:47.817 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:16:47 smithi069 ceph-mon[29472]: pgmap v108: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:16:47.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:16:47 smithi122 ceph-mon[25721]: pgmap v108: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:16:48.172 DEBUG:teuthology.orchestra.run.smithi069:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid dc326fc2-fa65-11ee-bc8e-c7b262605968 -e sha1=a971fd5f5719038c70b817bafacd6d5400b1d888 -- bash -c 'ceph orch ls' 2024-04-14T14:16:48.761 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:16:48 smithi069 ceph-mon[24818]: from='client.15420 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T14:16:48.761 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:16:48 smithi069 ceph-mon[29472]: from='client.15420 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T14:16:48.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:16:48 smithi122 ceph-mon[25721]: from='client.15420 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T14:16:49.669 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:16:49 smithi069 ceph-mon[24818]: pgmap v109: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:16:49.669 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:16:49 smithi069 ceph-mon[29472]: pgmap v109: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:16:49.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:16:49 smithi122 ceph-mon[25721]: pgmap v109: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:16:50.230 INFO:teuthology.orchestra.run.smithi069.stdout:NAME PORTS RUNNING REFRESHED AGE PLACEMENT 2024-04-14T14:16:50.231 INFO:teuthology.orchestra.run.smithi069.stdout:alertmanager ?:9093,9094 1/1 2m ago 21m smithi069=a;count:1 2024-04-14T14:16:50.231 INFO:teuthology.orchestra.run.smithi069.stdout:grafana ?:3000 0/1 2m ago 21m smithi122=a;count:1 2024-04-14T14:16:50.231 INFO:teuthology.orchestra.run.smithi069.stdout:iscsi.foo ?:5000 1/1 2m ago 19m count:1 2024-04-14T14:16:50.231 INFO:teuthology.orchestra.run.smithi069.stdout:mgr 2/2 2m ago 24m smithi069=y;smithi122=x;count:2 2024-04-14T14:16:50.231 INFO:teuthology.orchestra.run.smithi069.stdout:mon 3/3 2m ago 25m smithi069:172.21.15.69=a;smithi069:[v2:172.21.15.69:3301,v1:172.21.15.69:6790]=c;smithi122:172.21.15.122=b;count:3 2024-04-14T14:16:50.231 INFO:teuthology.orchestra.run.smithi069.stdout:node-exporter ?:9100 2/2 2m ago 21m smithi069=a;smithi122=b;count:2 2024-04-14T14:16:50.231 INFO:teuthology.orchestra.run.smithi069.stdout:osd 8 2m ago - 2024-04-14T14:16:50.231 INFO:teuthology.orchestra.run.smithi069.stdout:prometheus ?:9095 1/1 2m ago 21m smithi122=a;count:1 2024-04-14T14:16:50.231 INFO:teuthology.orchestra.run.smithi069.stdout:rgw.foo ?:8000 2/2 2m ago 19m count:2 2024-04-14T14:16:50.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:16:50 smithi122 ceph-mon[25721]: from='client.25273 -' entity='client.iscsi.foo.smithi069.yuumtr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T14:16:50.872 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:16:50 smithi069 ceph-mon[24818]: from='client.25273 -' entity='client.iscsi.foo.smithi069.yuumtr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T14:16:50.873 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:16:50 smithi069 ceph-mon[29472]: from='client.25273 -' entity='client.iscsi.foo.smithi069.yuumtr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T14:16:50.896 DEBUG:teuthology.orchestra.run.smithi069:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid dc326fc2-fa65-11ee-bc8e-c7b262605968 -e sha1=a971fd5f5719038c70b817bafacd6d5400b1d888 -- bash -c 'ceph versions' 2024-04-14T14:16:51.817 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:16:51 smithi069 ceph-mon[24818]: from='client.15426 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T14:16:51.817 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:16:51 smithi069 ceph-mon[24818]: pgmap v110: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:16:51.817 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:16:51 smithi069 ceph-mon[29472]: from='client.15426 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T14:16:51.817 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:16:51 smithi069 ceph-mon[29472]: pgmap v110: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:16:52.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:16:51 smithi122 ceph-mon[25721]: from='client.15426 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T14:16:52.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:16:51 smithi122 ceph-mon[25721]: pgmap v110: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:16:52.915 INFO:teuthology.orchestra.run.smithi069.stdout:{ 2024-04-14T14:16:52.915 INFO:teuthology.orchestra.run.smithi069.stdout: "mon": { 2024-04-14T14:16:52.915 INFO:teuthology.orchestra.run.smithi069.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 3 2024-04-14T14:16:52.915 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-14T14:16:52.915 INFO:teuthology.orchestra.run.smithi069.stdout: "mgr": { 2024-04-14T14:16:52.915 INFO:teuthology.orchestra.run.smithi069.stdout: "ceph version 19.0.0-2778-ga971fd5f (a971fd5f5719038c70b817bafacd6d5400b1d888) squid (dev)": 2 2024-04-14T14:16:52.915 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-14T14:16:52.915 INFO:teuthology.orchestra.run.smithi069.stdout: "osd": { 2024-04-14T14:16:52.915 INFO:teuthology.orchestra.run.smithi069.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 8 2024-04-14T14:16:52.915 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-14T14:16:52.915 INFO:teuthology.orchestra.run.smithi069.stdout: "mds": {}, 2024-04-14T14:16:52.916 INFO:teuthology.orchestra.run.smithi069.stdout: "rgw": { 2024-04-14T14:16:52.916 INFO:teuthology.orchestra.run.smithi069.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-04-14T14:16:52.916 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-14T14:16:52.916 INFO:teuthology.orchestra.run.smithi069.stdout: "overall": { 2024-04-14T14:16:52.916 INFO:teuthology.orchestra.run.smithi069.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 13, 2024-04-14T14:16:52.916 INFO:teuthology.orchestra.run.smithi069.stdout: "ceph version 19.0.0-2778-ga971fd5f (a971fd5f5719038c70b817bafacd6d5400b1d888) squid (dev)": 2 2024-04-14T14:16:52.916 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-14T14:16:52.916 INFO:teuthology.orchestra.run.smithi069.stdout:} 2024-04-14T14:16:53.095 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Apr 14 14:16:52 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-x[45487]: ::ffff:172.21.15.122 - - [14/Apr/2024:14:16:52] "GET /metrics HTTP/1.1" 200 37991 "" "Prometheus/2.33.4" 2024-04-14T14:16:53.560 DEBUG:teuthology.orchestra.run.smithi069:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid dc326fc2-fa65-11ee-bc8e-c7b262605968 -e sha1=a971fd5f5719038c70b817bafacd6d5400b1d888 -- bash -c 'ceph orch upgrade start --image quay.ceph.io/ceph-ci/ceph:$sha1 --daemon-types mgr' 2024-04-14T14:16:53.817 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:16:53 smithi069 ceph-mon[24818]: pgmap v111: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:16:53.817 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:16:53 smithi069 ceph-mon[24818]: from='client.? 172.21.15.69:0/1632328288' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T14:16:53.817 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:16:53 smithi069 ceph-mon[29472]: pgmap v111: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:16:53.817 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:16:53 smithi069 ceph-mon[29472]: from='client.? 172.21.15.69:0/1632328288' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T14:16:54.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:16:53 smithi122 ceph-mon[25721]: pgmap v111: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:16:54.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:16:53 smithi122 ceph-mon[25721]: from='client.? 172.21.15.69:0/1632328288' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T14:16:55.897 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:16:55 smithi122 ceph-mon[25721]: pgmap v112: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:16:56.041 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:16:55 smithi069 ceph-mon[24818]: pgmap v112: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:16:56.041 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:16:55 smithi069 ceph-mon[29472]: pgmap v112: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:16:56.886 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:16:56 smithi069 ceph-mon[24818]: from='client.15438 -' entity='client.admin' cmd=[{"prefix": "orch upgrade start", "image": "quay.ceph.io/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888", "daemon_types": "mgr", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T14:16:56.886 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:16:56 smithi069 ceph-mon[29472]: from='client.15438 -' entity='client.admin' cmd=[{"prefix": "orch upgrade start", "image": "quay.ceph.io/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888", "daemon_types": "mgr", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T14:16:56.936 INFO:teuthology.orchestra.run.smithi069.stdout:Initiating upgrade to quay.ceph.io/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888 2024-04-14T14:16:57.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:16:56 smithi122 ceph-mon[25721]: from='client.15438 -' entity='client.admin' cmd=[{"prefix": "orch upgrade start", "image": "quay.ceph.io/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888", "daemon_types": "mgr", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T14:16:57.596 DEBUG:teuthology.orchestra.run.smithi069:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid dc326fc2-fa65-11ee-bc8e-c7b262605968 -e sha1=a971fd5f5719038c70b817bafacd6d5400b1d888 -- bash -c 'while ceph orch upgrade status | jq '"'"'.in_progress'"'"' | grep true && ! ceph orch upgrade status | jq '"'"'.message'"'"' | grep Error ; do ceph orch ps ; ceph versions ; ceph orch upgrade status ; sleep 30 ; done' 2024-04-14T14:16:57.865 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:16:57 smithi069 ceph-mon[24818]: pgmap v113: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:16:57.865 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:16:57 smithi069 ceph-mon[24818]: from='mgr.24938 ' entity='mgr.x' 2024-04-14T14:16:57.865 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:16:57 smithi069 ceph-mon[24818]: from='mgr.24938 172.21.15.122:0/87298362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T14:16:57.865 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:16:57 smithi069 ceph-mon[24818]: from='mgr.24938 172.21.15.122:0/87298362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-14T14:16:57.865 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:16:57 smithi069 ceph-mon[24818]: from='mgr.24938 ' entity='mgr.x' 2024-04-14T14:16:57.866 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:16:57 smithi069 ceph-mon[29472]: pgmap v113: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:16:57.866 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:16:57 smithi069 ceph-mon[29472]: from='mgr.24938 ' entity='mgr.x' 2024-04-14T14:16:57.866 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:16:57 smithi069 ceph-mon[29472]: from='mgr.24938 172.21.15.122:0/87298362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T14:16:57.866 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:16:57 smithi069 ceph-mon[29472]: from='mgr.24938 172.21.15.122:0/87298362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-14T14:16:57.866 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:16:57 smithi069 ceph-mon[29472]: from='mgr.24938 ' entity='mgr.x' 2024-04-14T14:16:58.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:16:57 smithi122 ceph-mon[25721]: pgmap v113: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:16:58.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:16:57 smithi122 ceph-mon[25721]: from='mgr.24938 ' entity='mgr.x' 2024-04-14T14:16:58.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:16:57 smithi122 ceph-mon[25721]: from='mgr.24938 172.21.15.122:0/87298362' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T14:16:58.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:16:57 smithi122 ceph-mon[25721]: from='mgr.24938 172.21.15.122:0/87298362' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-14T14:16:58.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:16:57 smithi122 ceph-mon[25721]: from='mgr.24938 ' entity='mgr.x' 2024-04-14T14:16:58.671 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:16:58 smithi069 ceph-mon[29472]: Upgrade: Started with target quay.ceph.io/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888 2024-04-14T14:16:58.671 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:16:58 smithi069 ceph-mon[29472]: pgmap v114: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 962 B/s rd, 0 op/s 2024-04-14T14:16:58.671 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:16:58 smithi069 ceph-mon[29472]: pgmap v115: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:16:58.671 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:16:58 smithi069 ceph-mon[29472]: Upgrade: First pull of quay.ceph.io/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888 2024-04-14T14:16:58.671 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:16:58 smithi069 ceph-mon[29472]: from='mgr.24938 172.21.15.122:0/87298362' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T14:16:58.672 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:16:58 smithi069 ceph-mon[24818]: Upgrade: Started with target quay.ceph.io/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888 2024-04-14T14:16:58.672 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:16:58 smithi069 ceph-mon[24818]: pgmap v114: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 962 B/s rd, 0 op/s 2024-04-14T14:16:58.672 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:16:58 smithi069 ceph-mon[24818]: pgmap v115: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:16:58.672 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:16:58 smithi069 ceph-mon[24818]: Upgrade: First pull of quay.ceph.io/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888 2024-04-14T14:16:58.672 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:16:58 smithi069 ceph-mon[24818]: from='mgr.24938 172.21.15.122:0/87298362' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T14:16:59.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:16:58 smithi122 ceph-mon[25721]: Upgrade: Started with target quay.ceph.io/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888 2024-04-14T14:16:59.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:16:58 smithi122 ceph-mon[25721]: pgmap v114: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 962 B/s rd, 0 op/s 2024-04-14T14:16:59.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:16:58 smithi122 ceph-mon[25721]: pgmap v115: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:16:59.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:16:58 smithi122 ceph-mon[25721]: Upgrade: First pull of quay.ceph.io/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888 2024-04-14T14:16:59.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:16:58 smithi122 ceph-mon[25721]: from='mgr.24938 172.21.15.122:0/87298362' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T14:17:00.067 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:16:59 smithi069 ceph-mon[24818]: from='client.25273 -' entity='client.iscsi.foo.smithi069.yuumtr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T14:17:00.067 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:16:59 smithi069 ceph-mon[24818]: from='mgr.24938 ' entity='mgr.x' 2024-04-14T14:17:00.067 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:16:59 smithi069 ceph-mon[24818]: Upgrade: Target is version 19.0.0-2778-ga971fd5f (squid) 2024-04-14T14:17:00.068 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:16:59 smithi069 ceph-mon[24818]: Upgrade: Target container is quay.ceph.io/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888, digests ['quay.ceph.io/ceph-ci/ceph@sha256:02f0a191d4d693952654c64f93cc9c0152602c0915251b3c94c2c4e32855c608'] 2024-04-14T14:17:00.068 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:16:59 smithi069 ceph-mon[24818]: from='mgr.24938 172.21.15.122:0/87298362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-14T14:17:00.068 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:16:59 smithi069 ceph-mon[24818]: Upgrade: Need to upgrade myself (mgr.x) 2024-04-14T14:17:00.068 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:16:59 smithi069 ceph-mon[24818]: Upgrade: Need to upgrade myself (mgr.x) 2024-04-14T14:17:00.068 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:16:59 smithi069 ceph-mon[24818]: Failing over to other MGR 2024-04-14T14:17:00.068 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:16:59 smithi069 ceph-mon[24818]: from='mgr.24938 172.21.15.122:0/87298362' entity='mgr.x' cmd=[{"prefix": "mgr fail", "who": "x"}]: dispatch 2024-04-14T14:17:00.068 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:16:59 smithi069 ceph-mon[24818]: from='mgr.24938 ' entity='mgr.x' cmd=[{"prefix": "mgr fail", "who": "x"}]: dispatch 2024-04-14T14:17:00.068 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:16:59 smithi069 ceph-mon[24818]: osdmap e110: 8 total, 8 up, 8 in 2024-04-14T14:17:00.068 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:16:59 smithi069 ceph-mon[24818]: pgmap v117: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-04-14T14:17:00.068 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:16:59 smithi069 ceph-mon[24818]: Standby manager daemon x started 2024-04-14T14:17:00.068 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 14 14:16:59 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-y[63038]: [14/Apr/2024:14:16:59] ENGINE Bus STOPPING 2024-04-14T14:17:00.069 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 14 14:16:59 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-y[63038]: [14/Apr/2024:14:16:59] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-04-14T14:17:00.069 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 14 14:16:59 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-y[63038]: [14/Apr/2024:14:16:59] ENGINE Bus STOPPED 2024-04-14T14:17:00.069 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 14 14:16:59 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-y[63038]: [14/Apr/2024:14:16:59] ENGINE Bus STARTING 2024-04-14T14:17:00.069 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 14 14:17:00 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-y[63038]: [14/Apr/2024:14:17:00] ENGINE Serving on http://:::9283 2024-04-14T14:17:00.069 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 14 14:17:00 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-y[63038]: [14/Apr/2024:14:17:00] ENGINE Bus STARTED 2024-04-14T14:17:00.069 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:16:59 smithi069 ceph-mon[29472]: from='client.25273 -' entity='client.iscsi.foo.smithi069.yuumtr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T14:17:00.069 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:16:59 smithi069 ceph-mon[29472]: from='mgr.24938 ' entity='mgr.x' 2024-04-14T14:17:00.069 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:16:59 smithi069 ceph-mon[29472]: Upgrade: Target is version 19.0.0-2778-ga971fd5f (squid) 2024-04-14T14:17:00.069 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:16:59 smithi069 ceph-mon[29472]: Upgrade: Target container is quay.ceph.io/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888, digests ['quay.ceph.io/ceph-ci/ceph@sha256:02f0a191d4d693952654c64f93cc9c0152602c0915251b3c94c2c4e32855c608'] 2024-04-14T14:17:00.069 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:16:59 smithi069 ceph-mon[29472]: from='mgr.24938 172.21.15.122:0/87298362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-14T14:17:00.069 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:16:59 smithi069 ceph-mon[29472]: Upgrade: Need to upgrade myself (mgr.x) 2024-04-14T14:17:00.069 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:16:59 smithi069 ceph-mon[29472]: Upgrade: Need to upgrade myself (mgr.x) 2024-04-14T14:17:00.070 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:16:59 smithi069 ceph-mon[29472]: Failing over to other MGR 2024-04-14T14:17:00.070 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:16:59 smithi069 ceph-mon[29472]: from='mgr.24938 172.21.15.122:0/87298362' entity='mgr.x' cmd=[{"prefix": "mgr fail", "who": "x"}]: dispatch 2024-04-14T14:17:00.070 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:16:59 smithi069 ceph-mon[29472]: from='mgr.24938 ' entity='mgr.x' cmd=[{"prefix": "mgr fail", "who": "x"}]: dispatch 2024-04-14T14:17:00.070 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:16:59 smithi069 ceph-mon[29472]: osdmap e110: 8 total, 8 up, 8 in 2024-04-14T14:17:00.070 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:16:59 smithi069 ceph-mon[29472]: pgmap v117: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-04-14T14:17:00.070 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:16:59 smithi069 ceph-mon[29472]: Standby manager daemon x started 2024-04-14T14:17:00.095 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Apr 14 14:16:59 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-x[45487]: 2024-04-14T14:16:59.818+0000 7f9605151700 -1 mgr handle_mgr_map I was active but no longer am 2024-04-14T14:17:00.095 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Apr 14 14:16:59 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-x[45487]: ignoring --setuser ceph since I am not root 2024-04-14T14:17:00.095 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Apr 14 14:16:59 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-x[45487]: ignoring --setgroup ceph since I am not root 2024-04-14T14:17:00.095 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Apr 14 14:16:59 smithi122 ceph-mgr[45491]: -- 172.21.15.122:0/2037715572 <== mon.2 v2:172.21.15.122:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55d718c665a0 con 0x55d718c3a800 2024-04-14T14:17:00.096 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:16:59 smithi122 ceph-mon[25721]: from='client.25273 -' entity='client.iscsi.foo.smithi069.yuumtr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T14:17:00.096 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:16:59 smithi122 ceph-mon[25721]: from='mgr.24938 ' entity='mgr.x' 2024-04-14T14:17:00.096 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:16:59 smithi122 ceph-mon[25721]: Upgrade: Target is version 19.0.0-2778-ga971fd5f (squid) 2024-04-14T14:17:00.096 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:16:59 smithi122 ceph-mon[25721]: Upgrade: Target container is quay.ceph.io/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888, digests ['quay.ceph.io/ceph-ci/ceph@sha256:02f0a191d4d693952654c64f93cc9c0152602c0915251b3c94c2c4e32855c608'] 2024-04-14T14:17:00.096 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:16:59 smithi122 ceph-mon[25721]: from='mgr.24938 172.21.15.122:0/87298362' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-14T14:17:00.096 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:16:59 smithi122 ceph-mon[25721]: Upgrade: Need to upgrade myself (mgr.x) 2024-04-14T14:17:00.096 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:16:59 smithi122 ceph-mon[25721]: Upgrade: Need to upgrade myself (mgr.x) 2024-04-14T14:17:00.096 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:16:59 smithi122 ceph-mon[25721]: Failing over to other MGR 2024-04-14T14:17:00.096 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:16:59 smithi122 ceph-mon[25721]: from='mgr.24938 172.21.15.122:0/87298362' entity='mgr.x' cmd=[{"prefix": "mgr fail", "who": "x"}]: dispatch 2024-04-14T14:17:00.096 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:16:59 smithi122 ceph-mon[25721]: from='mgr.24938 ' entity='mgr.x' cmd=[{"prefix": "mgr fail", "who": "x"}]: dispatch 2024-04-14T14:17:00.096 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:16:59 smithi122 ceph-mon[25721]: osdmap e110: 8 total, 8 up, 8 in 2024-04-14T14:17:00.096 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:16:59 smithi122 ceph-mon[25721]: pgmap v117: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-04-14T14:17:00.096 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:16:59 smithi122 ceph-mon[25721]: Standby manager daemon x started 2024-04-14T14:17:00.406 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Apr 14 14:17:00 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-prometheus-a[58033]: ts=2024-04-14T14:17:00.118Z caller=refresh.go:98 level=error component="discovery manager scrape" discovery=http msg="Unable to refresh target groups" err="Get \"http://172.21.15.122:8765/sd/prometheus/sd-config?service=node-exporter\": dial tcp 172.21.15.122:8765: connect: connection refused" 2024-04-14T14:17:00.407 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Apr 14 14:17:00 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-prometheus-a[58033]: ts=2024-04-14T14:17:00.123Z caller=refresh.go:98 level=error component="discovery manager scrape" discovery=http msg="Unable to refresh target groups" err="Get \"http://172.21.15.122:8765/sd/prometheus/sd-config?service=ceph-exporter\": dial tcp 172.21.15.122:8765: connect: connection refused" 2024-04-14T14:17:00.407 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Apr 14 14:17:00 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-prometheus-a[58033]: ts=2024-04-14T14:17:00.123Z caller=refresh.go:98 level=error component="discovery manager scrape" discovery=http msg="Unable to refresh target groups" err="Get \"http://172.21.15.122:8765/sd/prometheus/sd-config?service=mgr-prometheus\": dial tcp 172.21.15.122:8765: connect: connection refused" 2024-04-14T14:17:00.407 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Apr 14 14:17:00 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-prometheus-a[58033]: ts=2024-04-14T14:17:00.123Z caller=refresh.go:98 level=error component="discovery manager scrape" discovery=http msg="Unable to refresh target groups" err="Get \"http://172.21.15.122:8765/sd/prometheus/sd-config?service=nvmeof\": dial tcp 172.21.15.122:8765: connect: connection refused" 2024-04-14T14:17:00.407 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Apr 14 14:17:00 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-prometheus-a[58033]: ts=2024-04-14T14:17:00.123Z caller=refresh.go:98 level=error component="discovery manager notify" discovery=http msg="Unable to refresh target groups" err="Get \"http://172.21.15.122:8765/sd/prometheus/sd-config?service=alertmanager\": dial tcp 172.21.15.122:8765: connect: connection refused" 2024-04-14T14:17:00.814 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Apr 14 14:17:00 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-x[45487]: 2024-04-14T14:17:00.404+0000 7f5d7fed1200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-14T14:17:00.814 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Apr 14 14:17:00 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-x[45487]: 2024-04-14T14:17:00.503+0000 7f5d7fed1200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-14T14:17:00.815 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:17:00 smithi069 ceph-mon[24818]: from='mgr.24938 ' entity='mgr.x' cmd='[{"prefix": "mgr fail", "who": "x"}]': finished 2024-04-14T14:17:00.815 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:17:00 smithi069 ceph-mon[24818]: mgrmap e37: y(active, starting, since 1.00404s), standbys: x 2024-04-14T14:17:00.815 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:17:00 smithi069 ceph-mon[24818]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-14T14:17:00.816 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:17:00 smithi069 ceph-mon[24818]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-14T14:17:00.816 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:17:00 smithi069 ceph-mon[24818]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-14T14:17:00.816 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:17:00 smithi069 ceph-mon[24818]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-04-14T14:17:00.816 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:17:00 smithi069 ceph-mon[24818]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-04-14T14:17:00.816 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:17:00 smithi069 ceph-mon[24818]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-14T14:17:00.816 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:17:00 smithi069 ceph-mon[24818]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-14T14:17:00.816 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:17:00 smithi069 ceph-mon[24818]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-14T14:17:00.816 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:17:00 smithi069 ceph-mon[24818]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-14T14:17:00.816 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:17:00 smithi069 ceph-mon[24818]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-14T14:17:00.816 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:17:00 smithi069 ceph-mon[24818]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-14T14:17:00.816 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:17:00 smithi069 ceph-mon[24818]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-14T14:17:00.816 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:17:00 smithi069 ceph-mon[29472]: from='mgr.24938 ' entity='mgr.x' cmd='[{"prefix": "mgr fail", "who": "x"}]': finished 2024-04-14T14:17:00.816 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:17:00 smithi069 ceph-mon[29472]: mgrmap e37: y(active, starting, since 1.00404s), standbys: x 2024-04-14T14:17:00.817 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:17:00 smithi069 ceph-mon[29472]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-14T14:17:00.817 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:17:00 smithi069 ceph-mon[29472]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-14T14:17:00.817 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:17:00 smithi069 ceph-mon[29472]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-14T14:17:00.817 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:17:00 smithi069 ceph-mon[29472]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-04-14T14:17:00.817 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:17:00 smithi069 ceph-mon[29472]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-04-14T14:17:00.817 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:17:00 smithi069 ceph-mon[29472]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-14T14:17:00.817 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:17:00 smithi069 ceph-mon[29472]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-14T14:17:00.817 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:17:00 smithi069 ceph-mon[29472]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-14T14:17:00.817 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:17:00 smithi069 ceph-mon[29472]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-14T14:17:00.817 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:17:00 smithi069 ceph-mon[29472]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-14T14:17:00.817 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:17:00 smithi069 ceph-mon[29472]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-14T14:17:00.817 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:17:00 smithi069 ceph-mon[29472]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-14T14:17:00.817 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:17:00 smithi069 ceph-mon[29472]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-14T14:17:00.817 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:17:00 smithi069 ceph-mon[29472]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-04-14T14:17:00.817 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:17:00 smithi069 ceph-mon[29472]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-04-14T14:17:00.817 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:17:00 smithi069 ceph-mon[29472]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-04-14T14:17:00.818 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:17:00 smithi069 ceph-mon[29472]: Manager daemon y is now available 2024-04-14T14:17:00.877 INFO:teuthology.orchestra.run.smithi069.stdout:true 2024-04-14T14:17:01.067 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:17:00 smithi069 ceph-mon[24818]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-14T14:17:01.067 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:17:00 smithi069 ceph-mon[24818]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-04-14T14:17:01.067 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:17:00 smithi069 ceph-mon[24818]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-04-14T14:17:01.067 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:17:00 smithi069 ceph-mon[24818]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-04-14T14:17:01.067 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:17:00 smithi069 ceph-mon[24818]: Manager daemon y is now available 2024-04-14T14:17:01.067 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:17:00 smithi069 ceph-mon[24818]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T14:17:01.068 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:17:00 smithi069 ceph-mon[24818]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-04-14T14:17:01.068 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:17:00 smithi069 ceph-mon[24818]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-04-14T14:17:01.068 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:17:00 smithi069 ceph-mon[24818]: [14/Apr/2024:14:17:00] ENGINE Bus STARTING 2024-04-14T14:17:01.068 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:17:00 smithi069 ceph-mon[24818]: [14/Apr/2024:14:17:00] ENGINE Serving on https://172.21.15.69:7150 2024-04-14T14:17:01.068 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:17:00 smithi069 ceph-mon[24818]: [14/Apr/2024:14:17:00] ENGINE Serving on http://172.21.15.69:8765 2024-04-14T14:17:01.068 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:17:00 smithi069 ceph-mon[24818]: [14/Apr/2024:14:17:00] ENGINE Bus STARTED 2024-04-14T14:17:01.068 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:17:00 smithi069 ceph-mon[29472]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T14:17:01.068 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:17:00 smithi069 ceph-mon[29472]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-04-14T14:17:01.068 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:17:00 smithi069 ceph-mon[29472]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-04-14T14:17:01.068 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:17:00 smithi069 ceph-mon[29472]: [14/Apr/2024:14:17:00] ENGINE Bus STARTING 2024-04-14T14:17:01.068 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:17:00 smithi069 ceph-mon[29472]: [14/Apr/2024:14:17:00] ENGINE Serving on https://172.21.15.69:7150 2024-04-14T14:17:01.068 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:17:00 smithi069 ceph-mon[29472]: [14/Apr/2024:14:17:00] ENGINE Serving on http://172.21.15.69:8765 2024-04-14T14:17:01.069 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:17:00 smithi069 ceph-mon[29472]: [14/Apr/2024:14:17:00] ENGINE Bus STARTED 2024-04-14T14:17:01.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:17:00 smithi122 ceph-mon[25721]: from='mgr.24938 ' entity='mgr.x' cmd='[{"prefix": "mgr fail", "who": "x"}]': finished 2024-04-14T14:17:01.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:17:00 smithi122 ceph-mon[25721]: mgrmap e37: y(active, starting, since 1.00404s), standbys: x 2024-04-14T14:17:01.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:17:00 smithi122 ceph-mon[25721]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-14T14:17:01.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:17:00 smithi122 ceph-mon[25721]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-14T14:17:01.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:17:00 smithi122 ceph-mon[25721]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-14T14:17:01.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:17:00 smithi122 ceph-mon[25721]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-04-14T14:17:01.096 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:17:00 smithi122 ceph-mon[25721]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-04-14T14:17:01.096 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:17:00 smithi122 ceph-mon[25721]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-14T14:17:01.096 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:17:00 smithi122 ceph-mon[25721]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-14T14:17:01.096 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:17:00 smithi122 ceph-mon[25721]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-14T14:17:01.096 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:17:00 smithi122 ceph-mon[25721]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-14T14:17:01.096 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:17:00 smithi122 ceph-mon[25721]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-14T14:17:01.096 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:17:00 smithi122 ceph-mon[25721]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-14T14:17:01.096 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:17:00 smithi122 ceph-mon[25721]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-14T14:17:01.096 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:17:00 smithi122 ceph-mon[25721]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-14T14:17:01.096 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:17:00 smithi122 ceph-mon[25721]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-04-14T14:17:01.096 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:17:00 smithi122 ceph-mon[25721]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-04-14T14:17:01.096 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:17:00 smithi122 ceph-mon[25721]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-04-14T14:17:01.096 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:17:00 smithi122 ceph-mon[25721]: Manager daemon y is now available 2024-04-14T14:17:01.096 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:17:00 smithi122 ceph-mon[25721]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T14:17:01.096 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:17:00 smithi122 ceph-mon[25721]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-04-14T14:17:01.096 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:17:00 smithi122 ceph-mon[25721]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-04-14T14:17:01.097 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:17:00 smithi122 ceph-mon[25721]: [14/Apr/2024:14:17:00] ENGINE Bus STARTING 2024-04-14T14:17:01.097 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:17:00 smithi122 ceph-mon[25721]: [14/Apr/2024:14:17:00] ENGINE Serving on https://172.21.15.69:7150 2024-04-14T14:17:01.097 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:17:00 smithi122 ceph-mon[25721]: [14/Apr/2024:14:17:00] ENGINE Serving on http://172.21.15.69:8765 2024-04-14T14:17:01.097 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:17:00 smithi122 ceph-mon[25721]: [14/Apr/2024:14:17:00] ENGINE Bus STARTED 2024-04-14T14:17:01.540 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Apr 14 14:17:01 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-x[45487]: 2024-04-14T14:17:01.198+0000 7f5d7fed1200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-14T14:17:01.540 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Apr 14 14:17:01 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-x[45487]: 2024-04-14T14:17:01.293+0000 7f5d7fed1200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-14T14:17:01.540 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Apr 14 14:17:01 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-x[45487]: 2024-04-14T14:17:01.438+0000 7f5d7fed1200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-14T14:17:01.540 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Apr 14 14:17:01 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-x[45487]: 2024-04-14T14:17:01.538+0000 7f5d7fed1200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-14T14:17:01.550 INFO:teuthology.orchestra.run.smithi069.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-14T14:17:01.550 INFO:teuthology.orchestra.run.smithi069.stdout:alertmanager.a smithi069 *:9093,9094 running (12m) 2m ago 20m 22.5M - 0.23.0 ba2b418f427c e300fc36e997 2024-04-14T14:17:01.550 INFO:teuthology.orchestra.run.smithi069.stdout:grafana.a smithi122 *:3000 error 2m ago 20m - - 2024-04-14T14:17:01.550 INFO:teuthology.orchestra.run.smithi069.stdout:iscsi.foo.smithi069.yuumtr smithi069 running (2m) 2m ago 19m 54.3M - 3.5 e1d6a67b021e 65932c077132 2024-04-14T14:17:01.551 INFO:teuthology.orchestra.run.smithi069.stdout:mgr.x smithi122 *:8443,9283 running (15m) 2m ago 24m 497M - 19.0.0-2778-ga971fd5f b96c509eef84 813744c2b615 2024-04-14T14:17:01.551 INFO:teuthology.orchestra.run.smithi069.stdout:mgr.y smithi069 *:8443,9283,8765 running (9m) 2m ago 26m 449M - 19.0.0-2778-ga971fd5f b96c509eef84 7af3d29d0e5f 2024-04-14T14:17:01.551 INFO:teuthology.orchestra.run.smithi069.stdout:mon.a smithi069 running (26m) 2m ago 26m 69.6M 2048M 17.2.0 e1d6a67b021e d3c553f91913 2024-04-14T14:17:01.551 INFO:teuthology.orchestra.run.smithi069.stdout:mon.b smithi122 running (25m) 2m ago 25m 55.6M 2048M 17.2.0 e1d6a67b021e 3b766bfce4be 2024-04-14T14:17:01.551 INFO:teuthology.orchestra.run.smithi069.stdout:mon.c smithi069 running (25m) 2m ago 25m 51.1M 2048M 17.2.0 e1d6a67b021e 23c078b9d1e5 2024-04-14T14:17:01.551 INFO:teuthology.orchestra.run.smithi069.stdout:node-exporter.a smithi069 *:9100 running (12m) 2m ago 21m 20.6M - 1.3.1 1dbe0e931976 ca074142ab73 2024-04-14T14:17:01.551 INFO:teuthology.orchestra.run.smithi069.stdout:node-exporter.b smithi122 *:9100 running (12m) 2m ago 21m 19.1M - 1.3.1 1dbe0e931976 996115353a0b 2024-04-14T14:17:01.551 INFO:teuthology.orchestra.run.smithi069.stdout:osd.0 smithi069 running (24m) 2m ago 24m 56.1M 2992M 17.2.0 e1d6a67b021e 46654654b6b6 2024-04-14T14:17:01.551 INFO:teuthology.orchestra.run.smithi069.stdout:osd.1 smithi069 running (24m) 2m ago 24m 53.0M 2992M 17.2.0 e1d6a67b021e cdba54a59609 2024-04-14T14:17:01.551 INFO:teuthology.orchestra.run.smithi069.stdout:osd.2 smithi069 running (23m) 2m ago 23m 50.3M 2992M 17.2.0 e1d6a67b021e bc37a1a9cba0 2024-04-14T14:17:01.551 INFO:teuthology.orchestra.run.smithi069.stdout:osd.3 smithi069 running (23m) 2m ago 23m 54.9M 2992M 17.2.0 e1d6a67b021e 0925ba3da801 2024-04-14T14:17:01.551 INFO:teuthology.orchestra.run.smithi069.stdout:osd.4 smithi122 running (23m) 2m ago 23m 53.1M 3248M 17.2.0 e1d6a67b021e c00975caf771 2024-04-14T14:17:01.551 INFO:teuthology.orchestra.run.smithi069.stdout:osd.5 smithi122 running (22m) 2m ago 22m 54.1M 3248M 17.2.0 e1d6a67b021e 5f52d217ceb8 2024-04-14T14:17:01.552 INFO:teuthology.orchestra.run.smithi069.stdout:osd.6 smithi122 running (22m) 2m ago 22m 49.8M 3248M 17.2.0 e1d6a67b021e a2acfec61475 2024-04-14T14:17:01.552 INFO:teuthology.orchestra.run.smithi069.stdout:osd.7 smithi122 running (21m) 2m ago 21m 55.3M 3248M 17.2.0 e1d6a67b021e 62411cd06801 2024-04-14T14:17:01.552 INFO:teuthology.orchestra.run.smithi069.stdout:prometheus.a smithi122 *:9095 running (3m) 2m ago 21m 42.2M - 2.33.4 514e6a882f6e 012f5f9d7df5 2024-04-14T14:17:01.552 INFO:teuthology.orchestra.run.smithi069.stdout:rgw.foo.smithi069.wesedz smithi069 *:8000 running (19m) 2m ago 19m 76.7M - 17.2.0 e1d6a67b021e 16de491440f3 2024-04-14T14:17:01.552 INFO:teuthology.orchestra.run.smithi069.stdout:rgw.foo.smithi122.jiobyv smithi122 *:8000 running (19m) 2m ago 19m 73.7M - 17.2.0 e1d6a67b021e 6d16de445dba 2024-04-14T14:17:01.811 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Apr 14 14:17:01 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-x[45487]: 2024-04-14T14:17:01.637+0000 7f5d7fed1200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-14T14:17:01.963 INFO:teuthology.orchestra.run.smithi069.stdout:{ 2024-04-14T14:17:01.964 INFO:teuthology.orchestra.run.smithi069.stdout: "mon": { 2024-04-14T14:17:01.964 INFO:teuthology.orchestra.run.smithi069.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 3 2024-04-14T14:17:01.964 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-14T14:17:01.964 INFO:teuthology.orchestra.run.smithi069.stdout: "mgr": { 2024-04-14T14:17:01.964 INFO:teuthology.orchestra.run.smithi069.stdout: "ceph version 19.0.0-2778-ga971fd5f (a971fd5f5719038c70b817bafacd6d5400b1d888) squid (dev)": 2 2024-04-14T14:17:01.964 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-14T14:17:01.964 INFO:teuthology.orchestra.run.smithi069.stdout: "osd": { 2024-04-14T14:17:01.964 INFO:teuthology.orchestra.run.smithi069.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 8 2024-04-14T14:17:01.964 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-14T14:17:01.964 INFO:teuthology.orchestra.run.smithi069.stdout: "mds": {}, 2024-04-14T14:17:01.964 INFO:teuthology.orchestra.run.smithi069.stdout: "rgw": { 2024-04-14T14:17:01.964 INFO:teuthology.orchestra.run.smithi069.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-04-14T14:17:01.964 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-14T14:17:01.965 INFO:teuthology.orchestra.run.smithi069.stdout: "overall": { 2024-04-14T14:17:01.965 INFO:teuthology.orchestra.run.smithi069.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 13, 2024-04-14T14:17:01.965 INFO:teuthology.orchestra.run.smithi069.stdout: "ceph version 19.0.0-2778-ga971fd5f (a971fd5f5719038c70b817bafacd6d5400b1d888) squid (dev)": 2 2024-04-14T14:17:01.965 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-14T14:17:01.965 INFO:teuthology.orchestra.run.smithi069.stdout:} 2024-04-14T14:17:02.095 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Apr 14 14:17:01 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-x[45487]: 2024-04-14T14:17:01.912+0000 7f5d7fed1200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-14T14:17:02.095 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Apr 14 14:17:02 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-x[45487]: 2024-04-14T14:17:02.027+0000 7f5d7fed1200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-14T14:17:02.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:17:01 smithi122 ceph-mon[25721]: mgrmap e38: y(active, since 2s), standbys: x 2024-04-14T14:17:02.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:17:01 smithi122 ceph-mon[25721]: pgmap v3: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail 2024-04-14T14:17:02.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:17:01 smithi122 ceph-mon[25721]: from='client.15468 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T14:17:02.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:17:01 smithi122 ceph-mon[25721]: from='client.15474 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T14:17:02.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:17:01 smithi122 ceph-mon[25721]: from='client.25381 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T14:17:02.317 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:17:01 smithi069 ceph-mon[24818]: mgrmap e38: y(active, since 2s), standbys: x 2024-04-14T14:17:02.317 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:17:01 smithi069 ceph-mon[24818]: pgmap v3: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail 2024-04-14T14:17:02.317 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:17:01 smithi069 ceph-mon[24818]: from='client.15468 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T14:17:02.317 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:17:01 smithi069 ceph-mon[24818]: from='client.15474 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T14:17:02.317 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:17:01 smithi069 ceph-mon[24818]: from='client.25381 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T14:17:02.317 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:17:01 smithi069 ceph-mon[29472]: mgrmap e38: y(active, since 2s), standbys: x 2024-04-14T14:17:02.317 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:17:01 smithi069 ceph-mon[29472]: pgmap v3: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail 2024-04-14T14:17:02.318 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:17:01 smithi069 ceph-mon[29472]: from='client.15468 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T14:17:02.318 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:17:01 smithi069 ceph-mon[29472]: from='client.15474 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T14:17:02.318 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:17:01 smithi069 ceph-mon[29472]: from='client.25381 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T14:17:02.328 INFO:teuthology.orchestra.run.smithi069.stdout:{ 2024-04-14T14:17:02.328 INFO:teuthology.orchestra.run.smithi069.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888", 2024-04-14T14:17:02.328 INFO:teuthology.orchestra.run.smithi069.stdout: "in_progress": true, 2024-04-14T14:17:02.328 INFO:teuthology.orchestra.run.smithi069.stdout: "which": "Upgrading daemons of type(s) mgr", 2024-04-14T14:17:02.328 INFO:teuthology.orchestra.run.smithi069.stdout: "services_complete": [ 2024-04-14T14:17:02.328 INFO:teuthology.orchestra.run.smithi069.stdout: "mgr" 2024-04-14T14:17:02.328 INFO:teuthology.orchestra.run.smithi069.stdout: ], 2024-04-14T14:17:02.328 INFO:teuthology.orchestra.run.smithi069.stdout: "progress": "2/2 daemons upgraded", 2024-04-14T14:17:02.328 INFO:teuthology.orchestra.run.smithi069.stdout: "message": "", 2024-04-14T14:17:02.328 INFO:teuthology.orchestra.run.smithi069.stdout: "is_paused": false 2024-04-14T14:17:02.328 INFO:teuthology.orchestra.run.smithi069.stdout:} 2024-04-14T14:17:02.423 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Apr 14 14:17:02 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-x[45487]: 2024-04-14T14:17:02.116+0000 7f5d7fed1200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-14T14:17:02.424 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Apr 14 14:17:02 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-x[45487]: 2024-04-14T14:17:02.212+0000 7f5d7fed1200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-14T14:17:02.424 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Apr 14 14:17:02 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-x[45487]: 2024-04-14T14:17:02.421+0000 7f5d7fed1200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-14T14:17:03.067 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:17:02 smithi069 ceph-mon[24818]: pgmap v4: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail 2024-04-14T14:17:03.067 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:17:02 smithi069 ceph-mon[24818]: Health check cleared: CEPHADM_FAILED_DAEMON (was: 1 failed cephadm daemon(s)) 2024-04-14T14:17:03.068 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:17:02 smithi069 ceph-mon[24818]: Cluster is now healthy 2024-04-14T14:17:03.068 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:17:02 smithi069 ceph-mon[24818]: from='client.? 172.21.15.69:0/2791224726' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T14:17:03.068 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:17:02 smithi069 ceph-mon[24818]: from='client.25399 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T14:17:03.068 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:17:02 smithi069 ceph-mon[29472]: pgmap v4: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail 2024-04-14T14:17:03.068 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:17:02 smithi069 ceph-mon[29472]: Health check cleared: CEPHADM_FAILED_DAEMON (was: 1 failed cephadm daemon(s)) 2024-04-14T14:17:03.068 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:17:02 smithi069 ceph-mon[29472]: Cluster is now healthy 2024-04-14T14:17:03.068 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:17:02 smithi069 ceph-mon[29472]: from='client.? 172.21.15.69:0/2791224726' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T14:17:03.068 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:17:02 smithi069 ceph-mon[29472]: from='client.25399 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T14:17:03.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:17:02 smithi122 ceph-mon[25721]: pgmap v4: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail 2024-04-14T14:17:03.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:17:02 smithi122 ceph-mon[25721]: Health check cleared: CEPHADM_FAILED_DAEMON (was: 1 failed cephadm daemon(s)) 2024-04-14T14:17:03.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:17:02 smithi122 ceph-mon[25721]: Cluster is now healthy 2024-04-14T14:17:03.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:17:02 smithi122 ceph-mon[25721]: from='client.? 172.21.15.69:0/2791224726' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T14:17:03.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:17:02 smithi122 ceph-mon[25721]: from='client.25399 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T14:17:03.845 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Apr 14 14:17:03 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-x[45487]: 2024-04-14T14:17:03.448+0000 7f5d7fed1200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-14T14:17:03.845 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Apr 14 14:17:03 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-x[45487]: 2024-04-14T14:17:03.585+0000 7f5d7fed1200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-14T14:17:04.247 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:17:03 smithi069 ceph-mon[24818]: mgrmap e39: y(active, since 4s), standbys: x 2024-04-14T14:17:04.247 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:17:03 smithi069 ceph-mon[29472]: mgrmap e39: y(active, since 4s), standbys: x 2024-04-14T14:17:04.260 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:17:03 smithi122 ceph-mon[25721]: mgrmap e39: y(active, since 4s), standbys: x 2024-04-14T14:17:04.261 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Apr 14 14:17:03 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-x[45487]: 2024-04-14T14:17:03.934+0000 7f5d7fed1200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-14T14:17:04.261 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Apr 14 14:17:04 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-x[45487]: 2024-04-14T14:17:04.040+0000 7f5d7fed1200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-14T14:17:04.261 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Apr 14 14:17:04 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-x[45487]: 2024-04-14T14:17:04.258+0000 7f5d7fed1200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-14T14:17:04.567 INFO:journalctl@ceph.alertmanager.a.smithi069.stdout:Apr 14 14:17:04 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-alertmanager-a[58923]: level=warn ts=2024-04-14T14:17:04.245Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi122.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.122:8443: connect: connection refused" 2024-04-14T14:17:04.595 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Apr 14 14:17:04 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-x[45487]: 2024-04-14T14:17:04.459+0000 7f5d7fed1200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-14T14:17:05.040 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Apr 14 14:17:04 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-x[45487]: 2024-04-14T14:17:04.767+0000 7f5d7fed1200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-14T14:17:05.040 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Apr 14 14:17:05 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-x[45487]: 2024-04-14T14:17:05.038+0000 7f5d7fed1200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-14T14:17:05.295 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:17:05 smithi122 ceph-mon[25721]: pgmap v5: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail 2024-04-14T14:17:05.295 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:17:05 smithi122 ceph-mon[25721]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' 2024-04-14T14:17:05.295 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:17:05 smithi122 ceph-mon[25721]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' 2024-04-14T14:17:05.295 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:17:05 smithi122 ceph-mon[25721]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' 2024-04-14T14:17:05.295 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:17:05 smithi122 ceph-mon[25721]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' 2024-04-14T14:17:05.567 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:17:05 smithi069 ceph-mon[24818]: pgmap v5: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail 2024-04-14T14:17:05.567 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:17:05 smithi069 ceph-mon[24818]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' 2024-04-14T14:17:05.567 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:17:05 smithi069 ceph-mon[24818]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' 2024-04-14T14:17:05.568 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:17:05 smithi069 ceph-mon[24818]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' 2024-04-14T14:17:05.568 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:17:05 smithi069 ceph-mon[24818]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' 2024-04-14T14:17:05.568 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:17:05 smithi069 ceph-mon[29472]: pgmap v5: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail 2024-04-14T14:17:05.568 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:17:05 smithi069 ceph-mon[29472]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' 2024-04-14T14:17:05.568 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:17:05 smithi069 ceph-mon[29472]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' 2024-04-14T14:17:05.568 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:17:05 smithi069 ceph-mon[29472]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' 2024-04-14T14:17:05.568 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:17:05 smithi069 ceph-mon[29472]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' 2024-04-14T14:17:05.594 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Apr 14 14:17:05 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-x[45487]: 2024-04-14T14:17:05.293+0000 7f5d7fed1200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-14T14:17:06.567 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:17:06 smithi069 ceph-mon[24818]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' 2024-04-14T14:17:06.568 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:17:06 smithi069 ceph-mon[24818]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' 2024-04-14T14:17:06.568 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:17:06 smithi069 ceph-mon[24818]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' 2024-04-14T14:17:06.568 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:17:06 smithi069 ceph-mon[24818]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' 2024-04-14T14:17:06.568 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:17:06 smithi069 ceph-mon[24818]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T14:17:06.568 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:17:06 smithi069 ceph-mon[24818]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-14T14:17:06.568 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:17:06 smithi069 ceph-mon[24818]: Updating smithi069:/etc/ceph/ceph.conf 2024-04-14T14:17:06.568 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:17:06 smithi069 ceph-mon[24818]: Updating smithi122:/etc/ceph/ceph.conf 2024-04-14T14:17:06.568 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:17:06 smithi069 ceph-mon[29472]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' 2024-04-14T14:17:06.568 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:17:06 smithi069 ceph-mon[29472]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' 2024-04-14T14:17:06.569 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:17:06 smithi069 ceph-mon[29472]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' 2024-04-14T14:17:06.569 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:17:06 smithi069 ceph-mon[29472]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' 2024-04-14T14:17:06.569 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:17:06 smithi069 ceph-mon[29472]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T14:17:06.569 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:17:06 smithi069 ceph-mon[29472]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-14T14:17:06.569 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:17:06 smithi069 ceph-mon[29472]: Updating smithi069:/etc/ceph/ceph.conf 2024-04-14T14:17:06.569 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:17:06 smithi069 ceph-mon[29472]: Updating smithi122:/etc/ceph/ceph.conf 2024-04-14T14:17:06.595 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Apr 14 14:17:06 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-x[45487]: 2024-04-14T14:17:06.140+0000 7f5d7fed1200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-14T14:17:06.595 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:17:06 smithi122 ceph-mon[25721]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' 2024-04-14T14:17:06.595 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:17:06 smithi122 ceph-mon[25721]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' 2024-04-14T14:17:06.595 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:17:06 smithi122 ceph-mon[25721]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' 2024-04-14T14:17:06.595 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:17:06 smithi122 ceph-mon[25721]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' 2024-04-14T14:17:06.595 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:17:06 smithi122 ceph-mon[25721]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T14:17:06.595 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:17:06 smithi122 ceph-mon[25721]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-14T14:17:06.595 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:17:06 smithi122 ceph-mon[25721]: Updating smithi069:/etc/ceph/ceph.conf 2024-04-14T14:17:06.595 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:17:06 smithi122 ceph-mon[25721]: Updating smithi122:/etc/ceph/ceph.conf 2024-04-14T14:17:07.068 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Apr 14 14:17:06 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-x[45487]: 2024-04-14T14:17:06.739+0000 7f5d7fed1200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-14T14:17:07.068 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Apr 14 14:17:06 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-x[45487]: 2024-04-14T14:17:06.841+0000 7f5d7fed1200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-14T14:17:07.068 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Apr 14 14:17:07 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-x[45487]: 2024-04-14T14:17:07.065+0000 7f5d7fed1200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-14T14:17:07.345 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Apr 14 14:17:07 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-x[45487]: [14/Apr/2024:14:17:07] ENGINE Bus STARTING 2024-04-14T14:17:07.345 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Apr 14 14:17:07 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-x[45487]: CherryPy Checker: 2024-04-14T14:17:07.345 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Apr 14 14:17:07 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-x[45487]: The Application mounted at '' has an empty config. 2024-04-14T14:17:07.345 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Apr 14 14:17:07 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-x[45487]: 2024-04-14T14:17:07.345 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Apr 14 14:17:07 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-x[45487]: [14/Apr/2024:14:17:07] ENGINE Serving on http://:::9283 2024-04-14T14:17:07.345 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Apr 14 14:17:07 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-x[45487]: [14/Apr/2024:14:17:07] ENGINE Bus STARTED 2024-04-14T14:17:07.817 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:17:07 smithi069 ceph-mon[29472]: pgmap v6: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail 2024-04-14T14:17:07.817 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:17:07 smithi069 ceph-mon[29472]: Updating smithi069:/var/lib/ceph/dc326fc2-fa65-11ee-bc8e-c7b262605968/config/ceph.conf 2024-04-14T14:17:07.817 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:17:07 smithi069 ceph-mon[29472]: Updating smithi122:/var/lib/ceph/dc326fc2-fa65-11ee-bc8e-c7b262605968/config/ceph.conf 2024-04-14T14:17:07.818 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:17:07 smithi069 ceph-mon[29472]: Updating smithi069:/etc/ceph/ceph.client.admin.keyring 2024-04-14T14:17:07.818 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:17:07 smithi069 ceph-mon[29472]: Updating smithi122:/etc/ceph/ceph.client.admin.keyring 2024-04-14T14:17:07.818 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:17:07 smithi069 ceph-mon[29472]: Updating smithi069:/var/lib/ceph/dc326fc2-fa65-11ee-bc8e-c7b262605968/config/ceph.client.admin.keyring 2024-04-14T14:17:07.818 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:17:07 smithi069 ceph-mon[29472]: Updating smithi122:/var/lib/ceph/dc326fc2-fa65-11ee-bc8e-c7b262605968/config/ceph.client.admin.keyring 2024-04-14T14:17:07.818 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:17:07 smithi069 ceph-mon[29472]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' 2024-04-14T14:17:07.818 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:17:07 smithi069 ceph-mon[29472]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' 2024-04-14T14:17:07.818 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:17:07 smithi069 ceph-mon[29472]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' 2024-04-14T14:17:07.818 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:17:07 smithi069 ceph-mon[29472]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' 2024-04-14T14:17:07.818 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:17:07 smithi069 ceph-mon[29472]: pgmap v7: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 20 op/s 2024-04-14T14:17:07.818 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:17:07 smithi069 ceph-mon[29472]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' 2024-04-14T14:17:07.818 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:17:07 smithi069 ceph-mon[29472]: Health check failed: 1 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON) 2024-04-14T14:17:07.818 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:17:07 smithi069 ceph-mon[29472]: Reconfiguring iscsi.foo.smithi069.yuumtr (dependencies changed)... 2024-04-14T14:17:07.818 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:17:07 smithi069 ceph-mon[29472]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi069.yuumtr", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-04-14T14:17:07.818 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:17:07 smithi069 ceph-mon[29472]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T14:17:07.819 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:17:07 smithi069 ceph-mon[29472]: Reconfiguring daemon iscsi.foo.smithi069.yuumtr on smithi069 2024-04-14T14:17:07.819 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:17:07 smithi069 ceph-mon[29472]: Standby manager daemon x restarted 2024-04-14T14:17:07.819 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:17:07 smithi069 ceph-mon[29472]: Standby manager daemon x started 2024-04-14T14:17:07.819 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:17:07 smithi069 ceph-mon[29472]: from='mgr.? 172.21.15.122:0/1646783247' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-04-14T14:17:07.819 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:17:07 smithi069 ceph-mon[29472]: from='mgr.? 172.21.15.122:0/1646783247' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-14T14:17:07.819 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:17:07 smithi069 ceph-mon[29472]: from='mgr.? 172.21.15.122:0/1646783247' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-04-14T14:17:07.819 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:17:07 smithi069 ceph-mon[29472]: from='mgr.? 172.21.15.122:0/1646783247' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-14T14:17:07.819 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:17:07 smithi069 ceph-mon[24818]: pgmap v6: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail 2024-04-14T14:17:07.819 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:17:07 smithi069 ceph-mon[24818]: Updating smithi069:/var/lib/ceph/dc326fc2-fa65-11ee-bc8e-c7b262605968/config/ceph.conf 2024-04-14T14:17:07.819 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:17:07 smithi069 ceph-mon[24818]: Updating smithi122:/var/lib/ceph/dc326fc2-fa65-11ee-bc8e-c7b262605968/config/ceph.conf 2024-04-14T14:17:07.819 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:17:07 smithi069 ceph-mon[24818]: Updating smithi069:/etc/ceph/ceph.client.admin.keyring 2024-04-14T14:17:07.819 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:17:07 smithi069 ceph-mon[24818]: Updating smithi122:/etc/ceph/ceph.client.admin.keyring 2024-04-14T14:17:07.819 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:17:07 smithi069 ceph-mon[24818]: Updating smithi069:/var/lib/ceph/dc326fc2-fa65-11ee-bc8e-c7b262605968/config/ceph.client.admin.keyring 2024-04-14T14:17:07.820 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:17:07 smithi069 ceph-mon[24818]: Updating smithi122:/var/lib/ceph/dc326fc2-fa65-11ee-bc8e-c7b262605968/config/ceph.client.admin.keyring 2024-04-14T14:17:07.820 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:17:07 smithi069 ceph-mon[24818]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' 2024-04-14T14:17:07.820 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:17:07 smithi069 ceph-mon[24818]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' 2024-04-14T14:17:07.820 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:17:07 smithi069 ceph-mon[24818]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' 2024-04-14T14:17:07.820 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:17:07 smithi069 ceph-mon[24818]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' 2024-04-14T14:17:07.820 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:17:07 smithi069 ceph-mon[24818]: pgmap v7: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 20 op/s 2024-04-14T14:17:07.820 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:17:07 smithi069 ceph-mon[24818]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' 2024-04-14T14:17:07.820 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:17:07 smithi069 ceph-mon[24818]: Health check failed: 1 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON) 2024-04-14T14:17:07.820 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:17:07 smithi069 ceph-mon[24818]: Reconfiguring iscsi.foo.smithi069.yuumtr (dependencies changed)... 2024-04-14T14:17:07.820 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:17:07 smithi069 ceph-mon[24818]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi069.yuumtr", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-04-14T14:17:07.820 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:17:07 smithi069 ceph-mon[24818]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T14:17:07.820 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:17:07 smithi069 ceph-mon[24818]: Reconfiguring daemon iscsi.foo.smithi069.yuumtr on smithi069 2024-04-14T14:17:07.820 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:17:07 smithi069 ceph-mon[24818]: Standby manager daemon x restarted 2024-04-14T14:17:07.820 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:17:07 smithi069 ceph-mon[24818]: Standby manager daemon x started 2024-04-14T14:17:07.821 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:17:07 smithi069 ceph-mon[24818]: from='mgr.? 172.21.15.122:0/1646783247' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-04-14T14:17:07.821 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:17:07 smithi069 ceph-mon[24818]: from='mgr.? 172.21.15.122:0/1646783247' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-14T14:17:07.821 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:17:07 smithi069 ceph-mon[24818]: from='mgr.? 172.21.15.122:0/1646783247' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-04-14T14:17:07.821 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:17:07 smithi069 ceph-mon[24818]: from='mgr.? 172.21.15.122:0/1646783247' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-14T14:17:08.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:17:07 smithi122 ceph-mon[25721]: pgmap v6: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail 2024-04-14T14:17:08.096 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:17:07 smithi122 ceph-mon[25721]: Updating smithi069:/var/lib/ceph/dc326fc2-fa65-11ee-bc8e-c7b262605968/config/ceph.conf 2024-04-14T14:17:08.096 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:17:07 smithi122 ceph-mon[25721]: Updating smithi122:/var/lib/ceph/dc326fc2-fa65-11ee-bc8e-c7b262605968/config/ceph.conf 2024-04-14T14:17:08.096 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:17:07 smithi122 ceph-mon[25721]: Updating smithi069:/etc/ceph/ceph.client.admin.keyring 2024-04-14T14:17:08.096 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:17:07 smithi122 ceph-mon[25721]: Updating smithi122:/etc/ceph/ceph.client.admin.keyring 2024-04-14T14:17:08.096 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:17:07 smithi122 ceph-mon[25721]: Updating smithi069:/var/lib/ceph/dc326fc2-fa65-11ee-bc8e-c7b262605968/config/ceph.client.admin.keyring 2024-04-14T14:17:08.096 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:17:07 smithi122 ceph-mon[25721]: Updating smithi122:/var/lib/ceph/dc326fc2-fa65-11ee-bc8e-c7b262605968/config/ceph.client.admin.keyring 2024-04-14T14:17:08.096 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:17:07 smithi122 ceph-mon[25721]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' 2024-04-14T14:17:08.096 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:17:07 smithi122 ceph-mon[25721]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' 2024-04-14T14:17:08.096 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:17:07 smithi122 ceph-mon[25721]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' 2024-04-14T14:17:08.096 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:17:07 smithi122 ceph-mon[25721]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' 2024-04-14T14:17:08.096 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:17:07 smithi122 ceph-mon[25721]: pgmap v7: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 20 op/s 2024-04-14T14:17:08.097 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:17:07 smithi122 ceph-mon[25721]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' 2024-04-14T14:17:08.097 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:17:07 smithi122 ceph-mon[25721]: Health check failed: 1 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON) 2024-04-14T14:17:08.097 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:17:07 smithi122 ceph-mon[25721]: Reconfiguring iscsi.foo.smithi069.yuumtr (dependencies changed)... 2024-04-14T14:17:08.097 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:17:07 smithi122 ceph-mon[25721]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi069.yuumtr", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-04-14T14:17:08.097 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:17:07 smithi122 ceph-mon[25721]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T14:17:08.097 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:17:07 smithi122 ceph-mon[25721]: Reconfiguring daemon iscsi.foo.smithi069.yuumtr on smithi069 2024-04-14T14:17:08.097 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:17:07 smithi122 ceph-mon[25721]: Standby manager daemon x restarted 2024-04-14T14:17:08.097 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:17:07 smithi122 ceph-mon[25721]: Standby manager daemon x started 2024-04-14T14:17:08.097 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:17:07 smithi122 ceph-mon[25721]: from='mgr.? 172.21.15.122:0/1646783247' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-04-14T14:17:08.097 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:17:07 smithi122 ceph-mon[25721]: from='mgr.? 172.21.15.122:0/1646783247' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-14T14:17:08.097 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:17:07 smithi122 ceph-mon[25721]: from='mgr.? 172.21.15.122:0/1646783247' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-04-14T14:17:08.097 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:17:07 smithi122 ceph-mon[25721]: from='mgr.? 172.21.15.122:0/1646783247' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-14T14:17:08.899 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:17:08 smithi069 ceph-mon[24818]: mgrmap e40: y(active, since 8s), standbys: x 2024-04-14T14:17:08.899 INFO:journalctl@ceph.alertmanager.a.smithi069.stdout:Apr 14 14:17:08 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-alertmanager-a[58923]: level=info ts=2024-04-14T14:17:08.649Z caller=notify.go:735 component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify success" attempts=5 2024-04-14T14:17:08.900 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:17:08 smithi069 ceph-mon[29472]: mgrmap e40: y(active, since 8s), standbys: x 2024-04-14T14:17:09.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:17:08 smithi122 ceph-mon[25721]: mgrmap e40: y(active, since 8s), standbys: x 2024-04-14T14:17:09.884 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:17:09 smithi069 ceph-mon[24818]: pgmap v8: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 27 KiB/s rd, 0 B/s wr, 15 op/s 2024-04-14T14:17:09.884 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:17:09 smithi069 ceph-mon[29472]: pgmap v8: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 27 KiB/s rd, 0 B/s wr, 15 op/s 2024-04-14T14:17:10.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:17:09 smithi122 ceph-mon[25721]: pgmap v8: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 27 KiB/s rd, 0 B/s wr, 15 op/s 2024-04-14T14:17:11.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:17:11 smithi122 ceph-mon[25721]: pgmap v9: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 22 KiB/s rd, 0 B/s wr, 12 op/s 2024-04-14T14:17:11.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:17:11 smithi122 ceph-mon[25721]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' 2024-04-14T14:17:11.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:17:11 smithi122 ceph-mon[25721]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' 2024-04-14T14:17:11.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:17:11 smithi122 ceph-mon[25721]: from='client.? 172.21.15.69:0/1839531089' entity='client.iscsi.foo.smithi069.yuumtr' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-04-14T14:17:12.067 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:17:11 smithi069 ceph-mon[24818]: pgmap v9: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 22 KiB/s rd, 0 B/s wr, 12 op/s 2024-04-14T14:17:12.067 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:17:11 smithi069 ceph-mon[24818]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' 2024-04-14T14:17:12.067 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:17:11 smithi069 ceph-mon[24818]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' 2024-04-14T14:17:12.067 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:17:11 smithi069 ceph-mon[24818]: from='client.? 172.21.15.69:0/1839531089' entity='client.iscsi.foo.smithi069.yuumtr' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-04-14T14:17:12.068 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:17:11 smithi069 ceph-mon[29472]: pgmap v9: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 22 KiB/s rd, 0 B/s wr, 12 op/s 2024-04-14T14:17:12.068 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:17:11 smithi069 ceph-mon[29472]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' 2024-04-14T14:17:12.068 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:17:11 smithi069 ceph-mon[29472]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' 2024-04-14T14:17:12.068 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:17:11 smithi069 ceph-mon[29472]: from='client.? 172.21.15.69:0/1839531089' entity='client.iscsi.foo.smithi069.yuumtr' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-04-14T14:17:12.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:17:12 smithi122 ceph-mon[25721]: Reconfiguring prometheus.a (dependencies changed)... 2024-04-14T14:17:12.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:17:12 smithi122 ceph-mon[25721]: Reconfiguring daemon prometheus.a on smithi122 2024-04-14T14:17:12.845 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Apr 14 14:17:12 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-x[45487]: ::ffff:172.21.15.122 - - [14/Apr/2024:14:17:12] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-04-14T14:17:13.067 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:17:12 smithi069 ceph-mon[24818]: Reconfiguring prometheus.a (dependencies changed)... 2024-04-14T14:17:13.067 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:17:12 smithi069 ceph-mon[24818]: Reconfiguring daemon prometheus.a on smithi122 2024-04-14T14:17:13.067 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:17:12 smithi069 ceph-mon[29472]: Reconfiguring prometheus.a (dependencies changed)... 2024-04-14T14:17:13.067 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:17:12 smithi069 ceph-mon[29472]: Reconfiguring daemon prometheus.a on smithi122 2024-04-14T14:17:13.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:17:13 smithi122 ceph-mon[25721]: pgmap v10: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 20 KiB/s rd, 0 B/s wr, 11 op/s 2024-04-14T14:17:14.067 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:17:13 smithi069 ceph-mon[24818]: pgmap v10: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 20 KiB/s rd, 0 B/s wr, 11 op/s 2024-04-14T14:17:14.067 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:17:13 smithi069 ceph-mon[29472]: pgmap v10: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 20 KiB/s rd, 0 B/s wr, 11 op/s 2024-04-14T14:17:14.281 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Apr 14 14:17:14 smithi122 systemd[1]: Stopping Ceph prometheus.a for dc326fc2-fa65-11ee-bc8e-c7b262605968... 2024-04-14T14:17:14.282 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Apr 14 14:17:14 smithi122 bash[60767]: Error: no container with name or ID "ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-prometheus.a" found: no such container 2024-04-14T14:17:14.566 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Apr 14 14:17:14 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-prometheus-a[58033]: ts=2024-04-14T14:17:14.280Z caller=main.go:775 level=warn msg="Received SIGTERM, exiting gracefully..." 2024-04-14T14:17:14.566 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Apr 14 14:17:14 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-prometheus-a[58033]: ts=2024-04-14T14:17:14.280Z caller=main.go:798 level=info msg="Stopping scrape discovery manager..." 2024-04-14T14:17:14.566 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Apr 14 14:17:14 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-prometheus-a[58033]: ts=2024-04-14T14:17:14.280Z caller=main.go:812 level=info msg="Stopping notify discovery manager..." 2024-04-14T14:17:14.566 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Apr 14 14:17:14 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-prometheus-a[58033]: ts=2024-04-14T14:17:14.280Z caller=main.go:834 level=info msg="Stopping scrape manager..." 2024-04-14T14:17:14.566 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Apr 14 14:17:14 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-prometheus-a[58033]: ts=2024-04-14T14:17:14.280Z caller=main.go:794 level=info msg="Scrape discovery manager stopped" 2024-04-14T14:17:14.567 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Apr 14 14:17:14 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-prometheus-a[58033]: ts=2024-04-14T14:17:14.280Z caller=main.go:808 level=info msg="Notify discovery manager stopped" 2024-04-14T14:17:14.567 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Apr 14 14:17:14 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-prometheus-a[58033]: ts=2024-04-14T14:17:14.280Z caller=manager.go:945 level=info component="rule manager" msg="Stopping rule manager..." 2024-04-14T14:17:14.567 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Apr 14 14:17:14 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-prometheus-a[58033]: ts=2024-04-14T14:17:14.280Z caller=main.go:828 level=info msg="Scrape manager stopped" 2024-04-14T14:17:14.567 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Apr 14 14:17:14 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-prometheus-a[58033]: ts=2024-04-14T14:17:14.280Z caller=manager.go:955 level=info component="rule manager" msg="Rule manager stopped" 2024-04-14T14:17:14.567 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Apr 14 14:17:14 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-prometheus-a[58033]: ts=2024-04-14T14:17:14.281Z caller=notifier.go:600 level=info component=notifier msg="Stopping notification manager..." 2024-04-14T14:17:14.567 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Apr 14 14:17:14 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-prometheus-a[58033]: ts=2024-04-14T14:17:14.281Z caller=main.go:1054 level=info msg="Notifier manager stopped" 2024-04-14T14:17:14.567 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Apr 14 14:17:14 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-prometheus-a[58033]: ts=2024-04-14T14:17:14.281Z caller=main.go:1066 level=info msg="See you next time!" 2024-04-14T14:17:14.567 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Apr 14 14:17:14 smithi122 podman[60781]: 2024-04-14 14:17:14.290785866 +0000 UTC m=+0.118307353 container died 012f5f9d7df590c1b959233887ee191cf2e883f6b040d0129d635034a550d3a4 (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-prometheus-a, maintainer=The Prometheus Authors ) 2024-04-14T14:17:14.845 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Apr 14 14:17:14 smithi122 podman[60781]: 2024-04-14 14:17:14.655022603 +0000 UTC m=+0.482544093 container cleanup 012f5f9d7df590c1b959233887ee191cf2e883f6b040d0129d635034a550d3a4 (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-prometheus-a, maintainer=The Prometheus Authors ) 2024-04-14T14:17:14.845 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Apr 14 14:17:14 smithi122 bash[60781]: ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-prometheus-a 2024-04-14T14:17:14.845 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Apr 14 14:17:14 smithi122 bash[60808]: Error: no container with name or ID "ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-prometheus.a" found: no such container 2024-04-14T14:17:15.286 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Apr 14 14:17:14 smithi122 podman[60796]: 2024-04-14 14:17:14.939601307 +0000 UTC m=+0.651845202 container remove 012f5f9d7df590c1b959233887ee191cf2e883f6b040d0129d635034a550d3a4 (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-prometheus-a, maintainer=The Prometheus Authors ) 2024-04-14T14:17:15.286 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Apr 14 14:17:14 smithi122 systemd[1]: ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968@prometheus.a.service: Deactivated successfully. 2024-04-14T14:17:15.286 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Apr 14 14:17:14 smithi122 systemd[1]: Stopped Ceph prometheus.a for dc326fc2-fa65-11ee-bc8e-c7b262605968. 2024-04-14T14:17:15.286 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Apr 14 14:17:14 smithi122 systemd[1]: ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968@prometheus.a.service: Consumed 1.414s CPU time. 2024-04-14T14:17:15.286 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Apr 14 14:17:14 smithi122 systemd[1]: Starting Ceph prometheus.a for dc326fc2-fa65-11ee-bc8e-c7b262605968... 2024-04-14T14:17:15.286 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Apr 14 14:17:15 smithi122 podman[60878]: 2024-04-14 14:17:15.074892039 +0000 UTC m=+0.019273176 image pull 514e6a882f6e74806a5856468489eeff8d7106095557578da96935e4d0ba4d9d quay.io/prometheus/prometheus:v2.33.4 2024-04-14T14:17:15.286 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Apr 14 14:17:15 smithi122 podman[60878]: 2024-04-14 14:17:15.173965629 +0000 UTC m=+0.118346754 container create c567cffcd74ece4c320fd2f1d544eb67aa8f5459fc8a90173c5cc945d6dfce28 (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-prometheus-a, maintainer=The Prometheus Authors ) 2024-04-14T14:17:15.595 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Apr 14 14:17:15 smithi122 podman[60878]: 2024-04-14 14:17:15.393021061 +0000 UTC m=+0.337402187 container init c567cffcd74ece4c320fd2f1d544eb67aa8f5459fc8a90173c5cc945d6dfce28 (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-prometheus-a, maintainer=The Prometheus Authors ) 2024-04-14T14:17:15.595 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Apr 14 14:17:15 smithi122 podman[60878]: 2024-04-14 14:17:15.39551696 +0000 UTC m=+0.339898098 container start c567cffcd74ece4c320fd2f1d544eb67aa8f5459fc8a90173c5cc945d6dfce28 (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-prometheus-a, maintainer=The Prometheus Authors ) 2024-04-14T14:17:15.596 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Apr 14 14:17:15 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-prometheus-a[60892]: ts=2024-04-14T14:17:15.419Z caller=main.go:475 level=info msg="No time or size retention was set so using the default time retention" duration=15d 2024-04-14T14:17:15.596 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Apr 14 14:17:15 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-prometheus-a[60892]: ts=2024-04-14T14:17:15.419Z caller=main.go:512 level=info msg="Starting Prometheus" version="(version=2.33.4, branch=HEAD, revision=83032011a5d3e6102624fe58241a374a7201fee8)" 2024-04-14T14:17:15.596 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Apr 14 14:17:15 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-prometheus-a[60892]: ts=2024-04-14T14:17:15.419Z caller=main.go:517 level=info build_context="(go=go1.17.7, user=root@d13bf69e7be8, date=20220222-16:51:28)" 2024-04-14T14:17:15.596 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Apr 14 14:17:15 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-prometheus-a[60892]: ts=2024-04-14T14:17:15.419Z caller=main.go:518 level=info host_details="(Linux 5.14.0-435.el9.x86_64 #1 SMP PREEMPT_DYNAMIC Fri Mar 29 18:46:37 UTC 2024 x86_64 smithi122 (none))" 2024-04-14T14:17:15.596 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Apr 14 14:17:15 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-prometheus-a[60892]: ts=2024-04-14T14:17:15.419Z caller=main.go:519 level=info fd_limits="(soft=1048576, hard=1048576)" 2024-04-14T14:17:15.596 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Apr 14 14:17:15 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-prometheus-a[60892]: ts=2024-04-14T14:17:15.419Z caller=main.go:520 level=info vm_limits="(soft=unlimited, hard=unlimited)" 2024-04-14T14:17:15.596 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Apr 14 14:17:15 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-prometheus-a[60892]: ts=2024-04-14T14:17:15.420Z caller=web.go:570 level=info component=web msg="Start listening for connections" address=:9095 2024-04-14T14:17:15.596 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Apr 14 14:17:15 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-prometheus-a[60892]: ts=2024-04-14T14:17:15.421Z caller=main.go:923 level=info msg="Starting TSDB ..." 2024-04-14T14:17:15.596 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Apr 14 14:17:15 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-prometheus-a[60892]: ts=2024-04-14T14:17:15.421Z caller=tls_config.go:195 level=info component=web msg="TLS is disabled." http2=false 2024-04-14T14:17:15.596 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Apr 14 14:17:15 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-prometheus-a[60892]: ts=2024-04-14T14:17:15.423Z caller=head.go:493 level=info component=tsdb msg="Replaying on-disk memory mappable chunks if any" 2024-04-14T14:17:15.596 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Apr 14 14:17:15 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-prometheus-a[60892]: ts=2024-04-14T14:17:15.424Z caller=head.go:527 level=info component=tsdb msg="On-disk memory mappable chunks replay completed" duration=1.347719ms 2024-04-14T14:17:15.596 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Apr 14 14:17:15 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-prometheus-a[60892]: ts=2024-04-14T14:17:15.424Z caller=head.go:533 level=info component=tsdb msg="Replaying WAL, this may take a while" 2024-04-14T14:17:15.596 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Apr 14 14:17:15 smithi122 bash[60878]: c567cffcd74ece4c320fd2f1d544eb67aa8f5459fc8a90173c5cc945d6dfce28 2024-04-14T14:17:15.596 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Apr 14 14:17:15 smithi122 systemd[1]: Started Ceph prometheus.a for dc326fc2-fa65-11ee-bc8e-c7b262605968. 2024-04-14T14:17:16.064 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Apr 14 14:17:15 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-prometheus-a[60892]: ts=2024-04-14T14:17:15.635Z caller=head.go:604 level=info component=tsdb msg="WAL segment loaded" segment=0 maxSegment=5 2024-04-14T14:17:16.064 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Apr 14 14:17:15 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-prometheus-a[60892]: ts=2024-04-14T14:17:15.863Z caller=head.go:604 level=info component=tsdb msg="WAL segment loaded" segment=1 maxSegment=5 2024-04-14T14:17:16.065 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Apr 14 14:17:16 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-prometheus-a[60892]: ts=2024-04-14T14:17:16.063Z caller=head.go:604 level=info component=tsdb msg="WAL segment loaded" segment=2 maxSegment=5 2024-04-14T14:17:16.065 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:17:15 smithi122 ceph-mon[25721]: pgmap v11: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 20 KiB/s rd, 0 B/s wr, 11 op/s 2024-04-14T14:17:16.065 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:17:15 smithi122 ceph-mon[25721]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T14:17:16.065 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:17:15 smithi122 ceph-mon[25721]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' 2024-04-14T14:17:16.065 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:17:15 smithi122 ceph-mon[25721]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' 2024-04-14T14:17:16.065 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:17:15 smithi122 ceph-mon[25721]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' cmd=[{"prefix": "dashboard iscsi-gateway-list"}]: dispatch 2024-04-14T14:17:16.065 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:17:15 smithi122 ceph-mon[25721]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' 2024-04-14T14:17:16.065 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:17:15 smithi122 ceph-mon[25721]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' cmd=[{"prefix": "dashboard set-iscsi-api-ssl-verification", "value": "true"}]: dispatch 2024-04-14T14:17:16.065 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:17:15 smithi122 ceph-mon[25721]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' cmd=[{"prefix": "dashboard iscsi-gateway-add", "name": "smithi069"}]: dispatch 2024-04-14T14:17:16.065 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:17:15 smithi122 ceph-mon[25721]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' 2024-04-14T14:17:16.065 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:17:15 smithi122 ceph-mon[25721]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-04-14T14:17:16.065 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:17:15 smithi122 ceph-mon[25721]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-14T14:17:16.067 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:17:15 smithi069 ceph-mon[24818]: pgmap v11: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 20 KiB/s rd, 0 B/s wr, 11 op/s 2024-04-14T14:17:16.067 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:17:15 smithi069 ceph-mon[24818]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T14:17:16.067 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:17:15 smithi069 ceph-mon[24818]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' 2024-04-14T14:17:16.067 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:17:15 smithi069 ceph-mon[24818]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' 2024-04-14T14:17:16.067 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:17:15 smithi069 ceph-mon[24818]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' cmd=[{"prefix": "dashboard iscsi-gateway-list"}]: dispatch 2024-04-14T14:17:16.067 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:17:15 smithi069 ceph-mon[24818]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' 2024-04-14T14:17:16.067 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:17:15 smithi069 ceph-mon[24818]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' cmd=[{"prefix": "dashboard set-iscsi-api-ssl-verification", "value": "true"}]: dispatch 2024-04-14T14:17:16.067 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:17:15 smithi069 ceph-mon[24818]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' cmd=[{"prefix": "dashboard iscsi-gateway-add", "name": "smithi069"}]: dispatch 2024-04-14T14:17:16.067 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:17:15 smithi069 ceph-mon[24818]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' 2024-04-14T14:17:16.067 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:17:15 smithi069 ceph-mon[24818]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-04-14T14:17:16.068 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:17:15 smithi069 ceph-mon[24818]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-14T14:17:16.068 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:17:15 smithi069 ceph-mon[29472]: pgmap v11: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 20 KiB/s rd, 0 B/s wr, 11 op/s 2024-04-14T14:17:16.068 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:17:15 smithi069 ceph-mon[29472]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T14:17:16.068 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:17:15 smithi069 ceph-mon[29472]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' 2024-04-14T14:17:16.068 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:17:15 smithi069 ceph-mon[29472]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' 2024-04-14T14:17:16.068 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:17:15 smithi069 ceph-mon[29472]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' cmd=[{"prefix": "dashboard iscsi-gateway-list"}]: dispatch 2024-04-14T14:17:16.068 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:17:15 smithi069 ceph-mon[29472]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' 2024-04-14T14:17:16.068 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:17:15 smithi069 ceph-mon[29472]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' cmd=[{"prefix": "dashboard set-iscsi-api-ssl-verification", "value": "true"}]: dispatch 2024-04-14T14:17:16.068 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:17:15 smithi069 ceph-mon[29472]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' cmd=[{"prefix": "dashboard iscsi-gateway-add", "name": "smithi069"}]: dispatch 2024-04-14T14:17:16.068 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:17:15 smithi069 ceph-mon[29472]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' 2024-04-14T14:17:16.068 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:17:15 smithi069 ceph-mon[29472]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-04-14T14:17:16.068 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:17:15 smithi069 ceph-mon[29472]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-14T14:17:16.345 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Apr 14 14:17:16 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-prometheus-a[60892]: ts=2024-04-14T14:17:16.115Z caller=head.go:604 level=info component=tsdb msg="WAL segment loaded" segment=3 maxSegment=5 2024-04-14T14:17:16.345 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Apr 14 14:17:16 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-prometheus-a[60892]: ts=2024-04-14T14:17:16.123Z caller=head.go:604 level=info component=tsdb msg="WAL segment loaded" segment=4 maxSegment=5 2024-04-14T14:17:16.345 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Apr 14 14:17:16 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-prometheus-a[60892]: ts=2024-04-14T14:17:16.123Z caller=head.go:604 level=info component=tsdb msg="WAL segment loaded" segment=5 maxSegment=5 2024-04-14T14:17:16.345 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Apr 14 14:17:16 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-prometheus-a[60892]: ts=2024-04-14T14:17:16.123Z caller=head.go:610 level=info component=tsdb msg="WAL replay completed" checkpoint_replay_duration=47.99µs wal_replay_duration=698.983431ms total_replay_duration=700.3952ms 2024-04-14T14:17:16.345 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Apr 14 14:17:16 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-prometheus-a[60892]: ts=2024-04-14T14:17:16.128Z caller=main.go:944 level=info fs_type=XFS_SUPER_MAGIC 2024-04-14T14:17:16.345 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Apr 14 14:17:16 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-prometheus-a[60892]: ts=2024-04-14T14:17:16.128Z caller=main.go:947 level=info msg="TSDB started" 2024-04-14T14:17:16.345 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Apr 14 14:17:16 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-prometheus-a[60892]: ts=2024-04-14T14:17:16.128Z caller=main.go:1128 level=info msg="Loading configuration file" filename=/etc/prometheus/prometheus.yml 2024-04-14T14:17:16.345 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Apr 14 14:17:16 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-prometheus-a[60892]: ts=2024-04-14T14:17:16.147Z caller=main.go:1165 level=info msg="Completed loading of configuration file" filename=/etc/prometheus/prometheus.yml totalDuration=18.682841ms db_storage=681ns remote_storage=1.194µs web_handler=365ns query_engine=680ns scrape=427.609µs scrape_sd=97.036µs notify=11.362µs notify_sd=21.464µs rules=17.819588ms 2024-04-14T14:17:16.346 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Apr 14 14:17:16 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-prometheus-a[60892]: ts=2024-04-14T14:17:16.147Z caller=main.go:896 level=info msg="Server is ready to receive web requests." 2024-04-14T14:17:17.067 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:17:16 smithi069 ceph-mon[24818]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard iscsi-gateway-list"}]: dispatch 2024-04-14T14:17:17.067 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:17:16 smithi069 ceph-mon[24818]: Adding iSCSI gateway http://:@172.21.15.69:5000 to Dashboard 2024-04-14T14:17:17.067 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:17:16 smithi069 ceph-mon[24818]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-iscsi-api-ssl-verification", "value": "true"}]: dispatch 2024-04-14T14:17:17.067 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:17:16 smithi069 ceph-mon[24818]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard iscsi-gateway-add", "name": "smithi069"}]: dispatch 2024-04-14T14:17:17.067 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:17:16 smithi069 ceph-mon[24818]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-04-14T14:17:17.068 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:17:16 smithi069 ceph-mon[29472]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard iscsi-gateway-list"}]: dispatch 2024-04-14T14:17:17.068 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:17:16 smithi069 ceph-mon[29472]: Adding iSCSI gateway http://:@172.21.15.69:5000 to Dashboard 2024-04-14T14:17:17.068 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:17:16 smithi069 ceph-mon[29472]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-iscsi-api-ssl-verification", "value": "true"}]: dispatch 2024-04-14T14:17:17.068 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:17:16 smithi069 ceph-mon[29472]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard iscsi-gateway-add", "name": "smithi069"}]: dispatch 2024-04-14T14:17:17.068 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:17:16 smithi069 ceph-mon[29472]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-04-14T14:17:17.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:17:16 smithi122 ceph-mon[25721]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard iscsi-gateway-list"}]: dispatch 2024-04-14T14:17:17.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:17:16 smithi122 ceph-mon[25721]: Adding iSCSI gateway http://:@172.21.15.69:5000 to Dashboard 2024-04-14T14:17:17.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:17:16 smithi122 ceph-mon[25721]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-iscsi-api-ssl-verification", "value": "true"}]: dispatch 2024-04-14T14:17:17.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:17:16 smithi122 ceph-mon[25721]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard iscsi-gateway-add", "name": "smithi069"}]: dispatch 2024-04-14T14:17:17.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:17:16 smithi122 ceph-mon[25721]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-04-14T14:17:17.809 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:17:17 smithi122 ceph-mon[25721]: pgmap v12: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 20 KiB/s rd, 0 B/s wr, 12 op/s 2024-04-14T14:17:17.809 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:17:17 smithi122 ceph-mon[25721]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-14T14:17:17.809 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:17:17 smithi122 ceph-mon[25721]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-14T14:17:17.809 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:17:17 smithi122 ceph-mon[25721]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T14:17:17.969 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:17:17 smithi069 ceph-mon[24818]: pgmap v12: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 20 KiB/s rd, 0 B/s wr, 12 op/s 2024-04-14T14:17:17.969 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:17:17 smithi069 ceph-mon[24818]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-14T14:17:17.969 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:17:17 smithi069 ceph-mon[24818]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-14T14:17:17.969 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:17:17 smithi069 ceph-mon[24818]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T14:17:17.969 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:17:17 smithi069 ceph-mon[29472]: pgmap v12: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 20 KiB/s rd, 0 B/s wr, 12 op/s 2024-04-14T14:17:17.969 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:17:17 smithi069 ceph-mon[29472]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-14T14:17:17.969 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:17:17 smithi069 ceph-mon[29472]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-14T14:17:17.969 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:17:17 smithi069 ceph-mon[29472]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T14:17:18.734 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:17:18 smithi122 ceph-mon[25721]: Upgrade: Updating mgr.x 2024-04-14T14:17:18.735 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:17:18 smithi122 ceph-mon[25721]: Deploying daemon mgr.x on smithi122 2024-04-14T14:17:18.981 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:17:18 smithi069 ceph-mon[24818]: Upgrade: Updating mgr.x 2024-04-14T14:17:18.981 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:17:18 smithi069 ceph-mon[24818]: Deploying daemon mgr.x on smithi122 2024-04-14T14:17:18.981 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:17:18 smithi069 ceph-mon[29472]: Upgrade: Updating mgr.x 2024-04-14T14:17:18.981 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:17:18 smithi069 ceph-mon[29472]: Deploying daemon mgr.x on smithi122 2024-04-14T14:17:19.345 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Apr 14 14:17:19 smithi122 systemd[1]: Stopping Ceph mgr.x for dc326fc2-fa65-11ee-bc8e-c7b262605968... 2024-04-14T14:17:19.683 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Apr 14 14:17:19 smithi122 podman[61236]: 2024-04-14 14:17:19.352157118 +0000 UTC m=+0.135940472 container died 813744c2b615145e5f46698e8c4e176ebcb682d3e3810002ac9c8baf37da0e71 (image=quay.ceph.io/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888, name=ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-x, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, GIT_CLEAN=True, RELEASE=main-a971fd5, ceph=True, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240403, org.label-schema.vendor=CentOS) 2024-04-14T14:17:20.007 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:17:19 smithi122 ceph-mon[25721]: pgmap v13: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 682 B/s rd, 0 op/s 2024-04-14T14:17:20.007 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Apr 14 14:17:19 smithi122 podman[61236]: 2024-04-14 14:17:19.704584047 +0000 UTC m=+0.488367398 container cleanup 813744c2b615145e5f46698e8c4e176ebcb682d3e3810002ac9c8baf37da0e71 (image=quay.ceph.io/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888, name=ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-x, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.vendor=CentOS, RELEASE=main-a971fd5, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.license=GPLv2, org.label-schema.build-date=20240403, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5) 2024-04-14T14:17:20.007 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Apr 14 14:17:19 smithi122 bash[61236]: ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-x 2024-04-14T14:17:20.067 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:17:19 smithi069 ceph-mon[24818]: pgmap v13: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 682 B/s rd, 0 op/s 2024-04-14T14:17:20.067 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:17:19 smithi069 ceph-mon[29472]: pgmap v13: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 682 B/s rd, 0 op/s 2024-04-14T14:17:20.325 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Apr 14 14:17:20 smithi122 podman[61250]: 2024-04-14 14:17:20.006142897 +0000 UTC m=+0.657404417 container remove 813744c2b615145e5f46698e8c4e176ebcb682d3e3810002ac9c8baf37da0e71 (image=quay.ceph.io/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888, name=ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-x, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240403, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, RELEASE=main-a971fd5, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, io.buildah.version=1.33.5) 2024-04-14T14:17:20.325 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Apr 14 14:17:20 smithi122 systemd[1]: ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968@mgr.x.service: Main process exited, code=exited, status=143/n/a 2024-04-14T14:17:20.325 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Apr 14 14:17:20 smithi122 systemd[1]: ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968@mgr.x.service: Failed with result 'exit-code'. 2024-04-14T14:17:20.325 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Apr 14 14:17:20 smithi122 systemd[1]: Stopped Ceph mgr.x for dc326fc2-fa65-11ee-bc8e-c7b262605968. 2024-04-14T14:17:20.326 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Apr 14 14:17:20 smithi122 systemd[1]: ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968@mgr.x.service: Consumed 59.246s CPU time. 2024-04-14T14:17:20.595 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Apr 14 14:17:20 smithi122 systemd[1]: Starting Ceph mgr.x for dc326fc2-fa65-11ee-bc8e-c7b262605968... 2024-04-14T14:17:20.595 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Apr 14 14:17:20 smithi122 podman[61382]: 2024-04-14 14:17:20.445446298 +0000 UTC m=+0.017456407 image pull b96c509eef841460b3271f5fdd331860da87b004524f4ed59414aa3680183b53 quay.ceph.io/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888 2024-04-14T14:17:20.595 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Apr 14 14:17:20 smithi122 podman[61382]: 2024-04-14 14:17:20.57410722 +0000 UTC m=+0.146117375 container create e29977590b43ad1ad13ba05c4608ae8f9bc96b9ee58f26504f2954a6a4a038e4 (image=quay.ceph.io/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888, name=ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-x, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=main-a971fd5, GIT_BRANCH=HEAD, org.label-schema.build-date=20240403, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0) 2024-04-14T14:17:21.095 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Apr 14 14:17:20 smithi122 podman[61382]: 2024-04-14 14:17:20.801596274 +0000 UTC m=+0.373606383 container init e29977590b43ad1ad13ba05c4608ae8f9bc96b9ee58f26504f2954a6a4a038e4 (image=quay.ceph.io/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888, name=ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-x, org.label-schema.vendor=CentOS, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=main-a971fd5, org.label-schema.build-date=20240403, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, ceph=True, GIT_BRANCH=HEAD, GIT_CLEAN=True) 2024-04-14T14:17:21.095 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Apr 14 14:17:20 smithi122 podman[61382]: 2024-04-14 14:17:20.804050978 +0000 UTC m=+0.376061096 container start e29977590b43ad1ad13ba05c4608ae8f9bc96b9ee58f26504f2954a6a4a038e4 (image=quay.ceph.io/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888, name=ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-x, io.buildah.version=1.33.5, ceph=True, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, org.label-schema.build-date=20240403, CEPH_POINT_RELEASE=, RELEASE=main-a971fd5, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, maintainer=Guillaume Abrioux ) 2024-04-14T14:17:21.095 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Apr 14 14:17:20 smithi122 ceph-mgr[61401]: -- 172.21.15.122:0/2476142196 <== mon.2 v2:172.21.15.122:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55dfaf7505a0 con 0x55dfaf724c00 2024-04-14T14:17:21.095 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Apr 14 14:17:20 smithi122 bash[61382]: e29977590b43ad1ad13ba05c4608ae8f9bc96b9ee58f26504f2954a6a4a038e4 2024-04-14T14:17:21.095 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Apr 14 14:17:20 smithi122 systemd[1]: Started Ceph mgr.x for dc326fc2-fa65-11ee-bc8e-c7b262605968. 2024-04-14T14:17:21.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:17:21 smithi122 ceph-mon[25721]: pgmap v14: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 682 B/s rd, 0 op/s 2024-04-14T14:17:21.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:17:21 smithi122 ceph-mon[25721]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' 2024-04-14T14:17:21.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:17:21 smithi122 ceph-mon[25721]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' 2024-04-14T14:17:21.845 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Apr 14 14:17:21 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-x[61397]: 2024-04-14T14:17:21.413+0000 7fe23a96f200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-14T14:17:21.845 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Apr 14 14:17:21 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-x[61397]: 2024-04-14T14:17:21.514+0000 7fe23a96f200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-14T14:17:21.964 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:17:21 smithi069 ceph-mon[29472]: pgmap v14: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 682 B/s rd, 0 op/s 2024-04-14T14:17:21.964 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:17:21 smithi069 ceph-mon[29472]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' 2024-04-14T14:17:21.964 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:17:21 smithi069 ceph-mon[29472]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' 2024-04-14T14:17:21.964 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:17:21 smithi069 ceph-mon[24818]: pgmap v14: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 682 B/s rd, 0 op/s 2024-04-14T14:17:21.964 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:17:21 smithi069 ceph-mon[24818]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' 2024-04-14T14:17:21.964 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:17:21 smithi069 ceph-mon[24818]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' 2024-04-14T14:17:22.545 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Apr 14 14:17:22 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-x[61397]: 2024-04-14T14:17:22.197+0000 7fe23a96f200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-14T14:17:22.546 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Apr 14 14:17:22 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-x[61397]: 2024-04-14T14:17:22.294+0000 7fe23a96f200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-14T14:17:22.546 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Apr 14 14:17:22 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-x[61397]: 2024-04-14T14:17:22.446+0000 7fe23a96f200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-14T14:17:22.546 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Apr 14 14:17:22 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-x[61397]: 2024-04-14T14:17:22.543+0000 7fe23a96f200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-14T14:17:22.817 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:17:22 smithi069 ceph-mon[24818]: from='client.25417 -' entity='client.iscsi.foo.smithi069.yuumtr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T14:17:22.817 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:17:22 smithi069 ceph-mon[29472]: from='client.25417 -' entity='client.iscsi.foo.smithi069.yuumtr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T14:17:22.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:17:22 smithi122 ceph-mon[25721]: from='client.25417 -' entity='client.iscsi.foo.smithi069.yuumtr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T14:17:22.845 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Apr 14 14:17:22 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-x[61397]: 2024-04-14T14:17:22.632+0000 7fe23a96f200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-14T14:17:23.217 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Apr 14 14:17:22 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-x[61397]: 2024-04-14T14:17:22.904+0000 7fe23a96f200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-14T14:17:23.217 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Apr 14 14:17:23 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-x[61397]: 2024-04-14T14:17:23.015+0000 7fe23a96f200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-14T14:17:23.217 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Apr 14 14:17:23 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-x[61397]: 2024-04-14T14:17:23.110+0000 7fe23a96f200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-14T14:17:23.217 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Apr 14 14:17:23 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-x[61397]: 2024-04-14T14:17:23.215+0000 7fe23a96f200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-14T14:17:23.504 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Apr 14 14:17:23 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-x[61397]: 2024-04-14T14:17:23.405+0000 7fe23a96f200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-14T14:17:23.817 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:17:23 smithi069 ceph-mon[24818]: pgmap v15: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 41 KiB/s rd, 0 B/s wr, 67 op/s 2024-04-14T14:17:23.817 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:17:23 smithi069 ceph-mon[29472]: pgmap v15: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 41 KiB/s rd, 0 B/s wr, 67 op/s 2024-04-14T14:17:23.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:17:23 smithi122 ceph-mon[25721]: pgmap v15: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 41 KiB/s rd, 0 B/s wr, 67 op/s 2024-04-14T14:17:24.492 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Apr 14 14:17:24 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-x[61397]: 2024-04-14T14:17:24.407+0000 7fe23a96f200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-14T14:17:24.845 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Apr 14 14:17:24 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-x[61397]: 2024-04-14T14:17:24.542+0000 7fe23a96f200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-14T14:17:25.226 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Apr 14 14:17:24 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-x[61397]: 2024-04-14T14:17:24.910+0000 7fe23a96f200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-14T14:17:25.227 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Apr 14 14:17:25 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-x[61397]: 2024-04-14T14:17:25.010+0000 7fe23a96f200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-14T14:17:25.227 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Apr 14 14:17:25 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-x[61397]: 2024-04-14T14:17:25.224+0000 7fe23a96f200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-14T14:17:25.595 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Apr 14 14:17:25 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-x[61397]: 2024-04-14T14:17:25.423+0000 7fe23a96f200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-14T14:17:26.023 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:17:25 smithi122 ceph-mon[25721]: pgmap v16: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 41 KiB/s rd, 0 B/s wr, 67 op/s 2024-04-14T14:17:26.024 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:17:25 smithi122 ceph-mon[25721]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' 2024-04-14T14:17:26.024 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:17:25 smithi122 ceph-mon[25721]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' 2024-04-14T14:17:26.024 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:17:25 smithi122 ceph-mon[25721]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' 2024-04-14T14:17:26.024 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:17:25 smithi122 ceph-mon[25721]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' 2024-04-14T14:17:26.024 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Apr 14 14:17:25 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-x[61397]: 2024-04-14T14:17:25.747+0000 7fe23a96f200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-14T14:17:26.024 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Apr 14 14:17:26 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-x[61397]: 2024-04-14T14:17:26.021+0000 7fe23a96f200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-14T14:17:26.067 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:17:25 smithi069 ceph-mon[24818]: pgmap v16: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 41 KiB/s rd, 0 B/s wr, 67 op/s 2024-04-14T14:17:26.067 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:17:25 smithi069 ceph-mon[24818]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' 2024-04-14T14:17:26.067 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:17:25 smithi069 ceph-mon[24818]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' 2024-04-14T14:17:26.067 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:17:25 smithi069 ceph-mon[24818]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' 2024-04-14T14:17:26.067 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:17:25 smithi069 ceph-mon[24818]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' 2024-04-14T14:17:26.067 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:17:25 smithi069 ceph-mon[29472]: pgmap v16: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 41 KiB/s rd, 0 B/s wr, 67 op/s 2024-04-14T14:17:26.067 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:17:25 smithi069 ceph-mon[29472]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' 2024-04-14T14:17:26.067 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:17:25 smithi069 ceph-mon[29472]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' 2024-04-14T14:17:26.067 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:17:25 smithi069 ceph-mon[29472]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' 2024-04-14T14:17:26.067 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:17:25 smithi069 ceph-mon[29472]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' 2024-04-14T14:17:26.595 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Apr 14 14:17:26 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-x[61397]: 2024-04-14T14:17:26.284+0000 7fe23a96f200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-14T14:17:26.817 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 14 14:17:26 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-y[63038]: ::ffff:172.21.15.122 - - [14/Apr/2024:14:17:26] "GET /metrics HTTP/1.1" 200 37991 "" "Prometheus/2.33.4" 2024-04-14T14:17:27.567 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:17:27 smithi069 ceph-mon[24818]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' 2024-04-14T14:17:27.567 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:17:27 smithi069 ceph-mon[24818]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' 2024-04-14T14:17:27.567 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:17:27 smithi069 ceph-mon[24818]: pgmap v17: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 73 KiB/s rd, 0 B/s wr, 120 op/s 2024-04-14T14:17:27.567 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:17:27 smithi069 ceph-mon[29472]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' 2024-04-14T14:17:27.568 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:17:27 smithi069 ceph-mon[29472]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' 2024-04-14T14:17:27.568 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:17:27 smithi069 ceph-mon[29472]: pgmap v17: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 73 KiB/s rd, 0 B/s wr, 120 op/s 2024-04-14T14:17:27.595 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Apr 14 14:17:27 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-x[61397]: 2024-04-14T14:17:27.159+0000 7fe23a96f200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-14T14:17:27.595 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:17:27 smithi122 ceph-mon[25721]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' 2024-04-14T14:17:27.595 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:17:27 smithi122 ceph-mon[25721]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' 2024-04-14T14:17:27.595 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:17:27 smithi122 ceph-mon[25721]: pgmap v17: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 73 KiB/s rd, 0 B/s wr, 120 op/s 2024-04-14T14:17:28.077 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Apr 14 14:17:27 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-x[61397]: 2024-04-14T14:17:27.741+0000 7fe23a96f200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-14T14:17:28.078 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Apr 14 14:17:27 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-x[61397]: 2024-04-14T14:17:27.843+0000 7fe23a96f200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-14T14:17:28.078 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Apr 14 14:17:28 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-x[61397]: 2024-04-14T14:17:28.075+0000 7fe23a96f200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-14T14:17:28.345 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:17:28 smithi122 ceph-mon[25721]: Standby manager daemon x restarted 2024-04-14T14:17:28.345 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:17:28 smithi122 ceph-mon[25721]: Standby manager daemon x started 2024-04-14T14:17:28.345 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:17:28 smithi122 ceph-mon[25721]: from='mgr.? 172.21.15.122:0/1068670701' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-04-14T14:17:28.345 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:17:28 smithi122 ceph-mon[25721]: from='mgr.? 172.21.15.122:0/1068670701' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-14T14:17:28.345 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:17:28 smithi122 ceph-mon[25721]: from='mgr.? 172.21.15.122:0/1068670701' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-04-14T14:17:28.345 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:17:28 smithi122 ceph-mon[25721]: from='mgr.? 172.21.15.122:0/1068670701' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-14T14:17:28.346 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Apr 14 14:17:28 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-x[61397]: [14/Apr/2024:14:17:28] ENGINE Bus STARTING 2024-04-14T14:17:28.346 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Apr 14 14:17:28 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-x[61397]: CherryPy Checker: 2024-04-14T14:17:28.346 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Apr 14 14:17:28 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-x[61397]: The Application mounted at '' has an empty config. 2024-04-14T14:17:28.346 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Apr 14 14:17:28 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-x[61397]: 2024-04-14T14:17:28.346 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Apr 14 14:17:28 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-x[61397]: [14/Apr/2024:14:17:28] ENGINE Serving on http://:::9283 2024-04-14T14:17:28.346 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Apr 14 14:17:28 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-x[61397]: [14/Apr/2024:14:17:28] ENGINE Bus STARTED 2024-04-14T14:17:28.567 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:17:28 smithi069 ceph-mon[24818]: Standby manager daemon x restarted 2024-04-14T14:17:28.567 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:17:28 smithi069 ceph-mon[24818]: Standby manager daemon x started 2024-04-14T14:17:28.567 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:17:28 smithi069 ceph-mon[24818]: from='mgr.? 172.21.15.122:0/1068670701' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-04-14T14:17:28.567 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:17:28 smithi069 ceph-mon[24818]: from='mgr.? 172.21.15.122:0/1068670701' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-14T14:17:28.567 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:17:28 smithi069 ceph-mon[24818]: from='mgr.? 172.21.15.122:0/1068670701' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-04-14T14:17:28.567 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:17:28 smithi069 ceph-mon[24818]: from='mgr.? 172.21.15.122:0/1068670701' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-14T14:17:28.567 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:17:28 smithi069 ceph-mon[29472]: Standby manager daemon x restarted 2024-04-14T14:17:28.568 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:17:28 smithi069 ceph-mon[29472]: Standby manager daemon x started 2024-04-14T14:17:28.568 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:17:28 smithi069 ceph-mon[29472]: from='mgr.? 172.21.15.122:0/1068670701' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-04-14T14:17:28.568 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:17:28 smithi069 ceph-mon[29472]: from='mgr.? 172.21.15.122:0/1068670701' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-14T14:17:28.568 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:17:28 smithi069 ceph-mon[29472]: from='mgr.? 172.21.15.122:0/1068670701' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-04-14T14:17:28.568 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:17:28 smithi069 ceph-mon[29472]: from='mgr.? 172.21.15.122:0/1068670701' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-14T14:17:29.526 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:17:29 smithi122 ceph-mon[25721]: mgrmap e41: y(active, since 29s), standbys: x 2024-04-14T14:17:29.526 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:17:29 smithi122 ceph-mon[25721]: pgmap v18: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 72 KiB/s rd, 0 B/s wr, 119 op/s 2024-04-14T14:17:29.567 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:17:29 smithi069 ceph-mon[24818]: mgrmap e41: y(active, since 29s), standbys: x 2024-04-14T14:17:29.567 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:17:29 smithi069 ceph-mon[24818]: pgmap v18: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 72 KiB/s rd, 0 B/s wr, 119 op/s 2024-04-14T14:17:29.567 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:17:29 smithi069 ceph-mon[29472]: mgrmap e41: y(active, since 29s), standbys: x 2024-04-14T14:17:29.567 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:17:29 smithi069 ceph-mon[29472]: pgmap v18: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 72 KiB/s rd, 0 B/s wr, 119 op/s 2024-04-14T14:17:30.818 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:17:30 smithi069 ceph-mon[24818]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' 2024-04-14T14:17:30.818 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:17:30 smithi069 ceph-mon[24818]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' 2024-04-14T14:17:30.818 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:17:30 smithi069 ceph-mon[24818]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T14:17:30.818 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:17:30 smithi069 ceph-mon[24818]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-14T14:17:30.818 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:17:30 smithi069 ceph-mon[24818]: pgmap v19: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 79 KiB/s rd, 0 B/s wr, 130 op/s 2024-04-14T14:17:30.819 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:17:30 smithi069 ceph-mon[24818]: pgmap v20: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 96 KiB/s rd, 0 B/s wr, 159 op/s 2024-04-14T14:17:30.819 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:17:30 smithi069 ceph-mon[24818]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' 2024-04-14T14:17:30.819 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:17:30 smithi069 ceph-mon[24818]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-14T14:17:30.819 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:17:30 smithi069 ceph-mon[24818]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T14:17:30.819 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:17:30 smithi069 ceph-mon[24818]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' 2024-04-14T14:17:30.819 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:17:30 smithi069 ceph-mon[24818]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr.x"}]: dispatch 2024-04-14T14:17:30.819 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:17:30 smithi069 ceph-mon[24818]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr.x"}]': finished 2024-04-14T14:17:30.819 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:17:30 smithi069 ceph-mon[24818]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr.y"}]: dispatch 2024-04-14T14:17:30.819 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:17:30 smithi069 ceph-mon[24818]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr.y"}]': finished 2024-04-14T14:17:30.820 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:17:30 smithi069 ceph-mon[24818]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T14:17:30.820 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:17:30 smithi069 ceph-mon[24818]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' 2024-04-14T14:17:30.820 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:17:30 smithi069 ceph-mon[24818]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T14:17:30.820 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:17:30 smithi069 ceph-mon[24818]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' 2024-04-14T14:17:30.820 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:17:30 smithi069 ceph-mon[24818]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T14:17:30.820 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:17:30 smithi069 ceph-mon[24818]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' 2024-04-14T14:17:30.820 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:17:30 smithi069 ceph-mon[24818]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T14:17:30.820 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:17:30 smithi069 ceph-mon[24818]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' 2024-04-14T14:17:30.821 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:17:30 smithi069 ceph-mon[24818]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T14:17:30.821 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:17:30 smithi069 ceph-mon[24818]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' 2024-04-14T14:17:30.821 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:17:30 smithi069 ceph-mon[24818]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T14:17:30.821 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:17:30 smithi069 ceph-mon[24818]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' 2024-04-14T14:17:30.821 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:17:30 smithi069 ceph-mon[24818]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T14:17:30.822 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:17:30 smithi069 ceph-mon[24818]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' 2024-04-14T14:17:30.822 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:17:30 smithi069 ceph-mon[24818]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T14:17:30.822 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:17:30 smithi069 ceph-mon[29472]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' 2024-04-14T14:17:30.822 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:17:30 smithi069 ceph-mon[29472]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' 2024-04-14T14:17:30.822 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:17:30 smithi069 ceph-mon[29472]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T14:17:30.822 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:17:30 smithi069 ceph-mon[29472]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-14T14:17:30.823 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:17:30 smithi069 ceph-mon[29472]: pgmap v19: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 79 KiB/s rd, 0 B/s wr, 130 op/s 2024-04-14T14:17:30.823 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:17:30 smithi069 ceph-mon[29472]: pgmap v20: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 96 KiB/s rd, 0 B/s wr, 159 op/s 2024-04-14T14:17:30.823 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:17:30 smithi069 ceph-mon[29472]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' 2024-04-14T14:17:30.823 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:17:30 smithi069 ceph-mon[29472]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-14T14:17:30.823 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:17:30 smithi069 ceph-mon[29472]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T14:17:30.824 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:17:30 smithi069 ceph-mon[29472]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' 2024-04-14T14:17:30.824 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:17:30 smithi069 ceph-mon[29472]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr.x"}]: dispatch 2024-04-14T14:17:30.824 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:17:30 smithi069 ceph-mon[29472]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr.x"}]': finished 2024-04-14T14:17:30.824 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:17:30 smithi069 ceph-mon[29472]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr.y"}]: dispatch 2024-04-14T14:17:30.825 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:17:30 smithi069 ceph-mon[29472]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr.y"}]': finished 2024-04-14T14:17:30.825 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:17:30 smithi069 ceph-mon[29472]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T14:17:30.825 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:17:30 smithi069 ceph-mon[29472]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' 2024-04-14T14:17:30.825 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:17:30 smithi069 ceph-mon[29472]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T14:17:30.825 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:17:30 smithi069 ceph-mon[29472]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' 2024-04-14T14:17:30.825 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:17:30 smithi069 ceph-mon[29472]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T14:17:30.825 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:17:30 smithi069 ceph-mon[29472]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' 2024-04-14T14:17:30.825 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:17:30 smithi069 ceph-mon[29472]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T14:17:30.825 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:17:30 smithi069 ceph-mon[29472]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' 2024-04-14T14:17:30.825 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:17:30 smithi069 ceph-mon[29472]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T14:17:30.826 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:17:30 smithi069 ceph-mon[29472]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' 2024-04-14T14:17:30.826 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:17:30 smithi069 ceph-mon[29472]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T14:17:30.826 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:17:30 smithi069 ceph-mon[29472]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' 2024-04-14T14:17:30.826 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:17:30 smithi069 ceph-mon[29472]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T14:17:30.826 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:17:30 smithi069 ceph-mon[29472]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' 2024-04-14T14:17:30.826 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:17:30 smithi069 ceph-mon[29472]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T14:17:31.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:17:30 smithi122 ceph-mon[25721]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' 2024-04-14T14:17:31.096 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:17:30 smithi122 ceph-mon[25721]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' 2024-04-14T14:17:31.096 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:17:30 smithi122 ceph-mon[25721]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T14:17:31.096 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:17:30 smithi122 ceph-mon[25721]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-14T14:17:31.096 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:17:30 smithi122 ceph-mon[25721]: pgmap v19: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 79 KiB/s rd, 0 B/s wr, 130 op/s 2024-04-14T14:17:31.096 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:17:30 smithi122 ceph-mon[25721]: pgmap v20: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 96 KiB/s rd, 0 B/s wr, 159 op/s 2024-04-14T14:17:31.096 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:17:30 smithi122 ceph-mon[25721]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' 2024-04-14T14:17:31.096 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:17:30 smithi122 ceph-mon[25721]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-14T14:17:31.096 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:17:30 smithi122 ceph-mon[25721]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T14:17:31.096 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:17:30 smithi122 ceph-mon[25721]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' 2024-04-14T14:17:31.096 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:17:30 smithi122 ceph-mon[25721]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr.x"}]: dispatch 2024-04-14T14:17:31.096 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:17:30 smithi122 ceph-mon[25721]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr.x"}]': finished 2024-04-14T14:17:31.096 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:17:30 smithi122 ceph-mon[25721]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr.y"}]: dispatch 2024-04-14T14:17:31.096 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:17:30 smithi122 ceph-mon[25721]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr.y"}]': finished 2024-04-14T14:17:31.096 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:17:30 smithi122 ceph-mon[25721]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T14:17:31.097 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:17:30 smithi122 ceph-mon[25721]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' 2024-04-14T14:17:31.097 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:17:30 smithi122 ceph-mon[25721]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T14:17:31.097 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:17:30 smithi122 ceph-mon[25721]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' 2024-04-14T14:17:31.097 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:17:30 smithi122 ceph-mon[25721]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T14:17:31.097 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:17:30 smithi122 ceph-mon[25721]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' 2024-04-14T14:17:31.097 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:17:30 smithi122 ceph-mon[25721]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T14:17:31.097 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:17:30 smithi122 ceph-mon[25721]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' 2024-04-14T14:17:31.097 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:17:30 smithi122 ceph-mon[25721]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T14:17:31.097 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:17:30 smithi122 ceph-mon[25721]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' 2024-04-14T14:17:31.097 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:17:30 smithi122 ceph-mon[25721]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T14:17:31.097 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:17:30 smithi122 ceph-mon[25721]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' 2024-04-14T14:17:31.097 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:17:30 smithi122 ceph-mon[25721]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T14:17:31.097 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:17:30 smithi122 ceph-mon[25721]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' 2024-04-14T14:17:31.098 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:17:30 smithi122 ceph-mon[25721]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T14:17:31.817 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:17:31 smithi069 ceph-mon[29472]: Upgrade: Setting container_image for all mgr 2024-04-14T14:17:31.817 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:17:31 smithi069 ceph-mon[29472]: Upgrade: Setting container_image for all crash 2024-04-14T14:17:31.817 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:17:31 smithi069 ceph-mon[29472]: Upgrade: Setting container_image for all mds 2024-04-14T14:17:31.817 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:17:31 smithi069 ceph-mon[29472]: Upgrade: Setting container_image for all rbd-mirror 2024-04-14T14:17:31.817 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:17:31 smithi069 ceph-mon[29472]: Upgrade: Setting container_image for all cephfs-mirror 2024-04-14T14:17:31.817 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:17:31 smithi069 ceph-mon[29472]: Upgrade: Setting container_image for all ceph-exporter 2024-04-14T14:17:31.817 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:17:31 smithi069 ceph-mon[29472]: Upgrade: Setting container_image for all nfs 2024-04-14T14:17:31.817 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:17:31 smithi069 ceph-mon[29472]: Upgrade: Setting container_image for all nvmeof 2024-04-14T14:17:31.818 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:17:31 smithi069 ceph-mon[24818]: Upgrade: Setting container_image for all mgr 2024-04-14T14:17:31.818 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:17:31 smithi069 ceph-mon[24818]: Upgrade: Setting container_image for all crash 2024-04-14T14:17:31.818 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:17:31 smithi069 ceph-mon[24818]: Upgrade: Setting container_image for all mds 2024-04-14T14:17:31.818 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:17:31 smithi069 ceph-mon[24818]: Upgrade: Setting container_image for all rbd-mirror 2024-04-14T14:17:31.818 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:17:31 smithi069 ceph-mon[24818]: Upgrade: Setting container_image for all cephfs-mirror 2024-04-14T14:17:31.818 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:17:31 smithi069 ceph-mon[24818]: Upgrade: Setting container_image for all ceph-exporter 2024-04-14T14:17:31.818 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:17:31 smithi069 ceph-mon[24818]: Upgrade: Setting container_image for all nfs 2024-04-14T14:17:31.818 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:17:31 smithi069 ceph-mon[24818]: Upgrade: Setting container_image for all nvmeof 2024-04-14T14:17:32.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:17:31 smithi122 ceph-mon[25721]: Upgrade: Setting container_image for all mgr 2024-04-14T14:17:32.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:17:31 smithi122 ceph-mon[25721]: Upgrade: Setting container_image for all crash 2024-04-14T14:17:32.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:17:31 smithi122 ceph-mon[25721]: Upgrade: Setting container_image for all mds 2024-04-14T14:17:32.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:17:31 smithi122 ceph-mon[25721]: Upgrade: Setting container_image for all rbd-mirror 2024-04-14T14:17:32.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:17:31 smithi122 ceph-mon[25721]: Upgrade: Setting container_image for all cephfs-mirror 2024-04-14T14:17:32.096 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:17:31 smithi122 ceph-mon[25721]: Upgrade: Setting container_image for all ceph-exporter 2024-04-14T14:17:32.096 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:17:31 smithi122 ceph-mon[25721]: Upgrade: Setting container_image for all nfs 2024-04-14T14:17:32.096 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:17:31 smithi122 ceph-mon[25721]: Upgrade: Setting container_image for all nvmeof 2024-04-14T14:17:32.686 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:17:32 smithi069 ceph-mon[24818]: from='client.25417 -' entity='client.iscsi.foo.smithi069.yuumtr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T14:17:32.687 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:17:32 smithi069 ceph-mon[24818]: Upgrade: Updating node-exporter.a (1/2) 2024-04-14T14:17:32.687 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:17:32 smithi069 ceph-mon[24818]: Deploying daemon node-exporter.a on smithi069 2024-04-14T14:17:32.687 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:17:32 smithi069 ceph-mon[24818]: pgmap v21: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 43 KiB/s rd, 0 B/s wr, 71 op/s 2024-04-14T14:17:32.687 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:17:32 smithi069 ceph-mon[29472]: from='client.25417 -' entity='client.iscsi.foo.smithi069.yuumtr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T14:17:32.687 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:17:32 smithi069 ceph-mon[29472]: Upgrade: Updating node-exporter.a (1/2) 2024-04-14T14:17:32.687 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:17:32 smithi069 ceph-mon[29472]: Deploying daemon node-exporter.a on smithi069 2024-04-14T14:17:32.687 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:17:32 smithi069 ceph-mon[29472]: pgmap v21: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 43 KiB/s rd, 0 B/s wr, 71 op/s 2024-04-14T14:17:32.688 INFO:journalctl@ceph.node-exporter.a.smithi069.stdout:Apr 14 14:17:32 smithi069 systemd[1]: Stopping Ceph node-exporter.a for dc326fc2-fa65-11ee-bc8e-c7b262605968... 2024-04-14T14:17:32.725 INFO:teuthology.orchestra.run.smithi069.stdout:true 2024-04-14T14:17:32.951 INFO:journalctl@ceph.node-exporter.a.smithi069.stdout:Apr 14 14:17:32 smithi069 podman[76962]: 2024-04-14 14:17:32.684604723 +0000 UTC m=+0.139556265 container died ca074142ab73b87b0f942e4242375445f3e1852cd0f6dbcba0fbb79b81e1027b (image=quay.io/prometheus/node-exporter:v1.3.1, name=ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-node-exporter-a, maintainer=The Prometheus Authors ) 2024-04-14T14:17:33.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:17:32 smithi122 ceph-mon[25721]: from='client.25417 -' entity='client.iscsi.foo.smithi069.yuumtr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T14:17:33.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:17:32 smithi122 ceph-mon[25721]: Upgrade: Updating node-exporter.a (1/2) 2024-04-14T14:17:33.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:17:32 smithi122 ceph-mon[25721]: Deploying daemon node-exporter.a on smithi069 2024-04-14T14:17:33.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:17:32 smithi122 ceph-mon[25721]: pgmap v21: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 43 KiB/s rd, 0 B/s wr, 71 op/s 2024-04-14T14:17:33.317 INFO:journalctl@ceph.node-exporter.a.smithi069.stdout:Apr 14 14:17:33 smithi069 podman[76962]: 2024-04-14 14:17:33.048500001 +0000 UTC m=+0.503451544 container cleanup ca074142ab73b87b0f942e4242375445f3e1852cd0f6dbcba0fbb79b81e1027b (image=quay.io/prometheus/node-exporter:v1.3.1, name=ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-node-exporter-a, maintainer=The Prometheus Authors ) 2024-04-14T14:17:33.317 INFO:journalctl@ceph.node-exporter.a.smithi069.stdout:Apr 14 14:17:33 smithi069 bash[76962]: ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-node-exporter-a 2024-04-14T14:17:33.409 INFO:teuthology.orchestra.run.smithi069.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-14T14:17:33.410 INFO:teuthology.orchestra.run.smithi069.stdout:alertmanager.a smithi069 *:9093,9094 running (13m) 8s ago 20m 22.6M - 0.23.0 ba2b418f427c e300fc36e997 2024-04-14T14:17:33.410 INFO:teuthology.orchestra.run.smithi069.stdout:grafana.a smithi122 *:3000 error 8s ago 20m - - 2024-04-14T14:17:33.410 INFO:teuthology.orchestra.run.smithi069.stdout:iscsi.foo.smithi069.yuumtr smithi069 running (23s) 8s ago 20m 44.2M - 3.5 e1d6a67b021e 38cc81ea8cfa 2024-04-14T14:17:33.410 INFO:teuthology.orchestra.run.smithi069.stdout:mgr.x smithi122 *:8443,9283,8765 running (12s) 8s ago 25m 65.5M - 19.0.0-2778-ga971fd5f b96c509eef84 e29977590b43 2024-04-14T14:17:33.410 INFO:teuthology.orchestra.run.smithi069.stdout:mgr.y smithi069 *:8443,9283,8765 running (9m) 8s ago 27m 498M - 19.0.0-2778-ga971fd5f b96c509eef84 7af3d29d0e5f 2024-04-14T14:17:33.410 INFO:teuthology.orchestra.run.smithi069.stdout:mon.a smithi069 running (27m) 8s ago 27m 76.8M 2048M 17.2.0 e1d6a67b021e d3c553f91913 2024-04-14T14:17:33.410 INFO:teuthology.orchestra.run.smithi069.stdout:mon.b smithi122 running (25m) 8s ago 25m 57.9M 2048M 17.2.0 e1d6a67b021e 3b766bfce4be 2024-04-14T14:17:33.410 INFO:teuthology.orchestra.run.smithi069.stdout:mon.c smithi069 running (25m) 8s ago 25m 56.9M 2048M 17.2.0 e1d6a67b021e 23c078b9d1e5 2024-04-14T14:17:33.410 INFO:teuthology.orchestra.run.smithi069.stdout:node-exporter.a smithi069 *:9100 running (13m) 8s ago 21m 17.9M - 1.3.1 1dbe0e931976 ca074142ab73 2024-04-14T14:17:33.410 INFO:teuthology.orchestra.run.smithi069.stdout:node-exporter.b smithi122 *:9100 running (12m) 8s ago 21m 21.7M - 1.3.1 1dbe0e931976 996115353a0b 2024-04-14T14:17:33.410 INFO:teuthology.orchestra.run.smithi069.stdout:osd.0 smithi069 running (25m) 8s ago 25m 56.5M 2992M 17.2.0 e1d6a67b021e 46654654b6b6 2024-04-14T14:17:33.410 INFO:teuthology.orchestra.run.smithi069.stdout:osd.1 smithi069 running (24m) 8s ago 24m 54.5M 2992M 17.2.0 e1d6a67b021e cdba54a59609 2024-04-14T14:17:33.410 INFO:teuthology.orchestra.run.smithi069.stdout:osd.2 smithi069 running (24m) 8s ago 24m 50.9M 2992M 17.2.0 e1d6a67b021e bc37a1a9cba0 2024-04-14T14:17:33.411 INFO:teuthology.orchestra.run.smithi069.stdout:osd.3 smithi069 running (24m) 8s ago 24m 56.3M 2992M 17.2.0 e1d6a67b021e 0925ba3da801 2024-04-14T14:17:33.411 INFO:teuthology.orchestra.run.smithi069.stdout:osd.4 smithi122 running (23m) 8s ago 23m 56.0M 3248M 17.2.0 e1d6a67b021e c00975caf771 2024-04-14T14:17:33.411 INFO:teuthology.orchestra.run.smithi069.stdout:osd.5 smithi122 running (23m) 8s ago 23m 55.0M 3248M 17.2.0 e1d6a67b021e 5f52d217ceb8 2024-04-14T14:17:33.411 INFO:teuthology.orchestra.run.smithi069.stdout:osd.6 smithi122 running (22m) 8s ago 22m 51.7M 3248M 17.2.0 e1d6a67b021e a2acfec61475 2024-04-14T14:17:33.411 INFO:teuthology.orchestra.run.smithi069.stdout:osd.7 smithi122 running (22m) 8s ago 22m 56.7M 3248M 17.2.0 e1d6a67b021e 62411cd06801 2024-04-14T14:17:33.411 INFO:teuthology.orchestra.run.smithi069.stdout:prometheus.a smithi122 *:9095 running (18s) 8s ago 21m 53.3M - 2.33.4 514e6a882f6e c567cffcd74e 2024-04-14T14:17:33.411 INFO:teuthology.orchestra.run.smithi069.stdout:rgw.foo.smithi069.wesedz smithi069 *:8000 running (20m) 8s ago 20m 77.6M - 17.2.0 e1d6a67b021e 16de491440f3 2024-04-14T14:17:33.411 INFO:teuthology.orchestra.run.smithi069.stdout:rgw.foo.smithi122.jiobyv smithi122 *:8000 running (20m) 8s ago 20m 74.6M - 17.2.0 e1d6a67b021e 6d16de445dba 2024-04-14T14:17:33.702 INFO:journalctl@ceph.node-exporter.a.smithi069.stdout:Apr 14 14:17:33 smithi069 podman[76976]: 2024-04-14 14:17:33.316405034 +0000 UTC m=+0.635884487 container remove ca074142ab73b87b0f942e4242375445f3e1852cd0f6dbcba0fbb79b81e1027b (image=quay.io/prometheus/node-exporter:v1.3.1, name=ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-node-exporter-a, maintainer=The Prometheus Authors ) 2024-04-14T14:17:33.702 INFO:journalctl@ceph.node-exporter.a.smithi069.stdout:Apr 14 14:17:33 smithi069 systemd[1]: ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968@node-exporter.a.service: Main process exited, code=exited, status=143/n/a 2024-04-14T14:17:33.702 INFO:journalctl@ceph.node-exporter.a.smithi069.stdout:Apr 14 14:17:33 smithi069 systemd[1]: ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968@node-exporter.a.service: Failed with result 'exit-code'. 2024-04-14T14:17:33.702 INFO:journalctl@ceph.node-exporter.a.smithi069.stdout:Apr 14 14:17:33 smithi069 systemd[1]: Stopped Ceph node-exporter.a for dc326fc2-fa65-11ee-bc8e-c7b262605968. 2024-04-14T14:17:33.702 INFO:journalctl@ceph.node-exporter.a.smithi069.stdout:Apr 14 14:17:33 smithi069 systemd[1]: ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968@node-exporter.a.service: Consumed 5.127s CPU time. 2024-04-14T14:17:33.851 INFO:teuthology.orchestra.run.smithi069.stdout:{ 2024-04-14T14:17:33.851 INFO:teuthology.orchestra.run.smithi069.stdout: "mon": { 2024-04-14T14:17:33.852 INFO:teuthology.orchestra.run.smithi069.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 3 2024-04-14T14:17:33.852 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-14T14:17:33.852 INFO:teuthology.orchestra.run.smithi069.stdout: "mgr": { 2024-04-14T14:17:33.852 INFO:teuthology.orchestra.run.smithi069.stdout: "ceph version 19.0.0-2778-ga971fd5f (a971fd5f5719038c70b817bafacd6d5400b1d888) squid (dev)": 2 2024-04-14T14:17:33.852 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-14T14:17:33.852 INFO:teuthology.orchestra.run.smithi069.stdout: "osd": { 2024-04-14T14:17:33.852 INFO:teuthology.orchestra.run.smithi069.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 8 2024-04-14T14:17:33.852 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-14T14:17:33.852 INFO:teuthology.orchestra.run.smithi069.stdout: "mds": {}, 2024-04-14T14:17:33.852 INFO:teuthology.orchestra.run.smithi069.stdout: "rgw": { 2024-04-14T14:17:33.852 INFO:teuthology.orchestra.run.smithi069.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-04-14T14:17:33.852 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-14T14:17:33.852 INFO:teuthology.orchestra.run.smithi069.stdout: "overall": { 2024-04-14T14:17:33.853 INFO:teuthology.orchestra.run.smithi069.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 13, 2024-04-14T14:17:33.853 INFO:teuthology.orchestra.run.smithi069.stdout: "ceph version 19.0.0-2778-ga971fd5f (a971fd5f5719038c70b817bafacd6d5400b1d888) squid (dev)": 2 2024-04-14T14:17:33.853 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-14T14:17:33.853 INFO:teuthology.orchestra.run.smithi069.stdout:} 2024-04-14T14:17:34.067 INFO:journalctl@ceph.node-exporter.a.smithi069.stdout:Apr 14 14:17:33 smithi069 systemd[1]: Starting Ceph node-exporter.a for dc326fc2-fa65-11ee-bc8e-c7b262605968... 2024-04-14T14:17:34.067 INFO:journalctl@ceph.node-exporter.a.smithi069.stdout:Apr 14 14:17:33 smithi069 bash[77174]: Trying to pull quay.io/prometheus/node-exporter:v1.5.0... 2024-04-14T14:17:34.208 INFO:teuthology.orchestra.run.smithi069.stdout:{ 2024-04-14T14:17:34.208 INFO:teuthology.orchestra.run.smithi069.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888", 2024-04-14T14:17:34.208 INFO:teuthology.orchestra.run.smithi069.stdout: "in_progress": true, 2024-04-14T14:17:34.208 INFO:teuthology.orchestra.run.smithi069.stdout: "which": "Upgrading daemons of type(s) mgr", 2024-04-14T14:17:34.208 INFO:teuthology.orchestra.run.smithi069.stdout: "services_complete": [ 2024-04-14T14:17:34.208 INFO:teuthology.orchestra.run.smithi069.stdout: "mgr" 2024-04-14T14:17:34.208 INFO:teuthology.orchestra.run.smithi069.stdout: ], 2024-04-14T14:17:34.208 INFO:teuthology.orchestra.run.smithi069.stdout: "progress": "2/2 daemons upgraded", 2024-04-14T14:17:34.208 INFO:teuthology.orchestra.run.smithi069.stdout: "message": "Currently upgrading node-exporter daemons", 2024-04-14T14:17:34.208 INFO:teuthology.orchestra.run.smithi069.stdout: "is_paused": false 2024-04-14T14:17:34.208 INFO:teuthology.orchestra.run.smithi069.stdout:} 2024-04-14T14:17:34.567 INFO:journalctl@ceph.node-exporter.a.smithi069.stdout:Apr 14 14:17:34 smithi069 bash[77174]: Getting image source signatures 2024-04-14T14:17:34.567 INFO:journalctl@ceph.node-exporter.a.smithi069.stdout:Apr 14 14:17:34 smithi069 bash[77174]: Copying blob sha256:c0e87333d3805a7b7e1c25783dd25f27da0fe70a8e262eee7ed190c8dc843fa7 2024-04-14T14:17:34.567 INFO:journalctl@ceph.node-exporter.a.smithi069.stdout:Apr 14 14:17:34 smithi069 bash[77174]: Copying blob sha256:22b70bddd3acadc892fca4c2af4260629bfda5dfd11ebc106a93ce24e752b5ed 2024-04-14T14:17:34.567 INFO:journalctl@ceph.node-exporter.a.smithi069.stdout:Apr 14 14:17:34 smithi069 bash[77174]: Copying blob sha256:5c12815fee558b157dee7f7509dedbaba0a8379098858a65ec869e1f1526ea0c 2024-04-14T14:17:35.067 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:17:34 smithi069 ceph-mon[24818]: from='client.15534 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T14:17:35.067 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:17:34 smithi069 ceph-mon[24818]: from='client.15540 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T14:17:35.067 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:17:34 smithi069 ceph-mon[24818]: from='client.15546 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T14:17:35.067 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:17:34 smithi069 ceph-mon[24818]: pgmap v22: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 43 KiB/s rd, 0 B/s wr, 71 op/s 2024-04-14T14:17:35.067 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:17:34 smithi069 ceph-mon[24818]: from='client.? 172.21.15.69:0/3264836552' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T14:17:35.067 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:17:34 smithi069 ceph-mon[29472]: from='client.15534 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T14:17:35.068 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:17:34 smithi069 ceph-mon[29472]: from='client.15540 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T14:17:35.068 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:17:34 smithi069 ceph-mon[29472]: from='client.15546 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T14:17:35.068 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:17:34 smithi069 ceph-mon[29472]: pgmap v22: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 43 KiB/s rd, 0 B/s wr, 71 op/s 2024-04-14T14:17:35.068 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:17:34 smithi069 ceph-mon[29472]: from='client.? 172.21.15.69:0/3264836552' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T14:17:35.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:17:34 smithi122 ceph-mon[25721]: from='client.15534 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T14:17:35.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:17:34 smithi122 ceph-mon[25721]: from='client.15540 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T14:17:35.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:17:34 smithi122 ceph-mon[25721]: from='client.15546 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T14:17:35.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:17:34 smithi122 ceph-mon[25721]: pgmap v22: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 43 KiB/s rd, 0 B/s wr, 71 op/s 2024-04-14T14:17:35.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:17:34 smithi122 ceph-mon[25721]: from='client.? 172.21.15.69:0/3264836552' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T14:17:36.067 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:17:35 smithi069 ceph-mon[24818]: from='client.25459 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T14:17:36.068 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:17:35 smithi069 ceph-mon[29472]: from='client.25459 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T14:17:36.068 INFO:journalctl@ceph.node-exporter.a.smithi069.stdout:Apr 14 14:17:35 smithi069 bash[77174]: Copying config sha256:0da6a335fe1356545476b749c68f022c897de3a2139e8f0054f6937349ee2b83 2024-04-14T14:17:36.068 INFO:journalctl@ceph.node-exporter.a.smithi069.stdout:Apr 14 14:17:35 smithi069 bash[77174]: Writing manifest to image destination 2024-04-14T14:17:36.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:17:35 smithi122 ceph-mon[25721]: from='client.25459 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T14:17:36.375 INFO:journalctl@ceph.node-exporter.a.smithi069.stdout:Apr 14 14:17:36 smithi069 podman[77174]: 2024-04-14 14:17:36.054239962 +0000 UTC m=+2.252431842 image pull 0da6a335fe1356545476b749c68f022c897de3a2139e8f0054f6937349ee2b83 quay.io/prometheus/node-exporter:v1.5.0 2024-04-14T14:17:36.375 INFO:journalctl@ceph.node-exporter.a.smithi069.stdout:Apr 14 14:17:36 smithi069 podman[77174]: 2024-04-14 14:17:36.169436185 +0000 UTC m=+2.367628053 container create 0de0eb2171b9a7fe308795e7e66f4d26c5fb3cf113e7431fd200527a1a20f72c (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-node-exporter-a, maintainer=The Prometheus Authors ) 2024-04-14T14:17:36.375 INFO:journalctl@ceph.node-exporter.a.smithi069.stdout:Apr 14 14:17:36 smithi069 podman[77174]: 2024-04-14 14:17:36.371749069 +0000 UTC m=+2.569940937 container init 0de0eb2171b9a7fe308795e7e66f4d26c5fb3cf113e7431fd200527a1a20f72c (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-node-exporter-a, maintainer=The Prometheus Authors ) 2024-04-14T14:17:36.632 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 14 14:17:36 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-y[63038]: ::ffff:172.21.15.122 - - [14/Apr/2024:14:17:36] "GET /metrics HTTP/1.1" 200 37988 "" "Prometheus/2.33.4" 2024-04-14T14:17:36.633 INFO:journalctl@ceph.node-exporter.a.smithi069.stdout:Apr 14 14:17:36 smithi069 podman[77174]: 2024-04-14 14:17:36.374471247 +0000 UTC m=+2.572663121 container start 0de0eb2171b9a7fe308795e7e66f4d26c5fb3cf113e7431fd200527a1a20f72c (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-node-exporter-a, maintainer=The Prometheus Authors ) 2024-04-14T14:17:36.633 INFO:journalctl@ceph.node-exporter.a.smithi069.stdout:Apr 14 14:17:36 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-node-exporter-a[77249]: ts=2024-04-14T14:17:36.378Z caller=node_exporter.go:180 level=info msg="Starting node_exporter" version="(version=1.5.0, branch=HEAD, revision=1b48970ffcf5630534fb00bb0687d73c66d1c959)" 2024-04-14T14:17:36.633 INFO:journalctl@ceph.node-exporter.a.smithi069.stdout:Apr 14 14:17:36 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-node-exporter-a[77249]: ts=2024-04-14T14:17:36.378Z caller=node_exporter.go:181 level=info msg="Build context" build_context="(go=go1.19.3, user=root@6e7732a7b81b, date=20221129-18:59:09)" 2024-04-14T14:17:36.633 INFO:journalctl@ceph.node-exporter.a.smithi069.stdout:Apr 14 14:17:36 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-node-exporter-a[77249]: ts=2024-04-14T14:17:36.379Z caller=diskstats_common.go:111 level=info collector=diskstats msg="Parsed flag --collector.diskstats.device-exclude" flag=^(ram|loop|fd|(h|s|v|xv)d[a-z]|nvme\d+n\d+p)\d+$ 2024-04-14T14:17:36.633 INFO:journalctl@ceph.node-exporter.a.smithi069.stdout:Apr 14 14:17:36 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-node-exporter-a[77249]: ts=2024-04-14T14:17:36.379Z caller=diskstats_linux.go:264 level=error collector=diskstats msg="Failed to open directory, disabling udev device properties" path=/run/udev/data 2024-04-14T14:17:36.633 INFO:journalctl@ceph.node-exporter.a.smithi069.stdout:Apr 14 14:17:36 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-node-exporter-a[77249]: ts=2024-04-14T14:17:36.379Z caller=filesystem_common.go:111 level=info collector=filesystem msg="Parsed flag --collector.filesystem.mount-points-exclude" flag=^/(dev|proc|run/credentials/.+|sys|var/lib/docker/.+|var/lib/containers/storage/.+)($|/) 2024-04-14T14:17:36.634 INFO:journalctl@ceph.node-exporter.a.smithi069.stdout:Apr 14 14:17:36 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-node-exporter-a[77249]: ts=2024-04-14T14:17:36.379Z caller=filesystem_common.go:113 level=info collector=filesystem msg="Parsed flag --collector.filesystem.fs-types-exclude" flag=^(autofs|binfmt_misc|bpf|cgroup2?|configfs|debugfs|devpts|devtmpfs|fusectl|hugetlbfs|iso9660|mqueue|nsfs|overlay|proc|procfs|pstore|rpc_pipefs|securityfs|selinuxfs|squashfs|sysfs|tracefs)$ 2024-04-14T14:17:36.634 INFO:journalctl@ceph.node-exporter.a.smithi069.stdout:Apr 14 14:17:36 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-node-exporter-a[77249]: ts=2024-04-14T14:17:36.380Z caller=node_exporter.go:110 level=info msg="Enabled collectors" 2024-04-14T14:17:36.634 INFO:journalctl@ceph.node-exporter.a.smithi069.stdout:Apr 14 14:17:36 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-node-exporter-a[77249]: ts=2024-04-14T14:17:36.380Z caller=node_exporter.go:117 level=info collector=arp 2024-04-14T14:17:36.634 INFO:journalctl@ceph.node-exporter.a.smithi069.stdout:Apr 14 14:17:36 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-node-exporter-a[77249]: ts=2024-04-14T14:17:36.380Z caller=node_exporter.go:117 level=info collector=bcache 2024-04-14T14:17:36.634 INFO:journalctl@ceph.node-exporter.a.smithi069.stdout:Apr 14 14:17:36 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-node-exporter-a[77249]: ts=2024-04-14T14:17:36.380Z caller=node_exporter.go:117 level=info collector=bonding 2024-04-14T14:17:36.634 INFO:journalctl@ceph.node-exporter.a.smithi069.stdout:Apr 14 14:17:36 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-node-exporter-a[77249]: ts=2024-04-14T14:17:36.380Z caller=node_exporter.go:117 level=info collector=btrfs 2024-04-14T14:17:36.634 INFO:journalctl@ceph.node-exporter.a.smithi069.stdout:Apr 14 14:17:36 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-node-exporter-a[77249]: ts=2024-04-14T14:17:36.380Z caller=node_exporter.go:117 level=info collector=conntrack 2024-04-14T14:17:36.634 INFO:journalctl@ceph.node-exporter.a.smithi069.stdout:Apr 14 14:17:36 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-node-exporter-a[77249]: ts=2024-04-14T14:17:36.380Z caller=node_exporter.go:117 level=info collector=cpu 2024-04-14T14:17:36.634 INFO:journalctl@ceph.node-exporter.a.smithi069.stdout:Apr 14 14:17:36 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-node-exporter-a[77249]: ts=2024-04-14T14:17:36.380Z caller=node_exporter.go:117 level=info collector=cpufreq 2024-04-14T14:17:36.634 INFO:journalctl@ceph.node-exporter.a.smithi069.stdout:Apr 14 14:17:36 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-node-exporter-a[77249]: ts=2024-04-14T14:17:36.380Z caller=node_exporter.go:117 level=info collector=diskstats 2024-04-14T14:17:36.634 INFO:journalctl@ceph.node-exporter.a.smithi069.stdout:Apr 14 14:17:36 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-node-exporter-a[77249]: ts=2024-04-14T14:17:36.380Z caller=node_exporter.go:117 level=info collector=dmi 2024-04-14T14:17:36.635 INFO:journalctl@ceph.node-exporter.a.smithi069.stdout:Apr 14 14:17:36 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-node-exporter-a[77249]: ts=2024-04-14T14:17:36.380Z caller=node_exporter.go:117 level=info collector=edac 2024-04-14T14:17:36.635 INFO:journalctl@ceph.node-exporter.a.smithi069.stdout:Apr 14 14:17:36 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-node-exporter-a[77249]: ts=2024-04-14T14:17:36.380Z caller=node_exporter.go:117 level=info collector=entropy 2024-04-14T14:17:36.635 INFO:journalctl@ceph.node-exporter.a.smithi069.stdout:Apr 14 14:17:36 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-node-exporter-a[77249]: ts=2024-04-14T14:17:36.380Z caller=node_exporter.go:117 level=info collector=fibrechannel 2024-04-14T14:17:36.635 INFO:journalctl@ceph.node-exporter.a.smithi069.stdout:Apr 14 14:17:36 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-node-exporter-a[77249]: ts=2024-04-14T14:17:36.380Z caller=node_exporter.go:117 level=info collector=filefd 2024-04-14T14:17:36.635 INFO:journalctl@ceph.node-exporter.a.smithi069.stdout:Apr 14 14:17:36 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-node-exporter-a[77249]: ts=2024-04-14T14:17:36.380Z caller=node_exporter.go:117 level=info collector=filesystem 2024-04-14T14:17:36.635 INFO:journalctl@ceph.node-exporter.a.smithi069.stdout:Apr 14 14:17:36 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-node-exporter-a[77249]: ts=2024-04-14T14:17:36.380Z caller=node_exporter.go:117 level=info collector=hwmon 2024-04-14T14:17:36.635 INFO:journalctl@ceph.node-exporter.a.smithi069.stdout:Apr 14 14:17:36 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-node-exporter-a[77249]: ts=2024-04-14T14:17:36.380Z caller=node_exporter.go:117 level=info collector=infiniband 2024-04-14T14:17:36.635 INFO:journalctl@ceph.node-exporter.a.smithi069.stdout:Apr 14 14:17:36 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-node-exporter-a[77249]: ts=2024-04-14T14:17:36.380Z caller=node_exporter.go:117 level=info collector=ipvs 2024-04-14T14:17:36.635 INFO:journalctl@ceph.node-exporter.a.smithi069.stdout:Apr 14 14:17:36 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-node-exporter-a[77249]: ts=2024-04-14T14:17:36.380Z caller=node_exporter.go:117 level=info collector=loadavg 2024-04-14T14:17:36.636 INFO:journalctl@ceph.node-exporter.a.smithi069.stdout:Apr 14 14:17:36 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-node-exporter-a[77249]: ts=2024-04-14T14:17:36.380Z caller=node_exporter.go:117 level=info collector=mdadm 2024-04-14T14:17:36.636 INFO:journalctl@ceph.node-exporter.a.smithi069.stdout:Apr 14 14:17:36 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-node-exporter-a[77249]: ts=2024-04-14T14:17:36.380Z caller=node_exporter.go:117 level=info collector=meminfo 2024-04-14T14:17:36.636 INFO:journalctl@ceph.node-exporter.a.smithi069.stdout:Apr 14 14:17:36 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-node-exporter-a[77249]: ts=2024-04-14T14:17:36.380Z caller=node_exporter.go:117 level=info collector=netclass 2024-04-14T14:17:36.636 INFO:journalctl@ceph.node-exporter.a.smithi069.stdout:Apr 14 14:17:36 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-node-exporter-a[77249]: ts=2024-04-14T14:17:36.380Z caller=node_exporter.go:117 level=info collector=netdev 2024-04-14T14:17:36.636 INFO:journalctl@ceph.node-exporter.a.smithi069.stdout:Apr 14 14:17:36 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-node-exporter-a[77249]: ts=2024-04-14T14:17:36.380Z caller=node_exporter.go:117 level=info collector=netstat 2024-04-14T14:17:36.636 INFO:journalctl@ceph.node-exporter.a.smithi069.stdout:Apr 14 14:17:36 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-node-exporter-a[77249]: ts=2024-04-14T14:17:36.380Z caller=node_exporter.go:117 level=info collector=nfs 2024-04-14T14:17:36.636 INFO:journalctl@ceph.node-exporter.a.smithi069.stdout:Apr 14 14:17:36 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-node-exporter-a[77249]: ts=2024-04-14T14:17:36.380Z caller=node_exporter.go:117 level=info collector=nfsd 2024-04-14T14:17:36.636 INFO:journalctl@ceph.node-exporter.a.smithi069.stdout:Apr 14 14:17:36 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-node-exporter-a[77249]: ts=2024-04-14T14:17:36.380Z caller=node_exporter.go:117 level=info collector=nvme 2024-04-14T14:17:36.636 INFO:journalctl@ceph.node-exporter.a.smithi069.stdout:Apr 14 14:17:36 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-node-exporter-a[77249]: ts=2024-04-14T14:17:36.380Z caller=node_exporter.go:117 level=info collector=os 2024-04-14T14:17:36.636 INFO:journalctl@ceph.node-exporter.a.smithi069.stdout:Apr 14 14:17:36 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-node-exporter-a[77249]: ts=2024-04-14T14:17:36.380Z caller=node_exporter.go:117 level=info collector=powersupplyclass 2024-04-14T14:17:36.636 INFO:journalctl@ceph.node-exporter.a.smithi069.stdout:Apr 14 14:17:36 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-node-exporter-a[77249]: ts=2024-04-14T14:17:36.380Z caller=node_exporter.go:117 level=info collector=pressure 2024-04-14T14:17:36.636 INFO:journalctl@ceph.node-exporter.a.smithi069.stdout:Apr 14 14:17:36 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-node-exporter-a[77249]: ts=2024-04-14T14:17:36.380Z caller=node_exporter.go:117 level=info collector=rapl 2024-04-14T14:17:36.636 INFO:journalctl@ceph.node-exporter.a.smithi069.stdout:Apr 14 14:17:36 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-node-exporter-a[77249]: ts=2024-04-14T14:17:36.380Z caller=node_exporter.go:117 level=info collector=schedstat 2024-04-14T14:17:36.636 INFO:journalctl@ceph.node-exporter.a.smithi069.stdout:Apr 14 14:17:36 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-node-exporter-a[77249]: ts=2024-04-14T14:17:36.380Z caller=node_exporter.go:117 level=info collector=selinux 2024-04-14T14:17:36.636 INFO:journalctl@ceph.node-exporter.a.smithi069.stdout:Apr 14 14:17:36 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-node-exporter-a[77249]: ts=2024-04-14T14:17:36.380Z caller=node_exporter.go:117 level=info collector=sockstat 2024-04-14T14:17:36.637 INFO:journalctl@ceph.node-exporter.a.smithi069.stdout:Apr 14 14:17:36 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-node-exporter-a[77249]: ts=2024-04-14T14:17:36.380Z caller=node_exporter.go:117 level=info collector=softnet 2024-04-14T14:17:36.637 INFO:journalctl@ceph.node-exporter.a.smithi069.stdout:Apr 14 14:17:36 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-node-exporter-a[77249]: ts=2024-04-14T14:17:36.380Z caller=node_exporter.go:117 level=info collector=stat 2024-04-14T14:17:36.637 INFO:journalctl@ceph.node-exporter.a.smithi069.stdout:Apr 14 14:17:36 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-node-exporter-a[77249]: ts=2024-04-14T14:17:36.380Z caller=node_exporter.go:117 level=info collector=tapestats 2024-04-14T14:17:36.637 INFO:journalctl@ceph.node-exporter.a.smithi069.stdout:Apr 14 14:17:36 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-node-exporter-a[77249]: ts=2024-04-14T14:17:36.380Z caller=node_exporter.go:117 level=info collector=textfile 2024-04-14T14:17:36.637 INFO:journalctl@ceph.node-exporter.a.smithi069.stdout:Apr 14 14:17:36 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-node-exporter-a[77249]: ts=2024-04-14T14:17:36.380Z caller=node_exporter.go:117 level=info collector=thermal_zone 2024-04-14T14:17:36.637 INFO:journalctl@ceph.node-exporter.a.smithi069.stdout:Apr 14 14:17:36 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-node-exporter-a[77249]: ts=2024-04-14T14:17:36.380Z caller=node_exporter.go:117 level=info collector=time 2024-04-14T14:17:36.637 INFO:journalctl@ceph.node-exporter.a.smithi069.stdout:Apr 14 14:17:36 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-node-exporter-a[77249]: ts=2024-04-14T14:17:36.380Z caller=node_exporter.go:117 level=info collector=udp_queues 2024-04-14T14:17:36.637 INFO:journalctl@ceph.node-exporter.a.smithi069.stdout:Apr 14 14:17:36 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-node-exporter-a[77249]: ts=2024-04-14T14:17:36.380Z caller=node_exporter.go:117 level=info collector=uname 2024-04-14T14:17:36.637 INFO:journalctl@ceph.node-exporter.a.smithi069.stdout:Apr 14 14:17:36 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-node-exporter-a[77249]: ts=2024-04-14T14:17:36.380Z caller=node_exporter.go:117 level=info collector=vmstat 2024-04-14T14:17:36.637 INFO:journalctl@ceph.node-exporter.a.smithi069.stdout:Apr 14 14:17:36 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-node-exporter-a[77249]: ts=2024-04-14T14:17:36.380Z caller=node_exporter.go:117 level=info collector=xfs 2024-04-14T14:17:36.637 INFO:journalctl@ceph.node-exporter.a.smithi069.stdout:Apr 14 14:17:36 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-node-exporter-a[77249]: ts=2024-04-14T14:17:36.380Z caller=node_exporter.go:117 level=info collector=zfs 2024-04-14T14:17:36.637 INFO:journalctl@ceph.node-exporter.a.smithi069.stdout:Apr 14 14:17:36 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-node-exporter-a[77249]: ts=2024-04-14T14:17:36.380Z caller=tls_config.go:232 level=info msg="Listening on" address=[::]:9100 2024-04-14T14:17:36.637 INFO:journalctl@ceph.node-exporter.a.smithi069.stdout:Apr 14 14:17:36 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-node-exporter-a[77249]: ts=2024-04-14T14:17:36.380Z caller=tls_config.go:235 level=info msg="TLS is disabled." http2=false address=[::]:9100 2024-04-14T14:17:36.637 INFO:journalctl@ceph.node-exporter.a.smithi069.stdout:Apr 14 14:17:36 smithi069 bash[77174]: 0de0eb2171b9a7fe308795e7e66f4d26c5fb3cf113e7431fd200527a1a20f72c 2024-04-14T14:17:36.638 INFO:journalctl@ceph.node-exporter.a.smithi069.stdout:Apr 14 14:17:36 smithi069 systemd[1]: Started Ceph node-exporter.a for dc326fc2-fa65-11ee-bc8e-c7b262605968. 2024-04-14T14:17:36.894 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:17:36 smithi069 ceph-mon[24818]: pgmap v23: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 568 B/s rd, 0 op/s 2024-04-14T14:17:36.894 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:17:36 smithi069 ceph-mon[24818]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' 2024-04-14T14:17:36.894 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:17:36 smithi069 ceph-mon[24818]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' 2024-04-14T14:17:36.895 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:17:36 smithi069 ceph-mon[29472]: pgmap v23: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 568 B/s rd, 0 op/s 2024-04-14T14:17:36.895 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:17:36 smithi069 ceph-mon[29472]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' 2024-04-14T14:17:36.895 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:17:36 smithi069 ceph-mon[29472]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' 2024-04-14T14:17:37.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:17:36 smithi122 ceph-mon[25721]: pgmap v23: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 568 B/s rd, 0 op/s 2024-04-14T14:17:37.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:17:36 smithi122 ceph-mon[25721]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' 2024-04-14T14:17:37.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:17:36 smithi122 ceph-mon[25721]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' 2024-04-14T14:17:39.049 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:17:38 smithi122 ceph-mon[25721]: pgmap v24: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-04-14T14:17:39.049 INFO:journalctl@ceph.node-exporter.b.smithi122.stdout:Apr 14 14:17:38 smithi122 systemd[1]: Stopping Ceph node-exporter.b for dc326fc2-fa65-11ee-bc8e-c7b262605968... 2024-04-14T14:17:39.067 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:17:38 smithi069 ceph-mon[24818]: pgmap v24: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-04-14T14:17:39.067 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:17:38 smithi069 ceph-mon[29472]: pgmap v24: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-04-14T14:17:39.345 INFO:journalctl@ceph.node-exporter.b.smithi122.stdout:Apr 14 14:17:39 smithi122 podman[62742]: 2024-04-14 14:17:39.047160392 +0000 UTC m=+0.121134484 container died 996115353a0b0ad63833e7ac18a29d257d8b9f2ae3c8ab1a0efe8d419490c6ad (image=quay.io/prometheus/node-exporter:v1.3.1, name=ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-node-exporter-b, maintainer=The Prometheus Authors ) 2024-04-14T14:17:39.642 INFO:journalctl@ceph.node-exporter.b.smithi122.stdout:Apr 14 14:17:39 smithi122 podman[62742]: 2024-04-14 14:17:39.427884899 +0000 UTC m=+0.501858993 container cleanup 996115353a0b0ad63833e7ac18a29d257d8b9f2ae3c8ab1a0efe8d419490c6ad (image=quay.io/prometheus/node-exporter:v1.3.1, name=ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-node-exporter-b, maintainer=The Prometheus Authors ) 2024-04-14T14:17:39.642 INFO:journalctl@ceph.node-exporter.b.smithi122.stdout:Apr 14 14:17:39 smithi122 bash[62742]: ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-node-exporter-b 2024-04-14T14:17:39.642 INFO:journalctl@ceph.node-exporter.b.smithi122.stdout:Apr 14 14:17:39 smithi122 podman[62756]: 2024-04-14 14:17:39.604274768 +0000 UTC m=+0.561139860 container remove 996115353a0b0ad63833e7ac18a29d257d8b9f2ae3c8ab1a0efe8d419490c6ad (image=quay.io/prometheus/node-exporter:v1.3.1, name=ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-node-exporter-b, maintainer=The Prometheus Authors ) 2024-04-14T14:17:39.642 INFO:journalctl@ceph.node-exporter.b.smithi122.stdout:Apr 14 14:17:39 smithi122 systemd[1]: ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968@node-exporter.b.service: Main process exited, code=exited, status=143/n/a 2024-04-14T14:17:40.032 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:17:39 smithi122 ceph-mon[25721]: Upgrade: Updating node-exporter.b (2/2) 2024-04-14T14:17:40.032 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:17:39 smithi122 ceph-mon[25721]: Deploying daemon node-exporter.b on smithi122 2024-04-14T14:17:40.032 INFO:journalctl@ceph.node-exporter.b.smithi122.stdout:Apr 14 14:17:39 smithi122 systemd[1]: ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968@node-exporter.b.service: Failed with result 'exit-code'. 2024-04-14T14:17:40.033 INFO:journalctl@ceph.node-exporter.b.smithi122.stdout:Apr 14 14:17:39 smithi122 systemd[1]: Stopped Ceph node-exporter.b for dc326fc2-fa65-11ee-bc8e-c7b262605968. 2024-04-14T14:17:40.033 INFO:journalctl@ceph.node-exporter.b.smithi122.stdout:Apr 14 14:17:39 smithi122 systemd[1]: ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968@node-exporter.b.service: Consumed 5.091s CPU time. 2024-04-14T14:17:40.033 INFO:journalctl@ceph.node-exporter.b.smithi122.stdout:Apr 14 14:17:40 smithi122 systemd[1]: Starting Ceph node-exporter.b for dc326fc2-fa65-11ee-bc8e-c7b262605968... 2024-04-14T14:17:40.067 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:17:39 smithi069 ceph-mon[24818]: Upgrade: Updating node-exporter.b (2/2) 2024-04-14T14:17:40.067 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:17:39 smithi069 ceph-mon[24818]: Deploying daemon node-exporter.b on smithi122 2024-04-14T14:17:40.067 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:17:39 smithi069 ceph-mon[29472]: Upgrade: Updating node-exporter.b (2/2) 2024-04-14T14:17:40.067 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:17:39 smithi069 ceph-mon[29472]: Deploying daemon node-exporter.b on smithi122 2024-04-14T14:17:40.345 INFO:journalctl@ceph.node-exporter.b.smithi122.stdout:Apr 14 14:17:40 smithi122 bash[62891]: Trying to pull quay.io/prometheus/node-exporter:v1.5.0... 2024-04-14T14:17:41.069 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:17:40 smithi069 ceph-mon[24818]: pgmap v25: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-14T14:17:41.070 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:17:40 smithi069 ceph-mon[29472]: pgmap v25: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-14T14:17:41.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:17:40 smithi122 ceph-mon[25721]: pgmap v25: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-14T14:17:41.095 INFO:journalctl@ceph.node-exporter.b.smithi122.stdout:Apr 14 14:17:40 smithi122 bash[62891]: Getting image source signatures 2024-04-14T14:17:41.095 INFO:journalctl@ceph.node-exporter.b.smithi122.stdout:Apr 14 14:17:40 smithi122 bash[62891]: Copying blob sha256:22b70bddd3acadc892fca4c2af4260629bfda5dfd11ebc106a93ce24e752b5ed 2024-04-14T14:17:41.095 INFO:journalctl@ceph.node-exporter.b.smithi122.stdout:Apr 14 14:17:40 smithi122 bash[62891]: Copying blob sha256:c0e87333d3805a7b7e1c25783dd25f27da0fe70a8e262eee7ed190c8dc843fa7 2024-04-14T14:17:41.095 INFO:journalctl@ceph.node-exporter.b.smithi122.stdout:Apr 14 14:17:40 smithi122 bash[62891]: Copying blob sha256:5c12815fee558b157dee7f7509dedbaba0a8379098858a65ec869e1f1526ea0c 2024-04-14T14:17:42.980 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:17:42 smithi122 ceph-mon[25721]: from='client.25417 -' entity='client.iscsi.foo.smithi069.yuumtr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T14:17:42.980 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:17:42 smithi122 ceph-mon[25721]: pgmap v26: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:17:42.980 INFO:journalctl@ceph.node-exporter.b.smithi122.stdout:Apr 14 14:17:42 smithi122 bash[62891]: Copying config sha256:0da6a335fe1356545476b749c68f022c897de3a2139e8f0054f6937349ee2b83 2024-04-14T14:17:42.980 INFO:journalctl@ceph.node-exporter.b.smithi122.stdout:Apr 14 14:17:42 smithi122 bash[62891]: Writing manifest to image destination 2024-04-14T14:17:43.067 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:17:42 smithi069 ceph-mon[24818]: from='client.25417 -' entity='client.iscsi.foo.smithi069.yuumtr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T14:17:43.067 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:17:42 smithi069 ceph-mon[24818]: pgmap v26: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:17:43.067 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:17:42 smithi069 ceph-mon[29472]: from='client.25417 -' entity='client.iscsi.foo.smithi069.yuumtr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T14:17:43.067 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:17:42 smithi069 ceph-mon[29472]: pgmap v26: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:17:43.506 INFO:journalctl@ceph.node-exporter.b.smithi122.stdout:Apr 14 14:17:43 smithi122 podman[62891]: 2024-04-14 14:17:43.233827081 +0000 UTC m=+3.106546966 image pull 0da6a335fe1356545476b749c68f022c897de3a2139e8f0054f6937349ee2b83 quay.io/prometheus/node-exporter:v1.5.0 2024-04-14T14:17:43.506 INFO:journalctl@ceph.node-exporter.b.smithi122.stdout:Apr 14 14:17:43 smithi122 podman[62891]: 2024-04-14 14:17:43.374705259 +0000 UTC m=+3.247425136 container create 67877bf02d8e884b1552ebfe5d91321d808459c31fa43b600784b6cc960cbc27 (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-node-exporter-b, maintainer=The Prometheus Authors ) 2024-04-14T14:17:43.845 INFO:journalctl@ceph.node-exporter.b.smithi122.stdout:Apr 14 14:17:43 smithi122 podman[62891]: 2024-04-14 14:17:43.58500108 +0000 UTC m=+3.457720968 container init 67877bf02d8e884b1552ebfe5d91321d808459c31fa43b600784b6cc960cbc27 (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-node-exporter-b, maintainer=The Prometheus Authors ) 2024-04-14T14:17:43.845 INFO:journalctl@ceph.node-exporter.b.smithi122.stdout:Apr 14 14:17:43 smithi122 podman[62891]: 2024-04-14 14:17:43.587780247 +0000 UTC m=+3.460500125 container start 67877bf02d8e884b1552ebfe5d91321d808459c31fa43b600784b6cc960cbc27 (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-node-exporter-b, maintainer=The Prometheus Authors ) 2024-04-14T14:17:43.846 INFO:journalctl@ceph.node-exporter.b.smithi122.stdout:Apr 14 14:17:43 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-node-exporter-b[62941]: ts=2024-04-14T14:17:43.591Z caller=node_exporter.go:180 level=info msg="Starting node_exporter" version="(version=1.5.0, branch=HEAD, revision=1b48970ffcf5630534fb00bb0687d73c66d1c959)" 2024-04-14T14:17:43.846 INFO:journalctl@ceph.node-exporter.b.smithi122.stdout:Apr 14 14:17:43 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-node-exporter-b[62941]: ts=2024-04-14T14:17:43.591Z caller=node_exporter.go:181 level=info msg="Build context" build_context="(go=go1.19.3, user=root@6e7732a7b81b, date=20221129-18:59:09)" 2024-04-14T14:17:43.846 INFO:journalctl@ceph.node-exporter.b.smithi122.stdout:Apr 14 14:17:43 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-node-exporter-b[62941]: ts=2024-04-14T14:17:43.592Z caller=filesystem_common.go:111 level=info collector=filesystem msg="Parsed flag --collector.filesystem.mount-points-exclude" flag=^/(dev|proc|run/credentials/.+|sys|var/lib/docker/.+|var/lib/containers/storage/.+)($|/) 2024-04-14T14:17:43.846 INFO:journalctl@ceph.node-exporter.b.smithi122.stdout:Apr 14 14:17:43 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-node-exporter-b[62941]: ts=2024-04-14T14:17:43.592Z caller=filesystem_common.go:113 level=info collector=filesystem msg="Parsed flag --collector.filesystem.fs-types-exclude" flag=^(autofs|binfmt_misc|bpf|cgroup2?|configfs|debugfs|devpts|devtmpfs|fusectl|hugetlbfs|iso9660|mqueue|nsfs|overlay|proc|procfs|pstore|rpc_pipefs|securityfs|selinuxfs|squashfs|sysfs|tracefs)$ 2024-04-14T14:17:43.846 INFO:journalctl@ceph.node-exporter.b.smithi122.stdout:Apr 14 14:17:43 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-node-exporter-b[62941]: ts=2024-04-14T14:17:43.592Z caller=diskstats_common.go:111 level=info collector=diskstats msg="Parsed flag --collector.diskstats.device-exclude" flag=^(ram|loop|fd|(h|s|v|xv)d[a-z]|nvme\d+n\d+p)\d+$ 2024-04-14T14:17:43.846 INFO:journalctl@ceph.node-exporter.b.smithi122.stdout:Apr 14 14:17:43 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-node-exporter-b[62941]: ts=2024-04-14T14:17:43.592Z caller=diskstats_linux.go:264 level=error collector=diskstats msg="Failed to open directory, disabling udev device properties" path=/run/udev/data 2024-04-14T14:17:43.846 INFO:journalctl@ceph.node-exporter.b.smithi122.stdout:Apr 14 14:17:43 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-node-exporter-b[62941]: ts=2024-04-14T14:17:43.593Z caller=node_exporter.go:110 level=info msg="Enabled collectors" 2024-04-14T14:17:43.846 INFO:journalctl@ceph.node-exporter.b.smithi122.stdout:Apr 14 14:17:43 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-node-exporter-b[62941]: ts=2024-04-14T14:17:43.593Z caller=node_exporter.go:117 level=info collector=arp 2024-04-14T14:17:43.846 INFO:journalctl@ceph.node-exporter.b.smithi122.stdout:Apr 14 14:17:43 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-node-exporter-b[62941]: ts=2024-04-14T14:17:43.593Z caller=node_exporter.go:117 level=info collector=bcache 2024-04-14T14:17:43.846 INFO:journalctl@ceph.node-exporter.b.smithi122.stdout:Apr 14 14:17:43 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-node-exporter-b[62941]: ts=2024-04-14T14:17:43.593Z caller=node_exporter.go:117 level=info collector=bonding 2024-04-14T14:17:43.846 INFO:journalctl@ceph.node-exporter.b.smithi122.stdout:Apr 14 14:17:43 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-node-exporter-b[62941]: ts=2024-04-14T14:17:43.593Z caller=node_exporter.go:117 level=info collector=btrfs 2024-04-14T14:17:43.846 INFO:journalctl@ceph.node-exporter.b.smithi122.stdout:Apr 14 14:17:43 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-node-exporter-b[62941]: ts=2024-04-14T14:17:43.593Z caller=node_exporter.go:117 level=info collector=conntrack 2024-04-14T14:17:43.846 INFO:journalctl@ceph.node-exporter.b.smithi122.stdout:Apr 14 14:17:43 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-node-exporter-b[62941]: ts=2024-04-14T14:17:43.593Z caller=node_exporter.go:117 level=info collector=cpu 2024-04-14T14:17:43.847 INFO:journalctl@ceph.node-exporter.b.smithi122.stdout:Apr 14 14:17:43 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-node-exporter-b[62941]: ts=2024-04-14T14:17:43.593Z caller=node_exporter.go:117 level=info collector=cpufreq 2024-04-14T14:17:43.847 INFO:journalctl@ceph.node-exporter.b.smithi122.stdout:Apr 14 14:17:43 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-node-exporter-b[62941]: ts=2024-04-14T14:17:43.593Z caller=node_exporter.go:117 level=info collector=diskstats 2024-04-14T14:17:43.847 INFO:journalctl@ceph.node-exporter.b.smithi122.stdout:Apr 14 14:17:43 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-node-exporter-b[62941]: ts=2024-04-14T14:17:43.593Z caller=node_exporter.go:117 level=info collector=dmi 2024-04-14T14:17:43.847 INFO:journalctl@ceph.node-exporter.b.smithi122.stdout:Apr 14 14:17:43 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-node-exporter-b[62941]: ts=2024-04-14T14:17:43.593Z caller=node_exporter.go:117 level=info collector=edac 2024-04-14T14:17:43.847 INFO:journalctl@ceph.node-exporter.b.smithi122.stdout:Apr 14 14:17:43 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-node-exporter-b[62941]: ts=2024-04-14T14:17:43.593Z caller=node_exporter.go:117 level=info collector=entropy 2024-04-14T14:17:43.847 INFO:journalctl@ceph.node-exporter.b.smithi122.stdout:Apr 14 14:17:43 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-node-exporter-b[62941]: ts=2024-04-14T14:17:43.593Z caller=node_exporter.go:117 level=info collector=fibrechannel 2024-04-14T14:17:43.847 INFO:journalctl@ceph.node-exporter.b.smithi122.stdout:Apr 14 14:17:43 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-node-exporter-b[62941]: ts=2024-04-14T14:17:43.593Z caller=node_exporter.go:117 level=info collector=filefd 2024-04-14T14:17:43.847 INFO:journalctl@ceph.node-exporter.b.smithi122.stdout:Apr 14 14:17:43 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-node-exporter-b[62941]: ts=2024-04-14T14:17:43.593Z caller=node_exporter.go:117 level=info collector=filesystem 2024-04-14T14:17:43.847 INFO:journalctl@ceph.node-exporter.b.smithi122.stdout:Apr 14 14:17:43 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-node-exporter-b[62941]: ts=2024-04-14T14:17:43.593Z caller=node_exporter.go:117 level=info collector=hwmon 2024-04-14T14:17:43.847 INFO:journalctl@ceph.node-exporter.b.smithi122.stdout:Apr 14 14:17:43 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-node-exporter-b[62941]: ts=2024-04-14T14:17:43.593Z caller=node_exporter.go:117 level=info collector=infiniband 2024-04-14T14:17:43.847 INFO:journalctl@ceph.node-exporter.b.smithi122.stdout:Apr 14 14:17:43 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-node-exporter-b[62941]: ts=2024-04-14T14:17:43.593Z caller=node_exporter.go:117 level=info collector=ipvs 2024-04-14T14:17:43.847 INFO:journalctl@ceph.node-exporter.b.smithi122.stdout:Apr 14 14:17:43 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-node-exporter-b[62941]: ts=2024-04-14T14:17:43.593Z caller=node_exporter.go:117 level=info collector=loadavg 2024-04-14T14:17:43.847 INFO:journalctl@ceph.node-exporter.b.smithi122.stdout:Apr 14 14:17:43 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-node-exporter-b[62941]: ts=2024-04-14T14:17:43.593Z caller=node_exporter.go:117 level=info collector=mdadm 2024-04-14T14:17:43.847 INFO:journalctl@ceph.node-exporter.b.smithi122.stdout:Apr 14 14:17:43 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-node-exporter-b[62941]: ts=2024-04-14T14:17:43.593Z caller=node_exporter.go:117 level=info collector=meminfo 2024-04-14T14:17:43.847 INFO:journalctl@ceph.node-exporter.b.smithi122.stdout:Apr 14 14:17:43 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-node-exporter-b[62941]: ts=2024-04-14T14:17:43.593Z caller=node_exporter.go:117 level=info collector=netclass 2024-04-14T14:17:43.848 INFO:journalctl@ceph.node-exporter.b.smithi122.stdout:Apr 14 14:17:43 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-node-exporter-b[62941]: ts=2024-04-14T14:17:43.593Z caller=node_exporter.go:117 level=info collector=netdev 2024-04-14T14:17:43.848 INFO:journalctl@ceph.node-exporter.b.smithi122.stdout:Apr 14 14:17:43 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-node-exporter-b[62941]: ts=2024-04-14T14:17:43.593Z caller=node_exporter.go:117 level=info collector=netstat 2024-04-14T14:17:43.848 INFO:journalctl@ceph.node-exporter.b.smithi122.stdout:Apr 14 14:17:43 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-node-exporter-b[62941]: ts=2024-04-14T14:17:43.593Z caller=node_exporter.go:117 level=info collector=nfs 2024-04-14T14:17:43.848 INFO:journalctl@ceph.node-exporter.b.smithi122.stdout:Apr 14 14:17:43 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-node-exporter-b[62941]: ts=2024-04-14T14:17:43.593Z caller=node_exporter.go:117 level=info collector=nfsd 2024-04-14T14:17:43.848 INFO:journalctl@ceph.node-exporter.b.smithi122.stdout:Apr 14 14:17:43 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-node-exporter-b[62941]: ts=2024-04-14T14:17:43.593Z caller=node_exporter.go:117 level=info collector=nvme 2024-04-14T14:17:43.848 INFO:journalctl@ceph.node-exporter.b.smithi122.stdout:Apr 14 14:17:43 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-node-exporter-b[62941]: ts=2024-04-14T14:17:43.593Z caller=node_exporter.go:117 level=info collector=os 2024-04-14T14:17:43.848 INFO:journalctl@ceph.node-exporter.b.smithi122.stdout:Apr 14 14:17:43 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-node-exporter-b[62941]: ts=2024-04-14T14:17:43.593Z caller=node_exporter.go:117 level=info collector=powersupplyclass 2024-04-14T14:17:43.848 INFO:journalctl@ceph.node-exporter.b.smithi122.stdout:Apr 14 14:17:43 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-node-exporter-b[62941]: ts=2024-04-14T14:17:43.593Z caller=node_exporter.go:117 level=info collector=pressure 2024-04-14T14:17:43.848 INFO:journalctl@ceph.node-exporter.b.smithi122.stdout:Apr 14 14:17:43 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-node-exporter-b[62941]: ts=2024-04-14T14:17:43.593Z caller=node_exporter.go:117 level=info collector=rapl 2024-04-14T14:17:43.848 INFO:journalctl@ceph.node-exporter.b.smithi122.stdout:Apr 14 14:17:43 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-node-exporter-b[62941]: ts=2024-04-14T14:17:43.593Z caller=node_exporter.go:117 level=info collector=schedstat 2024-04-14T14:17:43.849 INFO:journalctl@ceph.node-exporter.b.smithi122.stdout:Apr 14 14:17:43 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-node-exporter-b[62941]: ts=2024-04-14T14:17:43.593Z caller=node_exporter.go:117 level=info collector=selinux 2024-04-14T14:17:43.849 INFO:journalctl@ceph.node-exporter.b.smithi122.stdout:Apr 14 14:17:43 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-node-exporter-b[62941]: ts=2024-04-14T14:17:43.593Z caller=node_exporter.go:117 level=info collector=sockstat 2024-04-14T14:17:43.849 INFO:journalctl@ceph.node-exporter.b.smithi122.stdout:Apr 14 14:17:43 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-node-exporter-b[62941]: ts=2024-04-14T14:17:43.593Z caller=node_exporter.go:117 level=info collector=softnet 2024-04-14T14:17:43.849 INFO:journalctl@ceph.node-exporter.b.smithi122.stdout:Apr 14 14:17:43 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-node-exporter-b[62941]: ts=2024-04-14T14:17:43.593Z caller=node_exporter.go:117 level=info collector=stat 2024-04-14T14:17:43.849 INFO:journalctl@ceph.node-exporter.b.smithi122.stdout:Apr 14 14:17:43 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-node-exporter-b[62941]: ts=2024-04-14T14:17:43.593Z caller=node_exporter.go:117 level=info collector=tapestats 2024-04-14T14:17:43.849 INFO:journalctl@ceph.node-exporter.b.smithi122.stdout:Apr 14 14:17:43 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-node-exporter-b[62941]: ts=2024-04-14T14:17:43.593Z caller=node_exporter.go:117 level=info collector=textfile 2024-04-14T14:17:43.849 INFO:journalctl@ceph.node-exporter.b.smithi122.stdout:Apr 14 14:17:43 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-node-exporter-b[62941]: ts=2024-04-14T14:17:43.593Z caller=node_exporter.go:117 level=info collector=thermal_zone 2024-04-14T14:17:43.849 INFO:journalctl@ceph.node-exporter.b.smithi122.stdout:Apr 14 14:17:43 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-node-exporter-b[62941]: ts=2024-04-14T14:17:43.593Z caller=node_exporter.go:117 level=info collector=time 2024-04-14T14:17:43.849 INFO:journalctl@ceph.node-exporter.b.smithi122.stdout:Apr 14 14:17:43 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-node-exporter-b[62941]: ts=2024-04-14T14:17:43.593Z caller=node_exporter.go:117 level=info collector=udp_queues 2024-04-14T14:17:43.849 INFO:journalctl@ceph.node-exporter.b.smithi122.stdout:Apr 14 14:17:43 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-node-exporter-b[62941]: ts=2024-04-14T14:17:43.593Z caller=node_exporter.go:117 level=info collector=uname 2024-04-14T14:17:43.849 INFO:journalctl@ceph.node-exporter.b.smithi122.stdout:Apr 14 14:17:43 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-node-exporter-b[62941]: ts=2024-04-14T14:17:43.593Z caller=node_exporter.go:117 level=info collector=vmstat 2024-04-14T14:17:43.849 INFO:journalctl@ceph.node-exporter.b.smithi122.stdout:Apr 14 14:17:43 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-node-exporter-b[62941]: ts=2024-04-14T14:17:43.593Z caller=node_exporter.go:117 level=info collector=xfs 2024-04-14T14:17:43.849 INFO:journalctl@ceph.node-exporter.b.smithi122.stdout:Apr 14 14:17:43 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-node-exporter-b[62941]: ts=2024-04-14T14:17:43.593Z caller=node_exporter.go:117 level=info collector=zfs 2024-04-14T14:17:43.849 INFO:journalctl@ceph.node-exporter.b.smithi122.stdout:Apr 14 14:17:43 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-node-exporter-b[62941]: ts=2024-04-14T14:17:43.593Z caller=tls_config.go:232 level=info msg="Listening on" address=[::]:9100 2024-04-14T14:17:43.849 INFO:journalctl@ceph.node-exporter.b.smithi122.stdout:Apr 14 14:17:43 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-node-exporter-b[62941]: ts=2024-04-14T14:17:43.593Z caller=tls_config.go:235 level=info msg="TLS is disabled." http2=false address=[::]:9100 2024-04-14T14:17:43.850 INFO:journalctl@ceph.node-exporter.b.smithi122.stdout:Apr 14 14:17:43 smithi122 bash[62891]: 67877bf02d8e884b1552ebfe5d91321d808459c31fa43b600784b6cc960cbc27 2024-04-14T14:17:43.850 INFO:journalctl@ceph.node-exporter.b.smithi122.stdout:Apr 14 14:17:43 smithi122 systemd[1]: Started Ceph node-exporter.b for dc326fc2-fa65-11ee-bc8e-c7b262605968. 2024-04-14T14:17:44.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:17:44 smithi122 ceph-mon[25721]: pgmap v27: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:17:44.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:17:44 smithi122 ceph-mon[25721]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' 2024-04-14T14:17:44.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:17:44 smithi122 ceph-mon[25721]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' 2024-04-14T14:17:45.067 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:17:44 smithi069 ceph-mon[24818]: pgmap v27: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:17:45.067 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:17:44 smithi069 ceph-mon[24818]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' 2024-04-14T14:17:45.067 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:17:44 smithi069 ceph-mon[24818]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' 2024-04-14T14:17:45.067 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:17:44 smithi069 ceph-mon[29472]: pgmap v27: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:17:45.067 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:17:44 smithi069 ceph-mon[29472]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' 2024-04-14T14:17:45.067 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:17:44 smithi069 ceph-mon[29472]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' 2024-04-14T14:17:46.067 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:17:45 smithi069 ceph-mon[24818]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T14:17:46.067 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:17:45 smithi069 ceph-mon[29472]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T14:17:46.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:17:45 smithi122 ceph-mon[25721]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T14:17:46.817 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:17:46 smithi069 ceph-mon[24818]: pgmap v28: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:17:46.817 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 14 14:17:46 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-y[63038]: ::ffff:172.21.15.122 - - [14/Apr/2024:14:17:46] "GET /metrics HTTP/1.1" 200 37988 "" "Prometheus/2.33.4" 2024-04-14T14:17:46.817 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:17:46 smithi069 ceph-mon[29472]: pgmap v28: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:17:46.932 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:17:46 smithi122 ceph-mon[25721]: pgmap v28: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:17:48.817 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:17:48 smithi069 ceph-mon[24818]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' 2024-04-14T14:17:48.817 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:17:48 smithi069 ceph-mon[24818]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' 2024-04-14T14:17:48.817 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:17:48 smithi069 ceph-mon[24818]: pgmap v29: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:17:48.817 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:17:48 smithi069 ceph-mon[24818]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' 2024-04-14T14:17:48.817 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:17:48 smithi069 ceph-mon[24818]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' 2024-04-14T14:17:48.817 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:17:48 smithi069 ceph-mon[29472]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' 2024-04-14T14:17:48.818 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:17:48 smithi069 ceph-mon[29472]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' 2024-04-14T14:17:48.818 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:17:48 smithi069 ceph-mon[29472]: pgmap v29: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:17:48.818 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:17:48 smithi069 ceph-mon[29472]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' 2024-04-14T14:17:48.818 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:17:48 smithi069 ceph-mon[29472]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' 2024-04-14T14:17:48.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:17:48 smithi122 ceph-mon[25721]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' 2024-04-14T14:17:48.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:17:48 smithi122 ceph-mon[25721]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' 2024-04-14T14:17:48.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:17:48 smithi122 ceph-mon[25721]: pgmap v29: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:17:48.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:17:48 smithi122 ceph-mon[25721]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' 2024-04-14T14:17:48.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:17:48 smithi122 ceph-mon[25721]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' 2024-04-14T14:17:49.843 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:17:49 smithi069 ceph-mon[29472]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' 2024-04-14T14:17:49.843 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:17:49 smithi069 ceph-mon[29472]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' 2024-04-14T14:17:49.843 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:17:49 smithi069 ceph-mon[29472]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' 2024-04-14T14:17:49.843 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:17:49 smithi069 ceph-mon[29472]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' 2024-04-14T14:17:49.843 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:17:49 smithi069 ceph-mon[24818]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' 2024-04-14T14:17:49.843 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:17:49 smithi069 ceph-mon[24818]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' 2024-04-14T14:17:49.843 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:17:49 smithi069 ceph-mon[24818]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' 2024-04-14T14:17:49.844 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:17:49 smithi069 ceph-mon[24818]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' 2024-04-14T14:17:49.844 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:17:49 smithi122 ceph-mon[25721]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' 2024-04-14T14:17:49.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:17:49 smithi122 ceph-mon[25721]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' 2024-04-14T14:17:49.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:17:49 smithi122 ceph-mon[25721]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' 2024-04-14T14:17:49.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:17:49 smithi122 ceph-mon[25721]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' 2024-04-14T14:17:50.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:17:50 smithi122 ceph-mon[25721]: pgmap v30: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:17:50.847 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:17:50 smithi069 ceph-mon[24818]: pgmap v30: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:17:50.847 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:17:50 smithi069 ceph-mon[29472]: pgmap v30: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:17:52.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:17:52 smithi122 ceph-mon[25721]: from='client.25417 -' entity='client.iscsi.foo.smithi069.yuumtr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T14:17:52.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:17:52 smithi122 ceph-mon[25721]: pgmap v31: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:17:52.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:17:52 smithi122 ceph-mon[25721]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' 2024-04-14T14:17:52.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:17:52 smithi122 ceph-mon[25721]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' 2024-04-14T14:17:52.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:17:52 smithi122 ceph-mon[25721]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' 2024-04-14T14:17:52.846 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:17:52 smithi122 ceph-mon[25721]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' 2024-04-14T14:17:52.846 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:17:52 smithi122 ceph-mon[25721]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T14:17:52.846 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:17:52 smithi122 ceph-mon[25721]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-14T14:17:52.846 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:17:52 smithi122 ceph-mon[25721]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' 2024-04-14T14:17:53.067 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:17:52 smithi069 ceph-mon[24818]: from='client.25417 -' entity='client.iscsi.foo.smithi069.yuumtr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T14:17:53.067 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:17:52 smithi069 ceph-mon[24818]: pgmap v31: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:17:53.067 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:17:52 smithi069 ceph-mon[24818]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' 2024-04-14T14:17:53.067 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:17:52 smithi069 ceph-mon[24818]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' 2024-04-14T14:17:53.068 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:17:52 smithi069 ceph-mon[24818]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' 2024-04-14T14:17:53.068 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:17:52 smithi069 ceph-mon[24818]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' 2024-04-14T14:17:53.068 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:17:52 smithi069 ceph-mon[24818]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T14:17:53.068 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:17:52 smithi069 ceph-mon[24818]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-14T14:17:53.068 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:17:52 smithi069 ceph-mon[24818]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' 2024-04-14T14:17:53.068 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:17:52 smithi069 ceph-mon[29472]: from='client.25417 -' entity='client.iscsi.foo.smithi069.yuumtr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T14:17:53.068 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:17:52 smithi069 ceph-mon[29472]: pgmap v31: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:17:53.068 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:17:52 smithi069 ceph-mon[29472]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' 2024-04-14T14:17:53.068 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:17:52 smithi069 ceph-mon[29472]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' 2024-04-14T14:17:53.068 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:17:52 smithi069 ceph-mon[29472]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' 2024-04-14T14:17:53.068 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:17:52 smithi069 ceph-mon[29472]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' 2024-04-14T14:17:53.068 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:17:52 smithi069 ceph-mon[29472]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T14:17:53.069 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:17:52 smithi069 ceph-mon[29472]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-14T14:17:53.069 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:17:52 smithi069 ceph-mon[29472]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' 2024-04-14T14:17:54.067 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:17:53 smithi069 ceph-mon[24818]: pgmap v32: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 931 B/s rd, 0 op/s 2024-04-14T14:17:54.067 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:17:53 smithi069 ceph-mon[24818]: pgmap v33: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-04-14T14:17:54.067 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:17:53 smithi069 ceph-mon[24818]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-14T14:17:54.067 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:17:53 smithi069 ceph-mon[24818]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T14:17:54.067 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:17:53 smithi069 ceph-mon[24818]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T14:17:54.067 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:17:53 smithi069 ceph-mon[24818]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T14:17:54.067 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:17:53 smithi069 ceph-mon[24818]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T14:17:54.068 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:17:53 smithi069 ceph-mon[24818]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T14:17:54.068 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:17:53 smithi069 ceph-mon[24818]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T14:17:54.068 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:17:53 smithi069 ceph-mon[24818]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T14:17:54.068 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:17:53 smithi069 ceph-mon[24818]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T14:17:54.068 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:17:53 smithi069 ceph-mon[24818]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T14:17:54.068 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:17:53 smithi069 ceph-mon[29472]: pgmap v32: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 931 B/s rd, 0 op/s 2024-04-14T14:17:54.068 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:17:53 smithi069 ceph-mon[29472]: pgmap v33: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-04-14T14:17:54.068 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:17:53 smithi069 ceph-mon[29472]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-14T14:17:54.068 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:17:53 smithi069 ceph-mon[29472]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T14:17:54.068 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:17:53 smithi069 ceph-mon[29472]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T14:17:54.068 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:17:53 smithi069 ceph-mon[29472]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T14:17:54.068 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:17:53 smithi069 ceph-mon[29472]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T14:17:54.069 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:17:53 smithi069 ceph-mon[29472]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T14:17:54.069 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:17:53 smithi069 ceph-mon[29472]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T14:17:54.069 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:17:53 smithi069 ceph-mon[29472]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T14:17:54.069 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:17:53 smithi069 ceph-mon[29472]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T14:17:54.069 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:17:53 smithi069 ceph-mon[29472]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T14:17:54.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:17:53 smithi122 ceph-mon[25721]: pgmap v32: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 931 B/s rd, 0 op/s 2024-04-14T14:17:54.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:17:53 smithi122 ceph-mon[25721]: pgmap v33: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-04-14T14:17:54.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:17:53 smithi122 ceph-mon[25721]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-14T14:17:54.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:17:53 smithi122 ceph-mon[25721]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T14:17:54.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:17:53 smithi122 ceph-mon[25721]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T14:17:54.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:17:53 smithi122 ceph-mon[25721]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T14:17:54.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:17:53 smithi122 ceph-mon[25721]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T14:17:54.096 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:17:53 smithi122 ceph-mon[25721]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T14:17:54.096 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:17:53 smithi122 ceph-mon[25721]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T14:17:54.096 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:17:53 smithi122 ceph-mon[25721]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T14:17:54.096 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:17:53 smithi122 ceph-mon[25721]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T14:17:54.096 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:17:53 smithi122 ceph-mon[25721]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T14:17:56.067 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:17:55 smithi069 ceph-mon[24818]: Upgrade: Updating prometheus.a 2024-04-14T14:17:56.067 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:17:55 smithi069 ceph-mon[24818]: Deploying daemon prometheus.a on smithi122 2024-04-14T14:17:56.067 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:17:55 smithi069 ceph-mon[24818]: pgmap v34: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-04-14T14:17:56.068 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:17:55 smithi069 ceph-mon[29472]: Upgrade: Updating prometheus.a 2024-04-14T14:17:56.068 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:17:55 smithi069 ceph-mon[29472]: Deploying daemon prometheus.a on smithi122 2024-04-14T14:17:56.068 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:17:55 smithi069 ceph-mon[29472]: pgmap v34: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-04-14T14:17:56.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:17:55 smithi122 ceph-mon[25721]: Upgrade: Updating prometheus.a 2024-04-14T14:17:56.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:17:55 smithi122 ceph-mon[25721]: Deploying daemon prometheus.a on smithi122 2024-04-14T14:17:56.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:17:55 smithi122 ceph-mon[25721]: pgmap v34: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-04-14T14:17:56.817 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 14 14:17:56 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-y[63038]: ::ffff:172.21.15.122 - - [14/Apr/2024:14:17:56] "GET /metrics HTTP/1.1" 200 37988 "" "Prometheus/2.33.4" 2024-04-14T14:17:57.985 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:17:57 smithi069 ceph-mon[24818]: pgmap v35: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-04-14T14:17:57.986 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:17:57 smithi069 ceph-mon[29472]: pgmap v35: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-04-14T14:17:58.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:17:57 smithi122 ceph-mon[25721]: pgmap v35: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-04-14T14:17:58.095 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Apr 14 14:17:57 smithi122 systemd[1]: Stopping Ceph prometheus.a for dc326fc2-fa65-11ee-bc8e-c7b262605968... 2024-04-14T14:17:58.354 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Apr 14 14:17:58 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-prometheus-a[60892]: ts=2024-04-14T14:17:58.093Z caller=main.go:775 level=warn msg="Received SIGTERM, exiting gracefully..." 2024-04-14T14:17:58.354 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Apr 14 14:17:58 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-prometheus-a[60892]: ts=2024-04-14T14:17:58.093Z caller=main.go:798 level=info msg="Stopping scrape discovery manager..." 2024-04-14T14:17:58.355 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Apr 14 14:17:58 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-prometheus-a[60892]: ts=2024-04-14T14:17:58.093Z caller=main.go:812 level=info msg="Stopping notify discovery manager..." 2024-04-14T14:17:58.355 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Apr 14 14:17:58 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-prometheus-a[60892]: ts=2024-04-14T14:17:58.094Z caller=main.go:834 level=info msg="Stopping scrape manager..." 2024-04-14T14:17:58.355 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Apr 14 14:17:58 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-prometheus-a[60892]: ts=2024-04-14T14:17:58.093Z caller=main.go:794 level=info msg="Scrape discovery manager stopped" 2024-04-14T14:17:58.355 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Apr 14 14:17:58 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-prometheus-a[60892]: ts=2024-04-14T14:17:58.094Z caller=main.go:808 level=info msg="Notify discovery manager stopped" 2024-04-14T14:17:58.355 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Apr 14 14:17:58 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-prometheus-a[60892]: ts=2024-04-14T14:17:58.094Z caller=manager.go:945 level=info component="rule manager" msg="Stopping rule manager..." 2024-04-14T14:17:58.355 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Apr 14 14:17:58 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-prometheus-a[60892]: ts=2024-04-14T14:17:58.094Z caller=manager.go:955 level=info component="rule manager" msg="Rule manager stopped" 2024-04-14T14:17:58.355 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Apr 14 14:17:58 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-prometheus-a[60892]: ts=2024-04-14T14:17:58.094Z caller=main.go:828 level=info msg="Scrape manager stopped" 2024-04-14T14:17:58.355 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Apr 14 14:17:58 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-prometheus-a[60892]: ts=2024-04-14T14:17:58.094Z caller=notifier.go:600 level=info component=notifier msg="Stopping notification manager..." 2024-04-14T14:17:58.355 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Apr 14 14:17:58 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-prometheus-a[60892]: ts=2024-04-14T14:17:58.094Z caller=main.go:1054 level=info msg="Notifier manager stopped" 2024-04-14T14:17:58.355 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Apr 14 14:17:58 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-prometheus-a[60892]: ts=2024-04-14T14:17:58.094Z caller=main.go:1066 level=info msg="See you next time!" 2024-04-14T14:17:58.355 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Apr 14 14:17:58 smithi122 podman[64265]: 2024-04-14 14:17:58.103879898 +0000 UTC m=+0.102796830 container died c567cffcd74ece4c320fd2f1d544eb67aa8f5459fc8a90173c5cc945d6dfce28 (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-prometheus-a, maintainer=The Prometheus Authors ) 2024-04-14T14:17:58.637 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Apr 14 14:17:58 smithi122 podman[64265]: 2024-04-14 14:17:58.443335764 +0000 UTC m=+0.442252693 container cleanup c567cffcd74ece4c320fd2f1d544eb67aa8f5459fc8a90173c5cc945d6dfce28 (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-prometheus-a, maintainer=The Prometheus Authors ) 2024-04-14T14:17:58.637 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Apr 14 14:17:58 smithi122 bash[64265]: ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-prometheus-a 2024-04-14T14:17:59.053 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Apr 14 14:17:58 smithi122 podman[64279]: 2024-04-14 14:17:58.63627539 +0000 UTC m=+0.534924767 container remove c567cffcd74ece4c320fd2f1d544eb67aa8f5459fc8a90173c5cc945d6dfce28 (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-prometheus-a, maintainer=The Prometheus Authors ) 2024-04-14T14:17:59.053 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Apr 14 14:17:58 smithi122 systemd[1]: ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968@prometheus.a.service: Deactivated successfully. 2024-04-14T14:17:59.053 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Apr 14 14:17:58 smithi122 systemd[1]: Stopped Ceph prometheus.a for dc326fc2-fa65-11ee-bc8e-c7b262605968. 2024-04-14T14:17:59.053 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Apr 14 14:17:58 smithi122 systemd[1]: ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968@prometheus.a.service: Consumed 1.258s CPU time. 2024-04-14T14:17:59.345 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Apr 14 14:17:59 smithi122 systemd[1]: Starting Ceph prometheus.a for dc326fc2-fa65-11ee-bc8e-c7b262605968... 2024-04-14T14:17:59.345 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Apr 14 14:17:59 smithi122 podman[64408]: 2024-04-14 14:17:59.168159911 +0000 UTC m=+0.019086469 image pull a07b618ecd1dce142bce4c52f0e80982eaf1f14265a2415c2d35978ccaa0a464 quay.io/prometheus/prometheus:v2.43.0 2024-04-14T14:17:59.345 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Apr 14 14:17:59 smithi122 podman[64408]: 2024-04-14 14:17:59.279013939 +0000 UTC m=+0.129940479 container create 5c1f41d3bbe66fab3fa72f0ed86deea8cdd9704a9d1eb3aeca5a5d6ffcf194a5 (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-prometheus-a, maintainer=The Prometheus Authors ) 2024-04-14T14:17:59.691 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Apr 14 14:17:59 smithi122 podman[64408]: 2024-04-14 14:17:59.506160585 +0000 UTC m=+0.357087126 container init 5c1f41d3bbe66fab3fa72f0ed86deea8cdd9704a9d1eb3aeca5a5d6ffcf194a5 (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-prometheus-a, maintainer=The Prometheus Authors ) 2024-04-14T14:17:59.691 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Apr 14 14:17:59 smithi122 podman[64408]: 2024-04-14 14:17:59.508661325 +0000 UTC m=+0.359587867 container start 5c1f41d3bbe66fab3fa72f0ed86deea8cdd9704a9d1eb3aeca5a5d6ffcf194a5 (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-prometheus-a, maintainer=The Prometheus Authors ) 2024-04-14T14:17:59.691 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Apr 14 14:17:59 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-prometheus-a[64422]: ts=2024-04-14T14:17:59.532Z caller=main.go:564 level=info msg="Starting Prometheus Server" mode=server version="(version=2.43.0, branch=HEAD, revision=edfc3bcd025dd6fe296c167a14a216cab1e552ee)" 2024-04-14T14:17:59.691 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Apr 14 14:17:59 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-prometheus-a[64422]: ts=2024-04-14T14:17:59.532Z caller=main.go:569 level=info build_context="(go=go1.19.7, platform=linux/amd64, user=root@8a0ee342e522, date=20230321-12:56:07, tags=netgo,builtinassets)" 2024-04-14T14:17:59.691 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Apr 14 14:17:59 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-prometheus-a[64422]: ts=2024-04-14T14:17:59.532Z caller=main.go:570 level=info host_details="(Linux 5.14.0-435.el9.x86_64 #1 SMP PREEMPT_DYNAMIC Fri Mar 29 18:46:37 UTC 2024 x86_64 smithi122 (none))" 2024-04-14T14:17:59.691 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Apr 14 14:17:59 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-prometheus-a[64422]: ts=2024-04-14T14:17:59.532Z caller=main.go:571 level=info fd_limits="(soft=1048576, hard=1048576)" 2024-04-14T14:17:59.691 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Apr 14 14:17:59 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-prometheus-a[64422]: ts=2024-04-14T14:17:59.532Z caller=main.go:572 level=info vm_limits="(soft=unlimited, hard=unlimited)" 2024-04-14T14:17:59.691 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Apr 14 14:17:59 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-prometheus-a[64422]: ts=2024-04-14T14:17:59.533Z caller=web.go:561 level=info component=web msg="Start listening for connections" address=:9095 2024-04-14T14:17:59.692 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Apr 14 14:17:59 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-prometheus-a[64422]: ts=2024-04-14T14:17:59.534Z caller=main.go:1005 level=info msg="Starting TSDB ..." 2024-04-14T14:17:59.692 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Apr 14 14:17:59 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-prometheus-a[64422]: ts=2024-04-14T14:17:59.535Z caller=tls_config.go:232 level=info component=web msg="Listening on" address=[::]:9095 2024-04-14T14:17:59.692 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Apr 14 14:17:59 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-prometheus-a[64422]: ts=2024-04-14T14:17:59.536Z caller=tls_config.go:235 level=info component=web msg="TLS is disabled." http2=false address=[::]:9095 2024-04-14T14:17:59.692 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Apr 14 14:17:59 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-prometheus-a[64422]: ts=2024-04-14T14:17:59.537Z caller=head.go:587 level=info component=tsdb msg="Replaying on-disk memory mappable chunks if any" 2024-04-14T14:17:59.692 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Apr 14 14:17:59 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-prometheus-a[64422]: ts=2024-04-14T14:17:59.538Z caller=head.go:658 level=info component=tsdb msg="On-disk memory mappable chunks replay completed" duration=1.317343ms 2024-04-14T14:17:59.692 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Apr 14 14:17:59 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-prometheus-a[64422]: ts=2024-04-14T14:17:59.538Z caller=head.go:664 level=info component=tsdb msg="Replaying WAL, this may take a while" 2024-04-14T14:17:59.692 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Apr 14 14:17:59 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-prometheus-a[64422]: ts=2024-04-14T14:17:59.544Z caller=head.go:735 level=info component=tsdb msg="WAL segment loaded" segment=0 maxSegment=6 2024-04-14T14:17:59.692 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Apr 14 14:17:59 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-prometheus-a[64422]: ts=2024-04-14T14:17:59.558Z caller=head.go:735 level=info component=tsdb msg="WAL segment loaded" segment=1 maxSegment=6 2024-04-14T14:17:59.692 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Apr 14 14:17:59 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-prometheus-a[64422]: ts=2024-04-14T14:17:59.565Z caller=head.go:735 level=info component=tsdb msg="WAL segment loaded" segment=2 maxSegment=6 2024-04-14T14:17:59.692 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Apr 14 14:17:59 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-prometheus-a[64422]: ts=2024-04-14T14:17:59.571Z caller=head.go:735 level=info component=tsdb msg="WAL segment loaded" segment=3 maxSegment=6 2024-04-14T14:17:59.692 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Apr 14 14:17:59 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-prometheus-a[64422]: ts=2024-04-14T14:17:59.574Z caller=head.go:735 level=info component=tsdb msg="WAL segment loaded" segment=4 maxSegment=6 2024-04-14T14:17:59.692 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Apr 14 14:17:59 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-prometheus-a[64422]: ts=2024-04-14T14:17:59.575Z caller=head.go:735 level=info component=tsdb msg="WAL segment loaded" segment=5 maxSegment=6 2024-04-14T14:17:59.693 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Apr 14 14:17:59 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-prometheus-a[64422]: ts=2024-04-14T14:17:59.575Z caller=head.go:735 level=info component=tsdb msg="WAL segment loaded" segment=6 maxSegment=6 2024-04-14T14:17:59.693 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Apr 14 14:17:59 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-prometheus-a[64422]: ts=2024-04-14T14:17:59.575Z caller=head.go:772 level=info component=tsdb msg="WAL replay completed" checkpoint_replay_duration=61.406µs wal_replay_duration=36.649279ms wbl_replay_duration=96ns total_replay_duration=38.054126ms 2024-04-14T14:17:59.693 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Apr 14 14:17:59 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-prometheus-a[64422]: ts=2024-04-14T14:17:59.578Z caller=main.go:1026 level=info fs_type=XFS_SUPER_MAGIC 2024-04-14T14:17:59.693 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Apr 14 14:17:59 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-prometheus-a[64422]: ts=2024-04-14T14:17:59.578Z caller=main.go:1029 level=info msg="TSDB started" 2024-04-14T14:17:59.693 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Apr 14 14:17:59 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-prometheus-a[64422]: ts=2024-04-14T14:17:59.578Z caller=main.go:1209 level=info msg="Loading configuration file" filename=/etc/prometheus/prometheus.yml 2024-04-14T14:17:59.693 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Apr 14 14:17:59 smithi122 bash[64408]: 5c1f41d3bbe66fab3fa72f0ed86deea8cdd9704a9d1eb3aeca5a5d6ffcf194a5 2024-04-14T14:17:59.693 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Apr 14 14:17:59 smithi122 systemd[1]: Started Ceph prometheus.a for dc326fc2-fa65-11ee-bc8e-c7b262605968. 2024-04-14T14:17:59.693 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Apr 14 14:17:59 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-prometheus-a[64422]: ts=2024-04-14T14:17:59.598Z caller=main.go:1246 level=info msg="Completed loading of configuration file" filename=/etc/prometheus/prometheus.yml totalDuration=19.827728ms db_storage=844ns remote_storage=1.017µs web_handler=375ns query_engine=689ns scrape=450.634µs scrape_sd=85.668µs notify=10.918µs notify_sd=27.63µs rules=18.92555ms tracing=5.622µs 2024-04-14T14:17:59.693 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Apr 14 14:17:59 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-prometheus-a[64422]: ts=2024-04-14T14:17:59.598Z caller=main.go:990 level=info msg="Server is ready to receive web requests." 2024-04-14T14:17:59.693 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Apr 14 14:17:59 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-prometheus-a[64422]: ts=2024-04-14T14:17:59.598Z caller=manager.go:974 level=info component="rule manager" msg="Starting rule manager..." 2024-04-14T14:18:00.067 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:17:59 smithi069 ceph-mon[24818]: pgmap v36: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-04-14T14:18:00.067 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:17:59 smithi069 ceph-mon[24818]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' 2024-04-14T14:18:00.067 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:17:59 smithi069 ceph-mon[24818]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' 2024-04-14T14:18:00.067 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:17:59 smithi069 ceph-mon[29472]: pgmap v36: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-04-14T14:18:00.067 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:17:59 smithi069 ceph-mon[29472]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' 2024-04-14T14:18:00.067 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:17:59 smithi069 ceph-mon[29472]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' 2024-04-14T14:18:00.094 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:17:59 smithi122 ceph-mon[25721]: pgmap v36: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-04-14T14:18:00.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:17:59 smithi122 ceph-mon[25721]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' 2024-04-14T14:18:00.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:17:59 smithi122 ceph-mon[25721]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' 2024-04-14T14:18:00.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:18:00 smithi122 ceph-mon[25721]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T14:18:01.067 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:18:00 smithi069 ceph-mon[24818]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T14:18:01.067 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:18:00 smithi069 ceph-mon[29472]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T14:18:02.023 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:18:01 smithi122 ceph-mon[25721]: pgmap v37: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 569 B/s rd, 0 op/s 2024-04-14T14:18:02.067 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:18:01 smithi069 ceph-mon[24818]: pgmap v37: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 569 B/s rd, 0 op/s 2024-04-14T14:18:02.067 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:18:01 smithi069 ceph-mon[29472]: pgmap v37: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 569 B/s rd, 0 op/s 2024-04-14T14:18:03.067 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:18:02 smithi069 ceph-mon[24818]: from='client.25417 -' entity='client.iscsi.foo.smithi069.yuumtr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T14:18:03.067 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:18:02 smithi069 ceph-mon[29472]: from='client.25417 -' entity='client.iscsi.foo.smithi069.yuumtr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T14:18:03.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:18:02 smithi122 ceph-mon[25721]: from='client.25417 -' entity='client.iscsi.foo.smithi069.yuumtr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T14:18:04.067 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:18:03 smithi069 ceph-mon[24818]: pgmap v38: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-14T14:18:04.067 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:18:03 smithi069 ceph-mon[29472]: pgmap v38: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-14T14:18:04.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:18:03 smithi122 ceph-mon[25721]: pgmap v38: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-14T14:18:04.587 INFO:teuthology.orchestra.run.smithi069.stdout:true 2024-04-14T14:18:05.067 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:18:04 smithi069 ceph-mon[24818]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' 2024-04-14T14:18:05.067 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:18:04 smithi069 ceph-mon[24818]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' 2024-04-14T14:18:05.067 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:18:04 smithi069 ceph-mon[24818]: from='client.15564 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T14:18:05.067 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:18:04 smithi069 ceph-mon[24818]: pgmap v39: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:18:05.067 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:18:04 smithi069 ceph-mon[29472]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' 2024-04-14T14:18:05.067 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:18:04 smithi069 ceph-mon[29472]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' 2024-04-14T14:18:05.067 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:18:04 smithi069 ceph-mon[29472]: from='client.15564 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T14:18:05.067 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:18:04 smithi069 ceph-mon[29472]: pgmap v39: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:18:05.094 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:18:04 smithi122 ceph-mon[25721]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' 2024-04-14T14:18:05.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:18:04 smithi122 ceph-mon[25721]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' 2024-04-14T14:18:05.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:18:04 smithi122 ceph-mon[25721]: from='client.15564 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T14:18:05.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:18:04 smithi122 ceph-mon[25721]: pgmap v39: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:18:05.254 INFO:teuthology.orchestra.run.smithi069.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-14T14:18:05.255 INFO:teuthology.orchestra.run.smithi069.stdout:alertmanager.a smithi069 *:9093,9094 running (13m) 17s ago 21m 22.7M - 0.23.0 ba2b418f427c e300fc36e997 2024-04-14T14:18:05.255 INFO:teuthology.orchestra.run.smithi069.stdout:grafana.a smithi122 *:3000 error 1s ago 21m - - 2024-04-14T14:18:05.255 INFO:teuthology.orchestra.run.smithi069.stdout:iscsi.foo.smithi069.yuumtr smithi069 running (54s) 17s ago 20m 44.3M - 3.5 e1d6a67b021e 38cc81ea8cfa 2024-04-14T14:18:05.255 INFO:teuthology.orchestra.run.smithi069.stdout:mgr.x smithi122 *:8443,9283,8765 running (44s) 1s ago 25m 449M - 19.0.0-2778-ga971fd5f b96c509eef84 e29977590b43 2024-04-14T14:18:05.255 INFO:teuthology.orchestra.run.smithi069.stdout:mgr.y smithi069 *:8443,9283,8765 running (10m) 17s ago 27m 500M - 19.0.0-2778-ga971fd5f b96c509eef84 7af3d29d0e5f 2024-04-14T14:18:05.255 INFO:teuthology.orchestra.run.smithi069.stdout:mon.a smithi069 running (27m) 17s ago 27m 77.7M 2048M 17.2.0 e1d6a67b021e d3c553f91913 2024-04-14T14:18:05.255 INFO:teuthology.orchestra.run.smithi069.stdout:mon.b smithi122 running (26m) 1s ago 26m 58.2M 2048M 17.2.0 e1d6a67b021e 3b766bfce4be 2024-04-14T14:18:05.255 INFO:teuthology.orchestra.run.smithi069.stdout:mon.c smithi069 running (26m) 17s ago 26m 57.3M 2048M 17.2.0 e1d6a67b021e 23c078b9d1e5 2024-04-14T14:18:05.255 INFO:teuthology.orchestra.run.smithi069.stdout:node-exporter.a smithi069 *:9100 running (29s) 17s ago 22m 7982k - 1.5.0 0da6a335fe13 0de0eb2171b9 2024-04-14T14:18:05.255 INFO:teuthology.orchestra.run.smithi069.stdout:node-exporter.b smithi122 *:9100 running (22s) 1s ago 22m 7856k - 1.5.0 0da6a335fe13 67877bf02d8e 2024-04-14T14:18:05.255 INFO:teuthology.orchestra.run.smithi069.stdout:osd.0 smithi069 running (25m) 17s ago 25m 56.5M 2992M 17.2.0 e1d6a67b021e 46654654b6b6 2024-04-14T14:18:05.255 INFO:teuthology.orchestra.run.smithi069.stdout:osd.1 smithi069 running (25m) 17s ago 25m 54.5M 2992M 17.2.0 e1d6a67b021e cdba54a59609 2024-04-14T14:18:05.255 INFO:teuthology.orchestra.run.smithi069.stdout:osd.2 smithi069 running (24m) 17s ago 24m 50.9M 2992M 17.2.0 e1d6a67b021e bc37a1a9cba0 2024-04-14T14:18:05.255 INFO:teuthology.orchestra.run.smithi069.stdout:osd.3 smithi069 running (24m) 17s ago 24m 55.2M 2992M 17.2.0 e1d6a67b021e 0925ba3da801 2024-04-14T14:18:05.256 INFO:teuthology.orchestra.run.smithi069.stdout:osd.4 smithi122 running (24m) 1s ago 24m 56.0M 3248M 17.2.0 e1d6a67b021e c00975caf771 2024-04-14T14:18:05.256 INFO:teuthology.orchestra.run.smithi069.stdout:osd.5 smithi122 running (23m) 1s ago 23m 55.1M 3248M 17.2.0 e1d6a67b021e 5f52d217ceb8 2024-04-14T14:18:05.256 INFO:teuthology.orchestra.run.smithi069.stdout:osd.6 smithi122 running (23m) 1s ago 23m 51.7M 3248M 17.2.0 e1d6a67b021e a2acfec61475 2024-04-14T14:18:05.256 INFO:teuthology.orchestra.run.smithi069.stdout:osd.7 smithi122 running (23m) 1s ago 23m 56.8M 3248M 17.2.0 e1d6a67b021e 62411cd06801 2024-04-14T14:18:05.256 INFO:teuthology.orchestra.run.smithi069.stdout:prometheus.a smithi122 *:9095 running (6s) 1s ago 22m 49.7M - 2.43.0 a07b618ecd1d 5c1f41d3bbe6 2024-04-14T14:18:05.256 INFO:teuthology.orchestra.run.smithi069.stdout:rgw.foo.smithi069.wesedz smithi069 *:8000 running (20m) 17s ago 20m 77.7M - 17.2.0 e1d6a67b021e 16de491440f3 2024-04-14T14:18:05.256 INFO:teuthology.orchestra.run.smithi069.stdout:rgw.foo.smithi122.jiobyv smithi122 *:8000 running (20m) 1s ago 20m 74.7M - 17.2.0 e1d6a67b021e 6d16de445dba 2024-04-14T14:18:05.654 INFO:teuthology.orchestra.run.smithi069.stdout:{ 2024-04-14T14:18:05.654 INFO:teuthology.orchestra.run.smithi069.stdout: "mon": { 2024-04-14T14:18:05.654 INFO:teuthology.orchestra.run.smithi069.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 3 2024-04-14T14:18:05.654 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-14T14:18:05.654 INFO:teuthology.orchestra.run.smithi069.stdout: "mgr": { 2024-04-14T14:18:05.654 INFO:teuthology.orchestra.run.smithi069.stdout: "ceph version 19.0.0-2778-ga971fd5f (a971fd5f5719038c70b817bafacd6d5400b1d888) squid (dev)": 2 2024-04-14T14:18:05.654 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-14T14:18:05.654 INFO:teuthology.orchestra.run.smithi069.stdout: "osd": { 2024-04-14T14:18:05.654 INFO:teuthology.orchestra.run.smithi069.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 8 2024-04-14T14:18:05.655 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-14T14:18:05.655 INFO:teuthology.orchestra.run.smithi069.stdout: "mds": {}, 2024-04-14T14:18:05.655 INFO:teuthology.orchestra.run.smithi069.stdout: "rgw": { 2024-04-14T14:18:05.655 INFO:teuthology.orchestra.run.smithi069.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-04-14T14:18:05.655 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-14T14:18:05.655 INFO:teuthology.orchestra.run.smithi069.stdout: "overall": { 2024-04-14T14:18:05.655 INFO:teuthology.orchestra.run.smithi069.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 13, 2024-04-14T14:18:05.655 INFO:teuthology.orchestra.run.smithi069.stdout: "ceph version 19.0.0-2778-ga971fd5f (a971fd5f5719038c70b817bafacd6d5400b1d888) squid (dev)": 2 2024-04-14T14:18:05.655 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-14T14:18:05.655 INFO:teuthology.orchestra.run.smithi069.stdout:} 2024-04-14T14:18:05.991 INFO:teuthology.orchestra.run.smithi069.stdout:{ 2024-04-14T14:18:05.991 INFO:teuthology.orchestra.run.smithi069.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888", 2024-04-14T14:18:05.991 INFO:teuthology.orchestra.run.smithi069.stdout: "in_progress": true, 2024-04-14T14:18:05.991 INFO:teuthology.orchestra.run.smithi069.stdout: "which": "Upgrading daemons of type(s) mgr", 2024-04-14T14:18:05.991 INFO:teuthology.orchestra.run.smithi069.stdout: "services_complete": [ 2024-04-14T14:18:05.991 INFO:teuthology.orchestra.run.smithi069.stdout: "mgr" 2024-04-14T14:18:05.991 INFO:teuthology.orchestra.run.smithi069.stdout: ], 2024-04-14T14:18:05.991 INFO:teuthology.orchestra.run.smithi069.stdout: "progress": "2/2 daemons upgraded", 2024-04-14T14:18:05.991 INFO:teuthology.orchestra.run.smithi069.stdout: "message": "Currently upgrading prometheus daemons", 2024-04-14T14:18:05.991 INFO:teuthology.orchestra.run.smithi069.stdout: "is_paused": false 2024-04-14T14:18:05.992 INFO:teuthology.orchestra.run.smithi069.stdout:} 2024-04-14T14:18:05.992 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:18:05 smithi069 ceph-mon[24818]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' 2024-04-14T14:18:05.992 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:18:05 smithi069 ceph-mon[24818]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' 2024-04-14T14:18:05.992 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:18:05 smithi069 ceph-mon[24818]: from='client.15570 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T14:18:05.992 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:18:05 smithi069 ceph-mon[24818]: from='client.25477 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T14:18:05.992 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:18:05 smithi069 ceph-mon[24818]: from='client.? 172.21.15.69:0/1130878701' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T14:18:05.993 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:18:05 smithi069 ceph-mon[29472]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' 2024-04-14T14:18:05.993 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:18:05 smithi069 ceph-mon[29472]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' 2024-04-14T14:18:05.993 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:18:05 smithi069 ceph-mon[29472]: from='client.15570 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T14:18:05.993 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:18:05 smithi069 ceph-mon[29472]: from='client.25477 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T14:18:05.993 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:18:05 smithi069 ceph-mon[29472]: from='client.? 172.21.15.69:0/1130878701' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T14:18:06.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:18:05 smithi122 ceph-mon[25721]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' 2024-04-14T14:18:06.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:18:05 smithi122 ceph-mon[25721]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' 2024-04-14T14:18:06.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:18:05 smithi122 ceph-mon[25721]: from='client.15570 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T14:18:06.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:18:05 smithi122 ceph-mon[25721]: from='client.25477 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T14:18:06.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:18:05 smithi122 ceph-mon[25721]: from='client.? 172.21.15.69:0/1130878701' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T14:18:06.817 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 14 14:18:06 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-y[63038]: ::ffff:172.21.15.122 - - [14/Apr/2024:14:18:06] "GET /metrics HTTP/1.1" 200 37987 "" "Prometheus/2.43.0" 2024-04-14T14:18:07.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:18:06 smithi122 ceph-mon[25721]: from='client.25489 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T14:18:07.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:18:06 smithi122 ceph-mon[25721]: pgmap v40: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:18:07.317 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:18:06 smithi069 ceph-mon[24818]: from='client.25489 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T14:18:07.317 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:18:06 smithi069 ceph-mon[24818]: pgmap v40: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:18:07.317 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:18:06 smithi069 ceph-mon[29472]: from='client.25489 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T14:18:07.317 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:18:06 smithi069 ceph-mon[29472]: pgmap v40: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:18:09.567 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:18:09 smithi069 ceph-mon[24818]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' 2024-04-14T14:18:09.567 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:18:09 smithi069 ceph-mon[24818]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' 2024-04-14T14:18:09.568 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:18:09 smithi069 ceph-mon[24818]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T14:18:09.568 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:18:09 smithi069 ceph-mon[24818]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-14T14:18:09.568 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:18:09 smithi069 ceph-mon[24818]: pgmap v41: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 880 B/s rd, 0 op/s 2024-04-14T14:18:09.568 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:18:09 smithi069 ceph-mon[24818]: pgmap v42: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-04-14T14:18:09.568 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:18:09 smithi069 ceph-mon[24818]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' 2024-04-14T14:18:09.568 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:18:09 smithi069 ceph-mon[24818]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-04-14T14:18:09.568 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:18:09 smithi069 ceph-mon[24818]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-04-14T14:18:09.568 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:18:09 smithi069 ceph-mon[24818]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-14T14:18:09.568 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:18:09 smithi069 ceph-mon[24818]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T14:18:09.568 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:18:09 smithi069 ceph-mon[24818]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T14:18:09.568 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:18:09 smithi069 ceph-mon[24818]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T14:18:09.568 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:18:09 smithi069 ceph-mon[24818]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T14:18:09.568 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:18:09 smithi069 ceph-mon[24818]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T14:18:09.568 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:18:09 smithi069 ceph-mon[24818]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T14:18:09.569 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:18:09 smithi069 ceph-mon[24818]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T14:18:09.569 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:18:09 smithi069 ceph-mon[24818]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T14:18:09.569 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:18:09 smithi069 ceph-mon[24818]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T14:18:09.569 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:18:09 smithi069 ceph-mon[24818]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T14:18:09.569 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:18:09 smithi069 ceph-mon[29472]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' 2024-04-14T14:18:09.569 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:18:09 smithi069 ceph-mon[29472]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' 2024-04-14T14:18:09.569 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:18:09 smithi069 ceph-mon[29472]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T14:18:09.569 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:18:09 smithi069 ceph-mon[29472]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-14T14:18:09.569 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:18:09 smithi069 ceph-mon[29472]: pgmap v41: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 880 B/s rd, 0 op/s 2024-04-14T14:18:09.569 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:18:09 smithi069 ceph-mon[29472]: pgmap v42: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-04-14T14:18:09.569 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:18:09 smithi069 ceph-mon[29472]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' 2024-04-14T14:18:09.569 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:18:09 smithi069 ceph-mon[29472]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-04-14T14:18:09.570 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:18:09 smithi069 ceph-mon[29472]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-04-14T14:18:09.570 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:18:09 smithi069 ceph-mon[29472]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-14T14:18:09.570 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:18:09 smithi069 ceph-mon[29472]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T14:18:09.570 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:18:09 smithi069 ceph-mon[29472]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T14:18:09.570 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:18:09 smithi069 ceph-mon[29472]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T14:18:09.570 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:18:09 smithi069 ceph-mon[29472]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T14:18:09.570 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:18:09 smithi069 ceph-mon[29472]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T14:18:09.570 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:18:09 smithi069 ceph-mon[29472]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T14:18:09.570 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:18:09 smithi069 ceph-mon[29472]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T14:18:09.570 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:18:09 smithi069 ceph-mon[29472]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T14:18:09.570 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:18:09 smithi069 ceph-mon[29472]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T14:18:09.570 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:18:09 smithi069 ceph-mon[29472]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T14:18:09.595 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:18:09 smithi122 ceph-mon[25721]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' 2024-04-14T14:18:09.595 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:18:09 smithi122 ceph-mon[25721]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' 2024-04-14T14:18:09.595 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:18:09 smithi122 ceph-mon[25721]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T14:18:09.595 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:18:09 smithi122 ceph-mon[25721]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-14T14:18:09.596 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:18:09 smithi122 ceph-mon[25721]: pgmap v41: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 880 B/s rd, 0 op/s 2024-04-14T14:18:09.596 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:18:09 smithi122 ceph-mon[25721]: pgmap v42: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-04-14T14:18:09.596 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:18:09 smithi122 ceph-mon[25721]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' 2024-04-14T14:18:09.596 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:18:09 smithi122 ceph-mon[25721]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-04-14T14:18:09.596 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:18:09 smithi122 ceph-mon[25721]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-04-14T14:18:09.596 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:18:09 smithi122 ceph-mon[25721]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-14T14:18:09.596 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:18:09 smithi122 ceph-mon[25721]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T14:18:09.596 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:18:09 smithi122 ceph-mon[25721]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T14:18:09.596 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:18:09 smithi122 ceph-mon[25721]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T14:18:09.596 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:18:09 smithi122 ceph-mon[25721]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T14:18:09.596 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:18:09 smithi122 ceph-mon[25721]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T14:18:09.596 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:18:09 smithi122 ceph-mon[25721]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T14:18:09.596 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:18:09 smithi122 ceph-mon[25721]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T14:18:09.596 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:18:09 smithi122 ceph-mon[25721]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T14:18:09.596 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:18:09 smithi122 ceph-mon[25721]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T14:18:09.597 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:18:09 smithi122 ceph-mon[25721]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T14:18:11.837 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:18:11 smithi069 ceph-mon[24818]: Upgrade: Updating alertmanager.a 2024-04-14T14:18:11.837 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:18:11 smithi069 ceph-mon[24818]: Deploying daemon alertmanager.a on smithi069 2024-04-14T14:18:11.837 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:18:11 smithi069 ceph-mon[24818]: pgmap v43: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-04-14T14:18:11.837 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:18:11 smithi069 ceph-mon[29472]: Upgrade: Updating alertmanager.a 2024-04-14T14:18:11.838 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:18:11 smithi069 ceph-mon[29472]: Deploying daemon alertmanager.a on smithi069 2024-04-14T14:18:11.838 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:18:11 smithi069 ceph-mon[29472]: pgmap v43: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-04-14T14:18:11.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:18:11 smithi122 ceph-mon[25721]: Upgrade: Updating alertmanager.a 2024-04-14T14:18:11.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:18:11 smithi122 ceph-mon[25721]: Deploying daemon alertmanager.a on smithi069 2024-04-14T14:18:11.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:18:11 smithi122 ceph-mon[25721]: pgmap v43: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-04-14T14:18:12.817 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:18:12 smithi069 ceph-mon[24818]: from='client.25417 -' entity='client.iscsi.foo.smithi069.yuumtr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T14:18:12.817 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:18:12 smithi069 ceph-mon[29472]: from='client.25417 -' entity='client.iscsi.foo.smithi069.yuumtr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T14:18:12.844 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:18:12 smithi122 ceph-mon[25721]: from='client.25417 -' entity='client.iscsi.foo.smithi069.yuumtr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T14:18:13.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:18:13 smithi122 ceph-mon[25721]: pgmap v44: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-04-14T14:18:13.874 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:18:13 smithi069 ceph-mon[24818]: pgmap v44: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-04-14T14:18:13.875 INFO:journalctl@ceph.alertmanager.a.smithi069.stdout:Apr 14 14:18:13 smithi069 systemd[1]: Stopping Ceph alertmanager.a for dc326fc2-fa65-11ee-bc8e-c7b262605968... 2024-04-14T14:18:13.875 INFO:journalctl@ceph.alertmanager.a.smithi069.stdout:Apr 14 14:18:13 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-alertmanager-a[58923]: level=info ts=2024-04-14T14:18:13.832Z caller=main.go:557 msg="Received SIGTERM, exiting gracefully..." 2024-04-14T14:18:13.875 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:18:13 smithi069 ceph-mon[29472]: pgmap v44: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-04-14T14:18:14.167 INFO:journalctl@ceph.alertmanager.a.smithi069.stdout:Apr 14 14:18:13 smithi069 podman[78758]: 2024-04-14 14:18:13.873214057 +0000 UTC m=+0.138064739 container died e300fc36e997f965f8a8646eeb7d84bec0d051e4a005771ba12d0a7572eea343 (image=quay.io/prometheus/alertmanager:v0.23.0, name=ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-alertmanager-a, maintainer=The Prometheus Authors ) 2024-04-14T14:18:14.511 INFO:journalctl@ceph.alertmanager.a.smithi069.stdout:Apr 14 14:18:14 smithi069 podman[78758]: 2024-04-14 14:18:14.257432879 +0000 UTC m=+0.522283602 container cleanup e300fc36e997f965f8a8646eeb7d84bec0d051e4a005771ba12d0a7572eea343 (image=quay.io/prometheus/alertmanager:v0.23.0, name=ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-alertmanager-a, maintainer=The Prometheus Authors ) 2024-04-14T14:18:14.511 INFO:journalctl@ceph.alertmanager.a.smithi069.stdout:Apr 14 14:18:14 smithi069 bash[78758]: ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-alertmanager-a 2024-04-14T14:18:14.817 INFO:journalctl@ceph.alertmanager.a.smithi069.stdout:Apr 14 14:18:14 smithi069 podman[78772]: 2024-04-14 14:18:14.510634625 +0000 UTC m=+0.632930777 container remove e300fc36e997f965f8a8646eeb7d84bec0d051e4a005771ba12d0a7572eea343 (image=quay.io/prometheus/alertmanager:v0.23.0, name=ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-alertmanager-a, maintainer=The Prometheus Authors ) 2024-04-14T14:18:14.817 INFO:journalctl@ceph.alertmanager.a.smithi069.stdout:Apr 14 14:18:14 smithi069 podman[78772]: 2024-04-14 14:18:14.594499912 +0000 UTC m=+0.716796065 volume remove 4191fea74c2099335b5c03e4f08bfdefab90e7651f56bed2ceed4994d25dd4d1 2024-04-14T14:18:14.817 INFO:journalctl@ceph.alertmanager.a.smithi069.stdout:Apr 14 14:18:14 smithi069 systemd[1]: ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968@alertmanager.a.service: Deactivated successfully. 2024-04-14T14:18:14.817 INFO:journalctl@ceph.alertmanager.a.smithi069.stdout:Apr 14 14:18:14 smithi069 systemd[1]: Stopped Ceph alertmanager.a for dc326fc2-fa65-11ee-bc8e-c7b262605968. 2024-04-14T14:18:14.817 INFO:journalctl@ceph.alertmanager.a.smithi069.stdout:Apr 14 14:18:14 smithi069 systemd[1]: ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968@alertmanager.a.service: Consumed 1.557s CPU time. 2024-04-14T14:18:15.230 INFO:journalctl@ceph.alertmanager.a.smithi069.stdout:Apr 14 14:18:14 smithi069 systemd[1]: Starting Ceph alertmanager.a for dc326fc2-fa65-11ee-bc8e-c7b262605968... 2024-04-14T14:18:15.231 INFO:journalctl@ceph.alertmanager.a.smithi069.stdout:Apr 14 14:18:15 smithi069 podman[78903]: 2024-04-14 14:18:15.137759432 +0000 UTC m=+0.107036746 volume create e1e5bcaf6468175db501fffa778d8fcc073ffaf5189911d804ed10b9c66c7466 2024-04-14T14:18:15.231 INFO:journalctl@ceph.alertmanager.a.smithi069.stdout:Apr 14 14:18:15 smithi069 podman[78903]: 2024-04-14 14:18:15.049020339 +0000 UTC m=+0.018297614 image pull c8568f914cd25b2062c44e9f79f9c18da6e3b85fe0c47a12a2191c61426c2b19 quay.io/prometheus/alertmanager:v0.25.0 2024-04-14T14:18:15.231 INFO:journalctl@ceph.alertmanager.a.smithi069.stdout:Apr 14 14:18:15 smithi069 podman[78903]: 2024-04-14 14:18:15.229627977 +0000 UTC m=+0.198905248 container create 7524f106e7bd83a750de14b124ac87ed19c0457229c5bce7cd7ef398ce9dda93 (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-alertmanager-a, maintainer=The Prometheus Authors ) 2024-04-14T14:18:16.067 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:18:15 smithi069 ceph-mon[24818]: pgmap v45: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-04-14T14:18:16.067 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:18:15 smithi069 ceph-mon[24818]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T14:18:16.067 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:18:15 smithi069 ceph-mon[29472]: pgmap v45: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-04-14T14:18:16.067 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:18:15 smithi069 ceph-mon[29472]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T14:18:16.067 INFO:journalctl@ceph.alertmanager.a.smithi069.stdout:Apr 14 14:18:15 smithi069 podman[78903]: 2024-04-14 14:18:15.565861257 +0000 UTC m=+0.535138529 container init 7524f106e7bd83a750de14b124ac87ed19c0457229c5bce7cd7ef398ce9dda93 (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-alertmanager-a, maintainer=The Prometheus Authors ) 2024-04-14T14:18:16.068 INFO:journalctl@ceph.alertmanager.a.smithi069.stdout:Apr 14 14:18:15 smithi069 podman[78903]: 2024-04-14 14:18:15.568797071 +0000 UTC m=+0.538074344 container start 7524f106e7bd83a750de14b124ac87ed19c0457229c5bce7cd7ef398ce9dda93 (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-alertmanager-a, maintainer=The Prometheus Authors ) 2024-04-14T14:18:16.068 INFO:journalctl@ceph.alertmanager.a.smithi069.stdout:Apr 14 14:18:15 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-alertmanager-a[78917]: ts=2024-04-14T14:18:15.579Z caller=main.go:240 level=info msg="Starting Alertmanager" version="(version=0.25.0, branch=HEAD, revision=258fab7cdd551f2cf251ed0348f0ad7289aee789)" 2024-04-14T14:18:16.068 INFO:journalctl@ceph.alertmanager.a.smithi069.stdout:Apr 14 14:18:15 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-alertmanager-a[78917]: ts=2024-04-14T14:18:15.579Z caller=main.go:241 level=info build_context="(go=go1.19.4, user=root@abe866dd5717, date=20221222-14:51:36)" 2024-04-14T14:18:16.068 INFO:journalctl@ceph.alertmanager.a.smithi069.stdout:Apr 14 14:18:15 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-alertmanager-a[78917]: ts=2024-04-14T14:18:15.581Z caller=cluster.go:185 level=info component=cluster msg="setting advertise address explicitly" addr=172.21.15.69 port=9094 2024-04-14T14:18:16.068 INFO:journalctl@ceph.alertmanager.a.smithi069.stdout:Apr 14 14:18:15 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-alertmanager-a[78917]: ts=2024-04-14T14:18:15.582Z caller=cluster.go:681 level=info component=cluster msg="Waiting for gossip to settle..." interval=2s 2024-04-14T14:18:16.068 INFO:journalctl@ceph.alertmanager.a.smithi069.stdout:Apr 14 14:18:15 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-alertmanager-a[78917]: ts=2024-04-14T14:18:15.606Z caller=coordinator.go:113 level=info component=configuration msg="Loading configuration file" file=/etc/alertmanager/alertmanager.yml 2024-04-14T14:18:16.068 INFO:journalctl@ceph.alertmanager.a.smithi069.stdout:Apr 14 14:18:15 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-alertmanager-a[78917]: ts=2024-04-14T14:18:15.607Z caller=coordinator.go:126 level=info component=configuration msg="Completed loading of configuration file" file=/etc/alertmanager/alertmanager.yml 2024-04-14T14:18:16.068 INFO:journalctl@ceph.alertmanager.a.smithi069.stdout:Apr 14 14:18:15 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-alertmanager-a[78917]: ts=2024-04-14T14:18:15.609Z caller=tls_config.go:232 level=info msg="Listening on" address=[::]:9093 2024-04-14T14:18:16.068 INFO:journalctl@ceph.alertmanager.a.smithi069.stdout:Apr 14 14:18:15 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-alertmanager-a[78917]: ts=2024-04-14T14:18:15.610Z caller=tls_config.go:235 level=info msg="TLS is disabled." http2=false address=[::]:9093 2024-04-14T14:18:16.068 INFO:journalctl@ceph.alertmanager.a.smithi069.stdout:Apr 14 14:18:15 smithi069 bash[78903]: 7524f106e7bd83a750de14b124ac87ed19c0457229c5bce7cd7ef398ce9dda93 2024-04-14T14:18:16.068 INFO:journalctl@ceph.alertmanager.a.smithi069.stdout:Apr 14 14:18:15 smithi069 systemd[1]: Started Ceph alertmanager.a for dc326fc2-fa65-11ee-bc8e-c7b262605968. 2024-04-14T14:18:16.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:18:15 smithi122 ceph-mon[25721]: pgmap v45: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-04-14T14:18:16.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:18:15 smithi122 ceph-mon[25721]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T14:18:16.817 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 14 14:18:16 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-y[63038]: ::ffff:172.21.15.122 - - [14/Apr/2024:14:18:16] "GET /metrics HTTP/1.1" 200 37986 "" "Prometheus/2.43.0" 2024-04-14T14:18:16.817 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:18:16 smithi069 ceph-mon[24818]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' 2024-04-14T14:18:16.817 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:18:16 smithi069 ceph-mon[24818]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' 2024-04-14T14:18:16.817 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:18:16 smithi069 ceph-mon[24818]: pgmap v46: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-04-14T14:18:16.818 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:18:16 smithi069 ceph-mon[29472]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' 2024-04-14T14:18:16.818 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:18:16 smithi069 ceph-mon[29472]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' 2024-04-14T14:18:16.818 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:18:16 smithi069 ceph-mon[29472]: pgmap v46: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-04-14T14:18:16.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:18:16 smithi122 ceph-mon[25721]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' 2024-04-14T14:18:16.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:18:16 smithi122 ceph-mon[25721]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' 2024-04-14T14:18:16.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:18:16 smithi122 ceph-mon[25721]: pgmap v46: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-04-14T14:18:18.033 INFO:journalctl@ceph.alertmanager.a.smithi069.stdout:Apr 14 14:18:17 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-alertmanager-a[78917]: ts=2024-04-14T14:18:17.582Z caller=cluster.go:706 level=info component=cluster msg="gossip not settled" polls=0 before=0 now=1 elapsed=2.000052816s 2024-04-14T14:18:19.567 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:18:19 smithi069 ceph-mon[24818]: pgmap v47: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-14T14:18:19.567 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:18:19 smithi069 ceph-mon[29472]: pgmap v47: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-14T14:18:19.595 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:18:19 smithi122 ceph-mon[25721]: pgmap v47: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-14T14:18:20.817 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:18:20 smithi069 ceph-mon[24818]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' 2024-04-14T14:18:20.817 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:18:20 smithi069 ceph-mon[24818]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' 2024-04-14T14:18:20.817 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:18:20 smithi069 ceph-mon[24818]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' 2024-04-14T14:18:20.817 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:18:20 smithi069 ceph-mon[24818]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' 2024-04-14T14:18:20.817 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:18:20 smithi069 ceph-mon[29472]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' 2024-04-14T14:18:20.817 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:18:20 smithi069 ceph-mon[29472]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' 2024-04-14T14:18:20.817 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:18:20 smithi069 ceph-mon[29472]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' 2024-04-14T14:18:20.817 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:18:20 smithi069 ceph-mon[29472]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' 2024-04-14T14:18:20.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:18:20 smithi122 ceph-mon[25721]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' 2024-04-14T14:18:20.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:18:20 smithi122 ceph-mon[25721]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' 2024-04-14T14:18:20.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:18:20 smithi122 ceph-mon[25721]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' 2024-04-14T14:18:20.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:18:20 smithi122 ceph-mon[25721]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' 2024-04-14T14:18:21.842 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:18:21 smithi069 ceph-mon[29472]: pgmap v48: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:18:21.842 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:18:21 smithi069 ceph-mon[29472]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' 2024-04-14T14:18:21.842 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:18:21 smithi069 ceph-mon[29472]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' 2024-04-14T14:18:21.842 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:18:21 smithi069 ceph-mon[24818]: pgmap v48: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:18:21.842 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:18:21 smithi069 ceph-mon[24818]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' 2024-04-14T14:18:21.842 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:18:21 smithi069 ceph-mon[24818]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' 2024-04-14T14:18:21.844 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:18:21 smithi122 ceph-mon[25721]: pgmap v48: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:18:21.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:18:21 smithi122 ceph-mon[25721]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' 2024-04-14T14:18:21.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:18:21 smithi122 ceph-mon[25721]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' 2024-04-14T14:18:22.940 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:18:22 smithi069 ceph-mon[24818]: from='client.25417 -' entity='client.iscsi.foo.smithi069.yuumtr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T14:18:22.941 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:18:22 smithi069 ceph-mon[29472]: from='client.25417 -' entity='client.iscsi.foo.smithi069.yuumtr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T14:18:23.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:18:22 smithi122 ceph-mon[25721]: from='client.25417 -' entity='client.iscsi.foo.smithi069.yuumtr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T14:18:24.049 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:18:23 smithi069 ceph-mon[24818]: pgmap v49: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:18:24.049 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:18:23 smithi069 ceph-mon[29472]: pgmap v49: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:18:24.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:18:23 smithi122 ceph-mon[25721]: pgmap v49: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:18:25.763 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:18:25 smithi122 ceph-mon[25721]: pgmap v50: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:18:25.763 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:18:25 smithi122 ceph-mon[25721]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' 2024-04-14T14:18:25.763 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:18:25 smithi122 ceph-mon[25721]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' 2024-04-14T14:18:25.763 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:18:25 smithi122 ceph-mon[25721]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T14:18:25.763 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:18:25 smithi122 ceph-mon[25721]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-14T14:18:25.763 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:18:25 smithi122 ceph-mon[25721]: pgmap v51: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1006 B/s rd, 0 op/s 2024-04-14T14:18:25.764 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:18:25 smithi122 ceph-mon[25721]: pgmap v52: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 626 B/s rd, 0 op/s 2024-04-14T14:18:25.764 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:18:25 smithi122 ceph-mon[25721]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' 2024-04-14T14:18:25.764 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:18:25 smithi122 ceph-mon[25721]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-04-14T14:18:25.764 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:18:25 smithi122 ceph-mon[25721]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-04-14T14:18:25.764 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:18:25 smithi122 ceph-mon[25721]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-14T14:18:25.764 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:18:25 smithi122 ceph-mon[25721]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T14:18:25.764 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:18:25 smithi122 ceph-mon[25721]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T14:18:25.764 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:18:25 smithi122 ceph-mon[25721]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T14:18:25.764 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:18:25 smithi122 ceph-mon[25721]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T14:18:25.764 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:18:25 smithi122 ceph-mon[25721]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T14:18:25.764 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:18:25 smithi122 ceph-mon[25721]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T14:18:25.764 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:18:25 smithi122 ceph-mon[25721]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T14:18:25.764 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:18:25 smithi122 ceph-mon[25721]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T14:18:25.764 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:18:25 smithi122 ceph-mon[25721]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T14:18:25.765 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:18:25 smithi122 ceph-mon[25721]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T14:18:25.765 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:18:25 smithi122 ceph-mon[25721]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T14:18:25.817 INFO:journalctl@ceph.alertmanager.a.smithi069.stdout:Apr 14 14:18:25 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-alertmanager-a[78917]: ts=2024-04-14T14:18:25.584Z caller=cluster.go:698 level=info component=cluster msg="gossip settled; proceeding" elapsed=10.001771144s 2024-04-14T14:18:25.817 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:18:25 smithi069 ceph-mon[24818]: pgmap v50: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:18:25.817 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:18:25 smithi069 ceph-mon[24818]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' 2024-04-14T14:18:25.818 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:18:25 smithi069 ceph-mon[24818]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' 2024-04-14T14:18:25.818 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:18:25 smithi069 ceph-mon[24818]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T14:18:25.818 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:18:25 smithi069 ceph-mon[24818]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-14T14:18:25.818 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:18:25 smithi069 ceph-mon[24818]: pgmap v51: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1006 B/s rd, 0 op/s 2024-04-14T14:18:25.818 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:18:25 smithi069 ceph-mon[24818]: pgmap v52: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 626 B/s rd, 0 op/s 2024-04-14T14:18:25.818 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:18:25 smithi069 ceph-mon[24818]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' 2024-04-14T14:18:25.818 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:18:25 smithi069 ceph-mon[24818]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-04-14T14:18:25.818 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:18:25 smithi069 ceph-mon[24818]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-04-14T14:18:25.818 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:18:25 smithi069 ceph-mon[24818]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-14T14:18:25.818 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:18:25 smithi069 ceph-mon[24818]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T14:18:25.818 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:18:25 smithi069 ceph-mon[24818]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T14:18:25.819 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:18:25 smithi069 ceph-mon[24818]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T14:18:25.819 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:18:25 smithi069 ceph-mon[24818]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T14:18:25.819 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:18:25 smithi069 ceph-mon[24818]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T14:18:25.819 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:18:25 smithi069 ceph-mon[24818]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T14:18:25.819 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:18:25 smithi069 ceph-mon[24818]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T14:18:25.819 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:18:25 smithi069 ceph-mon[24818]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T14:18:25.819 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:18:25 smithi069 ceph-mon[24818]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T14:18:25.819 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:18:25 smithi069 ceph-mon[24818]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T14:18:25.819 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:18:25 smithi069 ceph-mon[24818]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T14:18:25.819 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:18:25 smithi069 ceph-mon[29472]: pgmap v50: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:18:25.820 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:18:25 smithi069 ceph-mon[29472]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' 2024-04-14T14:18:25.820 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:18:25 smithi069 ceph-mon[29472]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' 2024-04-14T14:18:25.820 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:18:25 smithi069 ceph-mon[29472]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T14:18:25.820 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:18:25 smithi069 ceph-mon[29472]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-14T14:18:25.820 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:18:25 smithi069 ceph-mon[29472]: pgmap v51: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1006 B/s rd, 0 op/s 2024-04-14T14:18:25.820 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:18:25 smithi069 ceph-mon[29472]: pgmap v52: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 626 B/s rd, 0 op/s 2024-04-14T14:18:25.820 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:18:25 smithi069 ceph-mon[29472]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' 2024-04-14T14:18:25.820 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:18:25 smithi069 ceph-mon[29472]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-04-14T14:18:25.820 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:18:25 smithi069 ceph-mon[29472]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-04-14T14:18:25.820 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:18:25 smithi069 ceph-mon[29472]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-14T14:18:25.820 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:18:25 smithi069 ceph-mon[29472]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T14:18:25.820 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:18:25 smithi069 ceph-mon[29472]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T14:18:25.820 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:18:25 smithi069 ceph-mon[29472]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T14:18:25.821 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:18:25 smithi069 ceph-mon[29472]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T14:18:25.821 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:18:25 smithi069 ceph-mon[29472]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T14:18:25.821 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:18:25 smithi069 ceph-mon[29472]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T14:18:25.821 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:18:25 smithi069 ceph-mon[29472]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T14:18:25.821 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:18:25 smithi069 ceph-mon[29472]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T14:18:25.821 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:18:25 smithi069 ceph-mon[29472]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T14:18:25.821 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:18:25 smithi069 ceph-mon[29472]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T14:18:25.821 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:18:25 smithi069 ceph-mon[29472]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T14:18:26.817 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 14 14:18:26 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-y[63038]: ::ffff:172.21.15.122 - - [14/Apr/2024:14:18:26] "GET /metrics HTTP/1.1" 200 37986 "" "Prometheus/2.43.0" 2024-04-14T14:18:27.741 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:18:27 smithi122 ceph-mon[25721]: Upgrade: Updating grafana.a 2024-04-14T14:18:27.741 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:18:27 smithi122 ceph-mon[25721]: Deploying daemon grafana.a on smithi122 2024-04-14T14:18:27.741 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:18:27 smithi122 ceph-mon[25721]: pgmap v53: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:18:28.067 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:18:27 smithi069 ceph-mon[24818]: Upgrade: Updating grafana.a 2024-04-14T14:18:28.067 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:18:27 smithi069 ceph-mon[24818]: Deploying daemon grafana.a on smithi122 2024-04-14T14:18:28.067 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:18:27 smithi069 ceph-mon[24818]: pgmap v53: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:18:28.067 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:18:27 smithi069 ceph-mon[29472]: Upgrade: Updating grafana.a 2024-04-14T14:18:28.067 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:18:27 smithi069 ceph-mon[29472]: Deploying daemon grafana.a on smithi122 2024-04-14T14:18:28.068 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:18:27 smithi069 ceph-mon[29472]: pgmap v53: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:18:29.345 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Apr 14 14:18:29 smithi122 systemd[1]: Starting Ceph grafana.a for dc326fc2-fa65-11ee-bc8e-c7b262605968... 2024-04-14T14:18:29.746 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:18:29 smithi122 ceph-mon[25721]: pgmap v54: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:18:29.746 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Apr 14 14:18:29 smithi122 podman[66379]: 2024-04-14 14:18:29.345449969 +0000 UTC m=+0.018047819 image pull 2bacad6d85d802cd176c146528adcc0e47a8a83e94bc4a8b7ba1a9b9ef2c65e7 quay.io/ceph/grafana:9.4.12 2024-04-14T14:18:29.746 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Apr 14 14:18:29 smithi122 podman[66379]: 2024-04-14 14:18:29.615359075 +0000 UTC m=+0.287956921 container create 5685178e3eba0255fddc102e698d6275d51cf023aec40b98e018ed2016edb934 (image=quay.io/ceph/grafana:9.4.12, name=ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-grafana-a, maintainer=Grafana Labs ) 2024-04-14T14:18:30.014 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Apr 14 14:18:29 smithi122 podman[66379]: 2024-04-14 14:18:29.859648662 +0000 UTC m=+0.532246509 container init 5685178e3eba0255fddc102e698d6275d51cf023aec40b98e018ed2016edb934 (image=quay.io/ceph/grafana:9.4.12, name=ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-grafana-a, maintainer=Grafana Labs ) 2024-04-14T14:18:30.014 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Apr 14 14:18:29 smithi122 podman[66379]: 2024-04-14 14:18:29.862271363 +0000 UTC m=+0.534869210 container start 5685178e3eba0255fddc102e698d6275d51cf023aec40b98e018ed2016edb934 (image=quay.io/ceph/grafana:9.4.12, name=ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-grafana-a, maintainer=Grafana Labs ) 2024-04-14T14:18:30.014 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Apr 14 14:18:29 smithi122 bash[66379]: 5685178e3eba0255fddc102e698d6275d51cf023aec40b98e018ed2016edb934 2024-04-14T14:18:30.015 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Apr 14 14:18:29 smithi122 systemd[1]: Started Ceph grafana.a for dc326fc2-fa65-11ee-bc8e-c7b262605968. 2024-04-14T14:18:30.067 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:18:29 smithi069 ceph-mon[24818]: pgmap v54: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:18:30.067 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:18:29 smithi069 ceph-mon[29472]: pgmap v54: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:18:30.346 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Apr 14 14:18:30 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-grafana-a[66394]: logger=settings t=2024-04-14T14:18:30.013330289Z level=info msg="Starting Grafana" version=9.4.12 commit=37ee25ba46 branch=HEAD compiled=2023-05-22T15:58:05Z 2024-04-14T14:18:30.346 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Apr 14 14:18:30 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-grafana-a[66394]: logger=settings t=2024-04-14T14:18:30.013536483Z level=warn msg="\"sentry\" frontend logging provider is deprecated and will be removed in the next major version. Use \"grafana\" provider instead." 2024-04-14T14:18:30.346 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Apr 14 14:18:30 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-grafana-a[66394]: logger=settings t=2024-04-14T14:18:30.013594074Z level=info msg="Config loaded from" file=/usr/share/grafana/conf/defaults.ini 2024-04-14T14:18:30.346 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Apr 14 14:18:30 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-grafana-a[66394]: logger=settings t=2024-04-14T14:18:30.013602305Z level=info msg="Config loaded from" file=/etc/grafana/grafana.ini 2024-04-14T14:18:30.346 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Apr 14 14:18:30 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-grafana-a[66394]: logger=settings t=2024-04-14T14:18:30.013607374Z level=info msg="Config overridden from command line" arg="default.paths.data=/var/lib/grafana" 2024-04-14T14:18:30.346 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Apr 14 14:18:30 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-grafana-a[66394]: logger=settings t=2024-04-14T14:18:30.013612365Z level=info msg="Config overridden from command line" arg="default.paths.logs=/var/log/grafana" 2024-04-14T14:18:30.346 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Apr 14 14:18:30 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-grafana-a[66394]: logger=settings t=2024-04-14T14:18:30.013616864Z level=info msg="Config overridden from command line" arg="default.paths.plugins=/var/lib/grafana/plugins" 2024-04-14T14:18:30.346 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Apr 14 14:18:30 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-grafana-a[66394]: logger=settings t=2024-04-14T14:18:30.013621651Z level=info msg="Config overridden from command line" arg="default.paths.provisioning=/etc/grafana/provisioning" 2024-04-14T14:18:30.346 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Apr 14 14:18:30 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-grafana-a[66394]: logger=settings t=2024-04-14T14:18:30.013625429Z level=info msg="Config overridden from command line" arg="default.log.mode=console" 2024-04-14T14:18:30.346 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Apr 14 14:18:30 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-grafana-a[66394]: logger=settings t=2024-04-14T14:18:30.01362986Z level=info msg="Config overridden from Environment variable" var="GF_PATHS_DATA=/var/lib/grafana" 2024-04-14T14:18:30.347 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Apr 14 14:18:30 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-grafana-a[66394]: logger=settings t=2024-04-14T14:18:30.013633994Z level=info msg="Config overridden from Environment variable" var="GF_PATHS_LOGS=/var/log/grafana" 2024-04-14T14:18:30.347 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Apr 14 14:18:30 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-grafana-a[66394]: logger=settings t=2024-04-14T14:18:30.013638608Z level=info msg="Config overridden from Environment variable" var="GF_PATHS_PLUGINS=/var/lib/grafana/plugins" 2024-04-14T14:18:30.347 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Apr 14 14:18:30 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-grafana-a[66394]: logger=settings t=2024-04-14T14:18:30.013643219Z level=info msg="Config overridden from Environment variable" var="GF_PATHS_PROVISIONING=/etc/grafana/provisioning" 2024-04-14T14:18:30.347 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Apr 14 14:18:30 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-grafana-a[66394]: logger=settings t=2024-04-14T14:18:30.013647944Z level=info msg="Path Home" path=/usr/share/grafana 2024-04-14T14:18:30.347 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Apr 14 14:18:30 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-grafana-a[66394]: logger=settings t=2024-04-14T14:18:30.013652781Z level=info msg="Path Data" path=/var/lib/grafana 2024-04-14T14:18:30.347 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Apr 14 14:18:30 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-grafana-a[66394]: logger=settings t=2024-04-14T14:18:30.013656993Z level=info msg="Path Logs" path=/var/log/grafana 2024-04-14T14:18:30.347 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Apr 14 14:18:30 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-grafana-a[66394]: logger=settings t=2024-04-14T14:18:30.013662042Z level=info msg="Path Plugins" path=/var/lib/grafana/plugins 2024-04-14T14:18:30.347 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Apr 14 14:18:30 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-grafana-a[66394]: logger=settings t=2024-04-14T14:18:30.013666519Z level=info msg="Path Provisioning" path=/etc/grafana/provisioning 2024-04-14T14:18:30.348 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Apr 14 14:18:30 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-grafana-a[66394]: logger=settings t=2024-04-14T14:18:30.013671222Z level=info msg="App mode production" 2024-04-14T14:18:30.348 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Apr 14 14:18:30 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-grafana-a[66394]: logger=sqlstore t=2024-04-14T14:18:30.013732818Z level=info msg="Connecting to DB" dbtype=sqlite3 2024-04-14T14:18:30.348 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Apr 14 14:18:30 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-grafana-a[66394]: logger=sqlstore t=2024-04-14T14:18:30.013751108Z level=warn msg="SQLite database file has broader permissions than it should" path=/var/lib/grafana/grafana.db mode=-rw-r--r-- expected=-rw-r----- 2024-04-14T14:18:30.348 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Apr 14 14:18:30 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-grafana-a[66394]: logger=migrator t=2024-04-14T14:18:30.026816161Z level=info msg="Starting DB migrations" 2024-04-14T14:18:30.348 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Apr 14 14:18:30 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-grafana-a[66394]: logger=migrator t=2024-04-14T14:18:30.030287949Z level=info msg="Executing migration" id="Update is_service_account column to nullable" 2024-04-14T14:18:30.348 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Apr 14 14:18:30 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-grafana-a[66394]: logger=migrator t=2024-04-14T14:18:30.041683076Z level=info msg="Executing migration" id="Add isPublic for dashboard" 2024-04-14T14:18:30.348 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Apr 14 14:18:30 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-grafana-a[66394]: logger=migrator t=2024-04-14T14:18:30.042785383Z level=info msg="Executing migration" id="set service account foreign key to nil if 0" 2024-04-14T14:18:30.348 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Apr 14 14:18:30 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-grafana-a[66394]: logger=migrator t=2024-04-14T14:18:30.044256709Z level=info msg="Executing migration" id="Add last_used_at to api_key table" 2024-04-14T14:18:30.348 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Apr 14 14:18:30 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-grafana-a[66394]: logger=migrator t=2024-04-14T14:18:30.046936865Z level=info msg="Executing migration" id="Add is_revoked column to api_key table" 2024-04-14T14:18:30.348 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Apr 14 14:18:30 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-grafana-a[66394]: logger=migrator t=2024-04-14T14:18:30.049303351Z level=info msg="Executing migration" id="Add column preferences.json_data" 2024-04-14T14:18:30.349 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Apr 14 14:18:30 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-grafana-a[66394]: logger=migrator t=2024-04-14T14:18:30.051872749Z level=info msg="Executing migration" id="alter preferences.json_data to mediumtext v1" 2024-04-14T14:18:30.349 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Apr 14 14:18:30 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-grafana-a[66394]: logger=migrator t=2024-04-14T14:18:30.054405656Z level=info msg="Executing migration" id="Add preferences index org_id" 2024-04-14T14:18:30.349 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Apr 14 14:18:30 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-grafana-a[66394]: logger=migrator t=2024-04-14T14:18:30.057128147Z level=info msg="Executing migration" id="Add preferences index user_id" 2024-04-14T14:18:30.349 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Apr 14 14:18:30 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-grafana-a[66394]: logger=migrator t=2024-04-14T14:18:30.059482591Z level=info msg="Executing migration" id="Increase tags column to length 4096" 2024-04-14T14:18:30.349 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Apr 14 14:18:30 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-grafana-a[66394]: logger=migrator t=2024-04-14T14:18:30.059838615Z level=info msg="Executing migration" id="Add OAuth ID token to user_auth" 2024-04-14T14:18:30.349 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Apr 14 14:18:30 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-grafana-a[66394]: logger=migrator t=2024-04-14T14:18:30.062484327Z level=info msg="Executing migration" id="alter table short_url alter column created_by type to bigint" 2024-04-14T14:18:30.349 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Apr 14 14:18:30 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-grafana-a[66394]: logger=migrator t=2024-04-14T14:18:30.064915832Z level=info msg="Executing migration" id="add current_reason column related to current_state" 2024-04-14T14:18:30.349 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Apr 14 14:18:30 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-grafana-a[66394]: logger=migrator t=2024-04-14T14:18:30.067425433Z level=info msg="Executing migration" id="add rule_group_idx column to alert_rule" 2024-04-14T14:18:30.349 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Apr 14 14:18:30 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-grafana-a[66394]: logger=migrator t=2024-04-14T14:18:30.068527122Z level=info msg="Executing migration" id="add is_paused column to alert_rule table" 2024-04-14T14:18:30.349 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Apr 14 14:18:30 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-grafana-a[66394]: logger=migrator t=2024-04-14T14:18:30.070923687Z level=info msg="Executing migration" id="fix is_paused column for alert_rule table" 2024-04-14T14:18:30.349 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Apr 14 14:18:30 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-grafana-a[66394]: logger=migrator t=2024-04-14T14:18:30.073425175Z level=info msg="Executing migration" id="add rule_group_idx column to alert_rule_version" 2024-04-14T14:18:30.349 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Apr 14 14:18:30 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-grafana-a[66394]: logger=migrator t=2024-04-14T14:18:30.075833392Z level=info msg="Executing migration" id="add is_paused column to alert_rule_versions table" 2024-04-14T14:18:30.349 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Apr 14 14:18:30 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-grafana-a[66394]: logger=migrator t=2024-04-14T14:18:30.077005887Z level=info msg="Executing migration" id="fix is_paused column for alert_rule_version table" 2024-04-14T14:18:30.349 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Apr 14 14:18:30 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-grafana-a[66394]: logger=migrator t=2024-04-14T14:18:30.079506007Z level=info msg="Executing migration" id="add configuration_hash column to alert_configuration" 2024-04-14T14:18:30.350 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Apr 14 14:18:30 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-grafana-a[66394]: logger=migrator t=2024-04-14T14:18:30.081869196Z level=info msg="Executing migration" id="add column send_alerts_to in ngalert_configuration" 2024-04-14T14:18:30.350 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Apr 14 14:18:30 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-grafana-a[66394]: logger=migrator t=2024-04-14T14:18:30.084424146Z level=info msg="Executing migration" id="create provenance_type table" 2024-04-14T14:18:30.350 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Apr 14 14:18:30 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-grafana-a[66394]: logger=migrator t=2024-04-14T14:18:30.08487456Z level=info msg="Executing migration" id="add index to uniquify (record_key, record_type, org_id) columns" 2024-04-14T14:18:30.350 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Apr 14 14:18:30 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-grafana-a[66394]: logger=migrator t=2024-04-14T14:18:30.085696754Z level=info msg="Executing migration" id="create alert_image table" 2024-04-14T14:18:30.350 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Apr 14 14:18:30 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-grafana-a[66394]: logger=migrator t=2024-04-14T14:18:30.086486834Z level=info msg="Executing migration" id="add unique index on token to alert_image table" 2024-04-14T14:18:30.350 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Apr 14 14:18:30 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-grafana-a[66394]: logger=migrator t=2024-04-14T14:18:30.087268835Z level=info msg="Executing migration" id="support longer URLs in alert_image table" 2024-04-14T14:18:30.350 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Apr 14 14:18:30 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-grafana-a[66394]: logger=migrator t=2024-04-14T14:18:30.087569368Z level=info msg="Executing migration" id=create_alert_configuration_history_table 2024-04-14T14:18:30.350 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Apr 14 14:18:30 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-grafana-a[66394]: logger=migrator t=2024-04-14T14:18:30.088359915Z level=info msg="Executing migration" id="drop non-unique orgID index on alert_configuration" 2024-04-14T14:18:30.350 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Apr 14 14:18:30 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-grafana-a[66394]: logger=migrator t=2024-04-14T14:18:30.089120942Z level=info msg="Executing migration" id="drop unique orgID index on alert_configuration if exists" 2024-04-14T14:18:30.350 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Apr 14 14:18:30 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-grafana-a[66394]: logger=migrator t=2024-04-14T14:18:30.089207604Z level=warn msg="Skipping migration: Already executed, but not recorded in migration log" id="drop unique orgID index on alert_configuration if exists" 2024-04-14T14:18:30.350 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Apr 14 14:18:30 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-grafana-a[66394]: logger=migrator t=2024-04-14T14:18:30.089897051Z level=info msg="Executing migration" id="extract alertmanager configuration history to separate table" 2024-04-14T14:18:30.350 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Apr 14 14:18:30 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-grafana-a[66394]: logger=migrator t=2024-04-14T14:18:30.090445992Z level=info msg="Executing migration" id="add unique index on orgID to alert_configuration" 2024-04-14T14:18:30.350 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Apr 14 14:18:30 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-grafana-a[66394]: logger=migrator t=2024-04-14T14:18:30.091202645Z level=info msg="Executing migration" id="increase max description length to 2048" 2024-04-14T14:18:30.351 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Apr 14 14:18:30 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-grafana-a[66394]: logger=migrator t=2024-04-14T14:18:30.091517413Z level=info msg="Executing migration" id="create secrets table" 2024-04-14T14:18:30.351 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Apr 14 14:18:30 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-grafana-a[66394]: logger=migrator t=2024-04-14T14:18:30.09200984Z level=info msg="Executing migration" id="rename data_keys name column to id" 2024-04-14T14:18:30.351 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Apr 14 14:18:30 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-grafana-a[66394]: logger=migrator t=2024-04-14T14:18:30.09822727Z level=info msg="Executing migration" id="add name column into data_keys" 2024-04-14T14:18:30.351 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Apr 14 14:18:30 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-grafana-a[66394]: logger=migrator t=2024-04-14T14:18:30.099415614Z level=info msg="Executing migration" id="copy data_keys id column values into name" 2024-04-14T14:18:30.351 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Apr 14 14:18:30 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-grafana-a[66394]: logger=migrator t=2024-04-14T14:18:30.1001972Z level=info msg="Executing migration" id="rename data_keys name column to label" 2024-04-14T14:18:30.351 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Apr 14 14:18:30 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-grafana-a[66394]: logger=migrator t=2024-04-14T14:18:30.106503951Z level=info msg="Executing migration" id="rename data_keys id column back to name" 2024-04-14T14:18:30.351 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Apr 14 14:18:30 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-grafana-a[66394]: logger=migrator t=2024-04-14T14:18:30.112842041Z level=info msg="Executing migration" id="add column hidden to role table" 2024-04-14T14:18:30.351 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Apr 14 14:18:30 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-grafana-a[66394]: logger=migrator t=2024-04-14T14:18:30.114005492Z level=info msg="Executing migration" id="create query_history table v1" 2024-04-14T14:18:30.351 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Apr 14 14:18:30 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-grafana-a[66394]: logger=migrator t=2024-04-14T14:18:30.115330383Z level=info msg="Executing migration" id="add index query_history.org_id-created_by-datasource_uid" 2024-04-14T14:18:30.351 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Apr 14 14:18:30 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-grafana-a[66394]: logger=migrator t=2024-04-14T14:18:30.115807969Z level=info msg="Executing migration" id="alter table query_history alter column created_by type to bigint" 2024-04-14T14:18:30.351 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Apr 14 14:18:30 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-grafana-a[66394]: logger=migrator t=2024-04-14T14:18:30.116569853Z level=info msg="Executing migration" id="teams permissions migration" 2024-04-14T14:18:30.351 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Apr 14 14:18:30 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-grafana-a[66394]: logger=migrator t=2024-04-14T14:18:30.117057762Z level=info msg="Executing migration" id="dashboard permissions" 2024-04-14T14:18:30.351 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Apr 14 14:18:30 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-grafana-a[66394]: logger=migrator t=2024-04-14T14:18:30.118448471Z level=info msg="Executing migration" id="dashboard permissions uid scopes" 2024-04-14T14:18:30.351 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Apr 14 14:18:30 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-grafana-a[66394]: logger=migrator t=2024-04-14T14:18:30.119850541Z level=info msg="Executing migration" id="drop managed folder create actions" 2024-04-14T14:18:30.352 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Apr 14 14:18:30 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-grafana-a[66394]: logger=migrator t=2024-04-14T14:18:30.120621206Z level=info msg="Executing migration" id="alerting notification permissions" 2024-04-14T14:18:30.352 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Apr 14 14:18:30 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-grafana-a[66394]: logger=migrator t=2024-04-14T14:18:30.120989945Z level=info msg="Executing migration" id="create query_history_star table v1" 2024-04-14T14:18:30.352 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Apr 14 14:18:30 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-grafana-a[66394]: logger=migrator t=2024-04-14T14:18:30.122205419Z level=info msg="Executing migration" id="add index query_history.user_id-query_uid" 2024-04-14T14:18:30.352 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Apr 14 14:18:30 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-grafana-a[66394]: logger=migrator t=2024-04-14T14:18:30.122689008Z level=info msg="Executing migration" id="add column org_id in query_history_star" 2024-04-14T14:18:30.352 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Apr 14 14:18:30 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-grafana-a[66394]: logger=migrator t=2024-04-14T14:18:30.123808197Z level=info msg="Executing migration" id="alter table query_history_star_mig column user_id type to bigint" 2024-04-14T14:18:30.353 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Apr 14 14:18:30 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-grafana-a[66394]: logger=migrator t=2024-04-14T14:18:30.124127425Z level=info msg="Executing migration" id="create correlation table v1" 2024-04-14T14:18:30.353 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Apr 14 14:18:30 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-grafana-a[66394]: logger=migrator t=2024-04-14T14:18:30.12502159Z level=info msg="Executing migration" id="add index correlations.uid" 2024-04-14T14:18:30.353 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Apr 14 14:18:30 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-grafana-a[66394]: logger=migrator t=2024-04-14T14:18:30.125755189Z level=info msg="Executing migration" id="add index correlations.source_uid" 2024-04-14T14:18:30.353 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Apr 14 14:18:30 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-grafana-a[66394]: logger=migrator t=2024-04-14T14:18:30.126565023Z level=info msg="Executing migration" id="add correlation config column" 2024-04-14T14:18:30.353 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Apr 14 14:18:30 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-grafana-a[66394]: logger=migrator t=2024-04-14T14:18:30.127746635Z level=info msg="Executing migration" id="create entity_events table" 2024-04-14T14:18:30.353 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Apr 14 14:18:30 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-grafana-a[66394]: logger=migrator t=2024-04-14T14:18:30.128557466Z level=info msg="Executing migration" id="create dashboard public config v1" 2024-04-14T14:18:30.353 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Apr 14 14:18:30 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-grafana-a[66394]: logger=migrator t=2024-04-14T14:18:30.131042298Z level=info msg="Executing migration" id="drop index UQE_dashboard_public_config_uid - v1" 2024-04-14T14:18:30.353 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Apr 14 14:18:30 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-grafana-a[66394]: logger=migrator t=2024-04-14T14:18:30.131120569Z level=warn msg="Skipping migration: Already executed, but not recorded in migration log" id="drop index UQE_dashboard_public_config_uid - v1" 2024-04-14T14:18:30.353 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Apr 14 14:18:30 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-grafana-a[66394]: logger=migrator t=2024-04-14T14:18:30.133730578Z level=info msg="Executing migration" id="drop index IDX_dashboard_public_config_org_id_dashboard_uid - v1" 2024-04-14T14:18:30.353 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Apr 14 14:18:30 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-grafana-a[66394]: logger=migrator t=2024-04-14T14:18:30.133805906Z level=warn msg="Skipping migration: Already executed, but not recorded in migration log" id="drop index IDX_dashboard_public_config_org_id_dashboard_uid - v1" 2024-04-14T14:18:30.353 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Apr 14 14:18:30 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-grafana-a[66394]: logger=migrator t=2024-04-14T14:18:30.134098665Z level=info msg="Executing migration" id="Drop old dashboard public config table" 2024-04-14T14:18:30.353 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Apr 14 14:18:30 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-grafana-a[66394]: logger=migrator t=2024-04-14T14:18:30.136736957Z level=info msg="Executing migration" id="recreate dashboard public config v1" 2024-04-14T14:18:30.353 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Apr 14 14:18:30 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-grafana-a[66394]: logger=migrator t=2024-04-14T14:18:30.139384971Z level=info msg="Executing migration" id="create index UQE_dashboard_public_config_uid - v1" 2024-04-14T14:18:30.353 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Apr 14 14:18:30 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-grafana-a[66394]: logger=migrator t=2024-04-14T14:18:30.141815782Z level=info msg="Executing migration" id="create index IDX_dashboard_public_config_org_id_dashboard_uid - v1" 2024-04-14T14:18:30.353 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Apr 14 14:18:30 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-grafana-a[66394]: logger=migrator t=2024-04-14T14:18:30.142291654Z level=info msg="Executing migration" id="drop index UQE_dashboard_public_config_uid - v2" 2024-04-14T14:18:30.354 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Apr 14 14:18:30 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-grafana-a[66394]: logger=migrator t=2024-04-14T14:18:30.144791055Z level=info msg="Executing migration" id="drop index IDX_dashboard_public_config_org_id_dashboard_uid - v2" 2024-04-14T14:18:30.354 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Apr 14 14:18:30 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-grafana-a[66394]: logger=migrator t=2024-04-14T14:18:30.1474882Z level=info msg="Executing migration" id="Drop public config table" 2024-04-14T14:18:30.354 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Apr 14 14:18:30 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-grafana-a[66394]: logger=migrator t=2024-04-14T14:18:30.150127224Z level=info msg="Executing migration" id="Recreate dashboard public config v2" 2024-04-14T14:18:30.354 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Apr 14 14:18:30 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-grafana-a[66394]: logger=migrator t=2024-04-14T14:18:30.150609949Z level=info msg="Executing migration" id="create index UQE_dashboard_public_config_uid - v2" 2024-04-14T14:18:30.354 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Apr 14 14:18:30 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-grafana-a[66394]: logger=migrator t=2024-04-14T14:18:30.153097755Z level=info msg="Executing migration" id="create index IDX_dashboard_public_config_org_id_dashboard_uid - v2" 2024-04-14T14:18:30.354 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Apr 14 14:18:30 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-grafana-a[66394]: logger=migrator t=2024-04-14T14:18:30.155643915Z level=info msg="Executing migration" id="create index UQE_dashboard_public_config_access_token - v2" 2024-04-14T14:18:30.354 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Apr 14 14:18:30 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-grafana-a[66394]: logger=migrator t=2024-04-14T14:18:30.15835267Z level=info msg="Executing migration" id="Rename table dashboard_public_config to dashboard_public - v2" 2024-04-14T14:18:30.354 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Apr 14 14:18:30 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-grafana-a[66394]: logger=migrator t=2024-04-14T14:18:30.162230879Z level=info msg="Executing migration" id="add annotations_enabled column" 2024-04-14T14:18:30.354 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Apr 14 14:18:30 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-grafana-a[66394]: logger=migrator t=2024-04-14T14:18:30.164951331Z level=info msg="Executing migration" id="add time_selection_enabled column" 2024-04-14T14:18:30.354 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Apr 14 14:18:30 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-grafana-a[66394]: logger=migrator t=2024-04-14T14:18:30.167717658Z level=info msg="Executing migration" id="delete orphaned public dashboards" 2024-04-14T14:18:30.354 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Apr 14 14:18:30 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-grafana-a[66394]: logger=migrator t=2024-04-14T14:18:30.17027082Z level=info msg="Executing migration" id="add share column" 2024-04-14T14:18:30.354 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Apr 14 14:18:30 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-grafana-a[66394]: logger=migrator t=2024-04-14T14:18:30.171552207Z level=info msg="Executing migration" id="create default alerting folders" 2024-04-14T14:18:30.354 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Apr 14 14:18:30 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-grafana-a[66394]: logger=migrator t=2024-04-14T14:18:30.172331447Z level=info msg="Executing migration" id="create file table" 2024-04-14T14:18:30.354 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Apr 14 14:18:30 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-grafana-a[66394]: logger=migrator t=2024-04-14T14:18:30.173113613Z level=info msg="Executing migration" id="file table idx: path natural pk" 2024-04-14T14:18:30.355 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Apr 14 14:18:30 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-grafana-a[66394]: logger=migrator t=2024-04-14T14:18:30.17389608Z level=info msg="Executing migration" id="file table idx: parent_folder_path_hash fast folder retrieval" 2024-04-14T14:18:30.355 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Apr 14 14:18:30 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-grafana-a[66394]: logger=migrator t=2024-04-14T14:18:30.174362574Z level=info msg="Executing migration" id="create file_meta table" 2024-04-14T14:18:30.355 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Apr 14 14:18:30 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-grafana-a[66394]: logger=migrator t=2024-04-14T14:18:30.175136862Z level=info msg="Executing migration" id="file table idx: path key" 2024-04-14T14:18:30.355 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Apr 14 14:18:30 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-grafana-a[66394]: logger=migrator t=2024-04-14T14:18:30.175896649Z level=info msg="Executing migration" id="set path collation in file table" 2024-04-14T14:18:30.355 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Apr 14 14:18:30 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-grafana-a[66394]: logger=migrator t=2024-04-14T14:18:30.176662509Z level=info msg="Executing migration" id="managed permissions migration" 2024-04-14T14:18:30.355 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Apr 14 14:18:30 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-grafana-a[66394]: logger=migrator t=2024-04-14T14:18:30.177799914Z level=info msg="Executing migration" id="managed folder permissions alert actions migration" 2024-04-14T14:18:30.355 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Apr 14 14:18:30 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-grafana-a[66394]: logger=migrator t=2024-04-14T14:18:30.178610205Z level=info msg="Executing migration" id="RBAC action name migrator" 2024-04-14T14:18:30.355 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Apr 14 14:18:30 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-grafana-a[66394]: logger=migrator t=2024-04-14T14:18:30.179421578Z level=info msg="Executing migration" id="Add UID column to playlist" 2024-04-14T14:18:30.355 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Apr 14 14:18:30 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-grafana-a[66394]: logger=migrator t=2024-04-14T14:18:30.180710108Z level=info msg="Executing migration" id="Update uid column values in playlist" 2024-04-14T14:18:30.355 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Apr 14 14:18:30 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-grafana-a[66394]: logger=migrator t=2024-04-14T14:18:30.181035086Z level=info msg="Executing migration" id="Add index for uid in playlist" 2024-04-14T14:18:30.355 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Apr 14 14:18:30 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-grafana-a[66394]: logger=migrator t=2024-04-14T14:18:30.181893422Z level=info msg="Executing migration" id="update group index for alert rules" 2024-04-14T14:18:30.355 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Apr 14 14:18:30 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-grafana-a[66394]: logger=migrator t=2024-04-14T14:18:30.182658691Z level=info msg="Executing migration" id="managed folder permissions alert actions repeated migration" 2024-04-14T14:18:30.355 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Apr 14 14:18:30 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-grafana-a[66394]: logger=migrator t=2024-04-14T14:18:30.18345047Z level=info msg="Executing migration" id="admin only folder/dashboard permission" 2024-04-14T14:18:30.355 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Apr 14 14:18:30 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-grafana-a[66394]: logger=migrator t=2024-04-14T14:18:30.183853981Z level=info msg="Executing migration" id="add action column to seed_assignment" 2024-04-14T14:18:30.356 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Apr 14 14:18:30 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-grafana-a[66394]: logger=migrator t=2024-04-14T14:18:30.185068376Z level=info msg="Executing migration" id="add scope column to seed_assignment" 2024-04-14T14:18:30.356 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Apr 14 14:18:30 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-grafana-a[66394]: logger=migrator t=2024-04-14T14:18:30.186273194Z level=info msg="Executing migration" id="remove unique index builtin_role_role_name before nullable update" 2024-04-14T14:18:30.356 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Apr 14 14:18:30 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-grafana-a[66394]: logger=migrator t=2024-04-14T14:18:30.187078621Z level=info msg="Executing migration" id="update seed_assignment role_name column to nullable" 2024-04-14T14:18:30.356 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Apr 14 14:18:30 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-grafana-a[66394]: logger=migrator t=2024-04-14T14:18:30.200369147Z level=info msg="Executing migration" id="add unique index builtin_role_name back" 2024-04-14T14:18:30.356 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Apr 14 14:18:30 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-grafana-a[66394]: logger=migrator t=2024-04-14T14:18:30.201115977Z level=info msg="Executing migration" id="add unique index builtin_role_action_scope" 2024-04-14T14:18:30.356 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Apr 14 14:18:30 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-grafana-a[66394]: logger=migrator t=2024-04-14T14:18:30.201933503Z level=info msg="Executing migration" id="add primary key to seed_assigment" 2024-04-14T14:18:30.356 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Apr 14 14:18:30 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-grafana-a[66394]: logger=migrator t=2024-04-14T14:18:30.206422317Z level=info msg="Executing migration" id="managed folder permissions alert actions repeated fixed migration" 2024-04-14T14:18:30.356 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Apr 14 14:18:30 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-grafana-a[66394]: logger=migrator t=2024-04-14T14:18:30.207160974Z level=info msg="Executing migration" id="migrate external alertmanagers to datsourcse" 2024-04-14T14:18:30.356 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Apr 14 14:18:30 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-grafana-a[66394]: logger=migrator t=2024-04-14T14:18:30.207985715Z level=info msg="Executing migration" id="create folder table" 2024-04-14T14:18:30.356 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Apr 14 14:18:30 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-grafana-a[66394]: logger=migrator t=2024-04-14T14:18:30.208824608Z level=info msg="Executing migration" id="Add index for parent_uid" 2024-04-14T14:18:30.356 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Apr 14 14:18:30 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-grafana-a[66394]: logger=migrator t=2024-04-14T14:18:30.211474036Z level=info msg="Executing migration" id="Add unique index for folder.uid and folder.org_id" 2024-04-14T14:18:30.356 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Apr 14 14:18:30 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-grafana-a[66394]: logger=migrator t=2024-04-14T14:18:30.21193906Z level=info msg="Executing migration" id="Update folder title length" 2024-04-14T14:18:30.356 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Apr 14 14:18:30 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-grafana-a[66394]: logger=migrator t=2024-04-14T14:18:30.214616832Z level=info msg="Executing migration" id="Add unique index for folder.title and folder.parent_uid" 2024-04-14T14:18:30.356 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Apr 14 14:18:30 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-grafana-a[66394]: logger=migrator t=2024-04-14T14:18:30.217248833Z level=info msg="migrations completed" performed=105 skipped=376 duration=187.001518ms 2024-04-14T14:18:30.356 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Apr 14 14:18:30 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-grafana-a[66394]: logger=sqlstore t=2024-04-14T14:18:30.217573864Z level=info msg="Created default organization" 2024-04-14T14:18:30.357 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Apr 14 14:18:30 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-grafana-a[66394]: logger=plugin.finder t=2024-04-14T14:18:30.250611779Z level=warn msg="Skipping finding plugins as directory does not exist" path=/usr/share/grafana/plugins-bundled 2024-04-14T14:18:30.357 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Apr 14 14:18:30 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-grafana-a[66394]: logger=secrets t=2024-04-14T14:18:30.250717239Z level=info msg="Envelope encryption state" enabled=true currentprovider=secretKey.v1 2024-04-14T14:18:30.357 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Apr 14 14:18:30 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-grafana-a[66394]: logger=query_data t=2024-04-14T14:18:30.254254637Z level=info msg="Query Service initialization" 2024-04-14T14:18:30.357 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Apr 14 14:18:30 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-grafana-a[66394]: logger=live.push_http t=2024-04-14T14:18:30.256683807Z level=info msg="Live Push Gateway initialization" 2024-04-14T14:18:30.867 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:18:30 smithi122 ceph-mon[25721]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T14:18:30.867 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:18:30 smithi122 ceph-mon[25721]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' 2024-04-14T14:18:30.867 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:18:30 smithi122 ceph-mon[25721]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' 2024-04-14T14:18:30.924 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:18:30 smithi069 ceph-mon[24818]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T14:18:30.924 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:18:30 smithi069 ceph-mon[24818]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' 2024-04-14T14:18:30.924 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:18:30 smithi069 ceph-mon[24818]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' 2024-04-14T14:18:30.924 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:18:30 smithi069 ceph-mon[29472]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T14:18:30.924 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:18:30 smithi069 ceph-mon[29472]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' 2024-04-14T14:18:30.925 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:18:30 smithi069 ceph-mon[29472]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' 2024-04-14T14:18:31.126 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Apr 14 14:18:30 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-grafana-a[66394]: logger=infra.usagestats.collector t=2024-04-14T14:18:30.889742728Z level=info msg="registering usage stat providers" usageStatsProvidersLen=2 2024-04-14T14:18:31.126 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Apr 14 14:18:30 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-grafana-a[66394]: logger=provisioning.datasources t=2024-04-14T14:18:30.891651029Z level=info msg="deleted correlations based on configuration" ds_name=Dashboard1 2024-04-14T14:18:31.126 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Apr 14 14:18:30 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-grafana-a[66394]: logger=provisioning.datasources t=2024-04-14T14:18:30.891659167Z level=info msg="deleted datasource based on configuration" name=Dashboard1 2024-04-14T14:18:31.126 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Apr 14 14:18:30 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-grafana-a[66394]: logger=provisioning.datasources t=2024-04-14T14:18:30.89175457Z level=info msg="inserting datasource from configuration " name=Dashboard1 uid=P43CA22E17D0F9596 2024-04-14T14:18:31.126 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Apr 14 14:18:30 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-grafana-a[66394]: logger=provisioning.datasources t=2024-04-14T14:18:30.902938131Z level=info msg="inserting datasource from configuration " name=Loki uid=P8E80F9AEF21F6940 2024-04-14T14:18:31.126 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Apr 14 14:18:30 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-grafana-a[66394]: logger=provisioning.alerting t=2024-04-14T14:18:30.913847032Z level=info msg="starting to provision alerting" 2024-04-14T14:18:31.126 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Apr 14 14:18:30 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-grafana-a[66394]: logger=provisioning.alerting t=2024-04-14T14:18:30.913857532Z level=info msg="finished to provision alerting" 2024-04-14T14:18:31.127 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Apr 14 14:18:30 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-grafana-a[66394]: logger=ngalert.state.manager t=2024-04-14T14:18:30.914032356Z level=info msg="Warming state cache for startup" 2024-04-14T14:18:31.127 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Apr 14 14:18:30 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-grafana-a[66394]: logger=grafanaStorageLogger t=2024-04-14T14:18:30.914055361Z level=info msg="storage starting" 2024-04-14T14:18:31.127 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Apr 14 14:18:30 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-grafana-a[66394]: logger=ngalert.state.manager t=2024-04-14T14:18:30.916377497Z level=info msg="State cache has been initialized" states=0 duration=2.334408ms 2024-04-14T14:18:31.127 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Apr 14 14:18:30 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-grafana-a[66394]: logger=ticker t=2024-04-14T14:18:30.916561675Z level=info msg=starting first_tick=2024-04-14T14:18:40Z 2024-04-14T14:18:31.127 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Apr 14 14:18:30 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-grafana-a[66394]: logger=ngalert.multiorg.alertmanager t=2024-04-14T14:18:30.916612108Z level=info msg="starting MultiOrg Alertmanager" 2024-04-14T14:18:31.127 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Apr 14 14:18:30 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-grafana-a[66394]: logger=http.server t=2024-04-14T14:18:30.921328104Z level=info msg="HTTP Server Listen" address=[::]:3000 protocol=https subUrl= socket= 2024-04-14T14:18:31.127 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Apr 14 14:18:30 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-grafana-a[66394]: logger=sqlstore.transactions t=2024-04-14T14:18:30.933198782Z level=info msg="Database locked, sleeping then retrying" error="database is locked" retry=0 code="database is locked" 2024-04-14T14:18:31.127 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Apr 14 14:18:30 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-grafana-a[66394]: logger=sqlstore.transactions t=2024-04-14T14:18:30.943891116Z level=info msg="Database locked, sleeping then retrying" error="database is locked" retry=0 code="database is locked" 2024-04-14T14:18:31.127 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Apr 14 14:18:30 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-grafana-a[66394]: logger=sqlstore.transactions t=2024-04-14T14:18:30.955597404Z level=info msg="Database locked, sleeping then retrying" error="database is locked" retry=1 code="database is locked" 2024-04-14T14:18:31.127 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Apr 14 14:18:30 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-grafana-a[66394]: logger=sqlstore.transactions t=2024-04-14T14:18:30.967204251Z level=info msg="Database locked, sleeping then retrying" error="database is locked" retry=2 code="database is locked" 2024-04-14T14:18:31.846 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:18:31 smithi122 ceph-mon[25721]: pgmap v55: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 626 B/s rd, 0 op/s 2024-04-14T14:18:32.067 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:18:31 smithi069 ceph-mon[24818]: pgmap v55: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 626 B/s rd, 0 op/s 2024-04-14T14:18:32.067 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:18:31 smithi069 ceph-mon[29472]: pgmap v55: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 626 B/s rd, 0 op/s 2024-04-14T14:18:32.764 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:18:32 smithi122 ceph-mon[25721]: from='client.25417 -' entity='client.iscsi.foo.smithi069.yuumtr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T14:18:33.067 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:18:32 smithi069 ceph-mon[24818]: from='client.25417 -' entity='client.iscsi.foo.smithi069.yuumtr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T14:18:33.067 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:18:32 smithi069 ceph-mon[29472]: from='client.25417 -' entity='client.iscsi.foo.smithi069.yuumtr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T14:18:33.928 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:18:33 smithi122 ceph-mon[25721]: pgmap v56: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:18:34.067 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:18:33 smithi069 ceph-mon[24818]: pgmap v56: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:18:34.067 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:18:33 smithi069 ceph-mon[29472]: pgmap v56: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:18:35.817 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:18:35 smithi069 ceph-mon[24818]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' 2024-04-14T14:18:35.817 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:18:35 smithi069 ceph-mon[24818]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' 2024-04-14T14:18:35.817 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:18:35 smithi069 ceph-mon[24818]: pgmap v57: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-14T14:18:35.818 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:18:35 smithi069 ceph-mon[29472]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' 2024-04-14T14:18:35.818 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:18:35 smithi069 ceph-mon[29472]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' 2024-04-14T14:18:35.818 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:18:35 smithi069 ceph-mon[29472]: pgmap v57: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-14T14:18:35.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:18:35 smithi122 ceph-mon[25721]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' 2024-04-14T14:18:35.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:18:35 smithi122 ceph-mon[25721]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' 2024-04-14T14:18:35.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:18:35 smithi122 ceph-mon[25721]: pgmap v57: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-14T14:18:36.350 INFO:teuthology.orchestra.run.smithi069.stdout:true 2024-04-14T14:18:36.595 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:18:36 smithi122 ceph-mon[25721]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' 2024-04-14T14:18:36.595 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:18:36 smithi122 ceph-mon[25721]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' 2024-04-14T14:18:36.817 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:18:36 smithi069 ceph-mon[24818]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' 2024-04-14T14:18:36.817 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:18:36 smithi069 ceph-mon[24818]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' 2024-04-14T14:18:36.817 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 14 14:18:36 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-y[63038]: ::ffff:172.21.15.122 - - [14/Apr/2024:14:18:36] "GET /metrics HTTP/1.1" 200 37987 "" "Prometheus/2.43.0" 2024-04-14T14:18:36.817 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:18:36 smithi069 ceph-mon[29472]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' 2024-04-14T14:18:36.818 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:18:36 smithi069 ceph-mon[29472]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' 2024-04-14T14:18:37.005 INFO:teuthology.orchestra.run.smithi069.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-14T14:18:37.005 INFO:teuthology.orchestra.run.smithi069.stdout:alertmanager.a smithi069 *:9093,9094 running (21s) 16s ago 21m 16.5M - 0.25.0 c8568f914cd2 7524f106e7bd 2024-04-14T14:18:37.005 INFO:teuthology.orchestra.run.smithi069.stdout:grafana.a smithi122 *:3000 running (7s) 2s ago 21m 53.1M - 9.4.12 2bacad6d85d8 5685178e3eba 2024-04-14T14:18:37.005 INFO:teuthology.orchestra.run.smithi069.stdout:iscsi.foo.smithi069.yuumtr smithi069 running (86s) 16s ago 21m 44.5M - 3.5 e1d6a67b021e 38cc81ea8cfa 2024-04-14T14:18:37.005 INFO:teuthology.orchestra.run.smithi069.stdout:mgr.x smithi122 *:8443,9283,8765 running (76s) 2s ago 26m 450M - 19.0.0-2778-ga971fd5f b96c509eef84 e29977590b43 2024-04-14T14:18:37.005 INFO:teuthology.orchestra.run.smithi069.stdout:mgr.y smithi069 *:8443,9283,8765 running (10m) 16s ago 28m 504M - 19.0.0-2778-ga971fd5f b96c509eef84 7af3d29d0e5f 2024-04-14T14:18:37.005 INFO:teuthology.orchestra.run.smithi069.stdout:mon.a smithi069 running (28m) 16s ago 28m 78.2M 2048M 17.2.0 e1d6a67b021e d3c553f91913 2024-04-14T14:18:37.005 INFO:teuthology.orchestra.run.smithi069.stdout:mon.b smithi122 running (26m) 2s ago 26m 55.4M 2048M 17.2.0 e1d6a67b021e 3b766bfce4be 2024-04-14T14:18:37.005 INFO:teuthology.orchestra.run.smithi069.stdout:mon.c smithi069 running (26m) 16s ago 26m 57.4M 2048M 17.2.0 e1d6a67b021e 23c078b9d1e5 2024-04-14T14:18:37.005 INFO:teuthology.orchestra.run.smithi069.stdout:node-exporter.a smithi069 *:9100 running (60s) 16s ago 22m 8258k - 1.5.0 0da6a335fe13 0de0eb2171b9 2024-04-14T14:18:37.006 INFO:teuthology.orchestra.run.smithi069.stdout:node-exporter.b smithi122 *:9100 running (53s) 2s ago 22m 16.4M - 1.5.0 0da6a335fe13 67877bf02d8e 2024-04-14T14:18:37.006 INFO:teuthology.orchestra.run.smithi069.stdout:osd.0 smithi069 running (26m) 16s ago 26m 56.4M 2992M 17.2.0 e1d6a67b021e 46654654b6b6 2024-04-14T14:18:37.006 INFO:teuthology.orchestra.run.smithi069.stdout:osd.1 smithi069 running (25m) 16s ago 25m 54.5M 2992M 17.2.0 e1d6a67b021e cdba54a59609 2024-04-14T14:18:37.006 INFO:teuthology.orchestra.run.smithi069.stdout:osd.2 smithi069 running (25m) 16s ago 25m 50.9M 2992M 17.2.0 e1d6a67b021e bc37a1a9cba0 2024-04-14T14:18:37.006 INFO:teuthology.orchestra.run.smithi069.stdout:osd.3 smithi069 running (25m) 16s ago 25m 55.2M 2992M 17.2.0 e1d6a67b021e 0925ba3da801 2024-04-14T14:18:37.006 INFO:teuthology.orchestra.run.smithi069.stdout:osd.4 smithi122 running (24m) 2s ago 24m 56.0M 3248M 17.2.0 e1d6a67b021e c00975caf771 2024-04-14T14:18:37.006 INFO:teuthology.orchestra.run.smithi069.stdout:osd.5 smithi122 running (24m) 2s ago 24m 55.1M 3248M 17.2.0 e1d6a67b021e 5f52d217ceb8 2024-04-14T14:18:37.006 INFO:teuthology.orchestra.run.smithi069.stdout:osd.6 smithi122 running (23m) 2s ago 23m 51.8M 3248M 17.2.0 e1d6a67b021e a2acfec61475 2024-04-14T14:18:37.006 INFO:teuthology.orchestra.run.smithi069.stdout:osd.7 smithi122 running (23m) 2s ago 23m 56.8M 3248M 17.2.0 e1d6a67b021e 62411cd06801 2024-04-14T14:18:37.006 INFO:teuthology.orchestra.run.smithi069.stdout:prometheus.a smithi122 *:9095 running (37s) 2s ago 22m 55.4M - 2.43.0 a07b618ecd1d 5c1f41d3bbe6 2024-04-14T14:18:37.006 INFO:teuthology.orchestra.run.smithi069.stdout:rgw.foo.smithi069.wesedz smithi069 *:8000 running (21m) 16s ago 21m 77.8M - 17.2.0 e1d6a67b021e 16de491440f3 2024-04-14T14:18:37.006 INFO:teuthology.orchestra.run.smithi069.stdout:rgw.foo.smithi122.jiobyv smithi122 *:8000 running (21m) 2s ago 21m 74.9M - 17.2.0 e1d6a67b021e 6d16de445dba 2024-04-14T14:18:37.395 INFO:teuthology.orchestra.run.smithi069.stdout:{ 2024-04-14T14:18:37.395 INFO:teuthology.orchestra.run.smithi069.stdout: "mon": { 2024-04-14T14:18:37.395 INFO:teuthology.orchestra.run.smithi069.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 3 2024-04-14T14:18:37.395 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-14T14:18:37.395 INFO:teuthology.orchestra.run.smithi069.stdout: "mgr": { 2024-04-14T14:18:37.395 INFO:teuthology.orchestra.run.smithi069.stdout: "ceph version 19.0.0-2778-ga971fd5f (a971fd5f5719038c70b817bafacd6d5400b1d888) squid (dev)": 2 2024-04-14T14:18:37.395 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-14T14:18:37.395 INFO:teuthology.orchestra.run.smithi069.stdout: "osd": { 2024-04-14T14:18:37.395 INFO:teuthology.orchestra.run.smithi069.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 8 2024-04-14T14:18:37.395 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-14T14:18:37.396 INFO:teuthology.orchestra.run.smithi069.stdout: "mds": {}, 2024-04-14T14:18:37.396 INFO:teuthology.orchestra.run.smithi069.stdout: "rgw": { 2024-04-14T14:18:37.396 INFO:teuthology.orchestra.run.smithi069.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-04-14T14:18:37.396 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-14T14:18:37.396 INFO:teuthology.orchestra.run.smithi069.stdout: "overall": { 2024-04-14T14:18:37.396 INFO:teuthology.orchestra.run.smithi069.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 13, 2024-04-14T14:18:37.396 INFO:teuthology.orchestra.run.smithi069.stdout: "ceph version 19.0.0-2778-ga971fd5f (a971fd5f5719038c70b817bafacd6d5400b1d888) squid (dev)": 2 2024-04-14T14:18:37.396 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-14T14:18:37.396 INFO:teuthology.orchestra.run.smithi069.stdout:} 2024-04-14T14:18:37.733 INFO:teuthology.orchestra.run.smithi069.stdout:{ 2024-04-14T14:18:37.733 INFO:teuthology.orchestra.run.smithi069.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888", 2024-04-14T14:18:37.733 INFO:teuthology.orchestra.run.smithi069.stdout: "in_progress": true, 2024-04-14T14:18:37.733 INFO:teuthology.orchestra.run.smithi069.stdout: "which": "Upgrading daemons of type(s) mgr", 2024-04-14T14:18:37.733 INFO:teuthology.orchestra.run.smithi069.stdout: "services_complete": [ 2024-04-14T14:18:37.733 INFO:teuthology.orchestra.run.smithi069.stdout: "mgr" 2024-04-14T14:18:37.733 INFO:teuthology.orchestra.run.smithi069.stdout: ], 2024-04-14T14:18:37.733 INFO:teuthology.orchestra.run.smithi069.stdout: "progress": "2/2 daemons upgraded", 2024-04-14T14:18:37.733 INFO:teuthology.orchestra.run.smithi069.stdout: "message": "Currently upgrading grafana daemons", 2024-04-14T14:18:37.733 INFO:teuthology.orchestra.run.smithi069.stdout: "is_paused": false 2024-04-14T14:18:37.734 INFO:teuthology.orchestra.run.smithi069.stdout:} 2024-04-14T14:18:37.734 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:18:37 smithi069 ceph-mon[24818]: from='client.15594 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T14:18:37.734 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:18:37 smithi069 ceph-mon[24818]: pgmap v58: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:18:37.734 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:18:37 smithi069 ceph-mon[24818]: from='client.15600 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T14:18:37.734 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:18:37 smithi069 ceph-mon[24818]: from='client.? 172.21.15.69:0/1359711347' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T14:18:37.734 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:18:37 smithi069 ceph-mon[29472]: from='client.15594 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T14:18:37.734 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:18:37 smithi069 ceph-mon[29472]: pgmap v58: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:18:37.734 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:18:37 smithi069 ceph-mon[29472]: from='client.15600 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T14:18:37.735 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:18:37 smithi069 ceph-mon[29472]: from='client.? 172.21.15.69:0/1359711347' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T14:18:37.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:18:37 smithi122 ceph-mon[25721]: from='client.15594 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T14:18:37.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:18:37 smithi122 ceph-mon[25721]: pgmap v58: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:18:37.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:18:37 smithi122 ceph-mon[25721]: from='client.15600 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T14:18:37.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:18:37 smithi122 ceph-mon[25721]: from='client.? 172.21.15.69:0/1359711347' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T14:18:39.067 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:18:38 smithi069 ceph-mon[24818]: from='client.15606 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T14:18:39.067 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:18:38 smithi069 ceph-mon[29472]: from='client.15606 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T14:18:39.094 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:18:38 smithi122 ceph-mon[25721]: from='client.15606 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T14:18:40.067 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:18:39 smithi069 ceph-mon[24818]: from='client.15618 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T14:18:40.068 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:18:39 smithi069 ceph-mon[24818]: pgmap v59: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:18:40.068 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:18:39 smithi069 ceph-mon[24818]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' 2024-04-14T14:18:40.068 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:18:39 smithi069 ceph-mon[24818]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' 2024-04-14T14:18:40.068 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:18:39 smithi069 ceph-mon[24818]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T14:18:40.068 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:18:39 smithi069 ceph-mon[24818]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-14T14:18:40.068 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:18:39 smithi069 ceph-mon[24818]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' 2024-04-14T14:18:40.068 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:18:39 smithi069 ceph-mon[24818]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-04-14T14:18:40.068 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:18:39 smithi069 ceph-mon[24818]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-14T14:18:40.068 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:18:39 smithi069 ceph-mon[24818]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T14:18:40.068 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:18:39 smithi069 ceph-mon[24818]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T14:18:40.068 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:18:39 smithi069 ceph-mon[24818]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T14:18:40.068 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:18:39 smithi069 ceph-mon[24818]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T14:18:40.068 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:18:39 smithi069 ceph-mon[24818]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T14:18:40.068 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:18:39 smithi069 ceph-mon[24818]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T14:18:40.069 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:18:39 smithi069 ceph-mon[24818]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T14:18:40.069 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:18:39 smithi069 ceph-mon[24818]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T14:18:40.069 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:18:39 smithi069 ceph-mon[24818]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T14:18:40.069 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:18:39 smithi069 ceph-mon[24818]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T14:18:40.069 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:18:39 smithi069 ceph-mon[24818]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T14:18:40.069 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:18:39 smithi069 ceph-mon[24818]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T14:18:40.069 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:18:39 smithi069 ceph-mon[24818]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T14:18:40.069 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:18:39 smithi069 ceph-mon[24818]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T14:18:40.069 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:18:39 smithi069 ceph-mon[24818]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' 2024-04-14T14:18:40.069 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:18:39 smithi069 ceph-mon[24818]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]: dispatch 2024-04-14T14:18:40.069 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:18:39 smithi069 ceph-mon[24818]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]': finished 2024-04-14T14:18:40.069 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:18:39 smithi069 ceph-mon[24818]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-14T14:18:40.069 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:18:39 smithi069 ceph-mon[24818]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon"}]': finished 2024-04-14T14:18:40.070 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:18:39 smithi069 ceph-mon[29472]: from='client.15618 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T14:18:40.070 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:18:39 smithi069 ceph-mon[29472]: pgmap v59: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:18:40.070 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:18:39 smithi069 ceph-mon[29472]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' 2024-04-14T14:18:40.070 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:18:39 smithi069 ceph-mon[29472]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' 2024-04-14T14:18:40.070 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:18:39 smithi069 ceph-mon[29472]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T14:18:40.070 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:18:39 smithi069 ceph-mon[29472]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-14T14:18:40.070 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:18:39 smithi069 ceph-mon[29472]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' 2024-04-14T14:18:40.070 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:18:39 smithi069 ceph-mon[29472]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-04-14T14:18:40.070 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:18:39 smithi069 ceph-mon[29472]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-14T14:18:40.070 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:18:39 smithi069 ceph-mon[29472]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T14:18:40.070 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:18:39 smithi069 ceph-mon[29472]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T14:18:40.070 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:18:39 smithi069 ceph-mon[29472]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T14:18:40.071 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:18:39 smithi069 ceph-mon[29472]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T14:18:40.071 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:18:39 smithi069 ceph-mon[29472]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T14:18:40.071 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:18:39 smithi069 ceph-mon[29472]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T14:18:40.071 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:18:39 smithi069 ceph-mon[29472]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T14:18:40.071 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:18:39 smithi069 ceph-mon[29472]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T14:18:40.071 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:18:39 smithi069 ceph-mon[29472]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T14:18:40.071 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:18:39 smithi069 ceph-mon[29472]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T14:18:40.071 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:18:39 smithi069 ceph-mon[29472]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T14:18:40.071 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:18:39 smithi069 ceph-mon[29472]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T14:18:40.071 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:18:39 smithi069 ceph-mon[29472]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T14:18:40.071 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:18:39 smithi069 ceph-mon[29472]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T14:18:40.071 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:18:39 smithi069 ceph-mon[29472]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' 2024-04-14T14:18:40.071 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:18:39 smithi069 ceph-mon[29472]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]: dispatch 2024-04-14T14:18:40.071 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:18:39 smithi069 ceph-mon[29472]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]': finished 2024-04-14T14:18:40.071 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:18:39 smithi069 ceph-mon[29472]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-14T14:18:40.072 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:18:39 smithi069 ceph-mon[29472]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon"}]': finished 2024-04-14T14:18:40.072 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:18:39 smithi069 ceph-mon[24818]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]: dispatch 2024-04-14T14:18:40.072 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:18:39 smithi069 ceph-mon[24818]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]': finished 2024-04-14T14:18:40.072 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:18:39 smithi069 ceph-mon[24818]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd"}]: dispatch 2024-04-14T14:18:40.072 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:18:39 smithi069 ceph-mon[24818]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mds"}]: dispatch 2024-04-14T14:18:40.072 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:18:39 smithi069 ceph-mon[24818]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mds"}]': finished 2024-04-14T14:18:40.072 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:18:39 smithi069 ceph-mon[24818]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw"}]: dispatch 2024-04-14T14:18:40.072 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:18:39 smithi069 ceph-mon[24818]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"}]: dispatch 2024-04-14T14:18:40.072 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:18:39 smithi069 ceph-mon[24818]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"}]': finished 2024-04-14T14:18:40.072 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:18:39 smithi069 ceph-mon[24818]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-14T14:18:40.072 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:18:39 smithi069 ceph-mon[24818]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter"}]: dispatch 2024-04-14T14:18:40.073 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:18:39 smithi069 ceph-mon[24818]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter"}]': finished 2024-04-14T14:18:40.073 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:18:39 smithi069 ceph-mon[24818]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.iscsi"}]: dispatch 2024-04-14T14:18:40.073 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:18:39 smithi069 ceph-mon[24818]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nfs"}]: dispatch 2024-04-14T14:18:40.073 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:18:39 smithi069 ceph-mon[24818]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.nfs"}]': finished 2024-04-14T14:18:40.073 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:18:39 smithi069 ceph-mon[24818]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]: dispatch 2024-04-14T14:18:40.073 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:18:39 smithi069 ceph-mon[24818]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]': finished 2024-04-14T14:18:40.073 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:18:39 smithi069 ceph-mon[24818]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-14T14:18:40.073 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:18:39 smithi069 ceph-mon[24818]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-14T14:18:40.073 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:18:39 smithi069 ceph-mon[24818]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-14T14:18:40.073 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:18:39 smithi069 ceph-mon[24818]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-14T14:18:40.073 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:18:39 smithi069 ceph-mon[24818]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-14T14:18:40.074 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:18:39 smithi069 ceph-mon[29472]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]: dispatch 2024-04-14T14:18:40.074 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:18:39 smithi069 ceph-mon[29472]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]': finished 2024-04-14T14:18:40.074 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:18:39 smithi069 ceph-mon[29472]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd"}]: dispatch 2024-04-14T14:18:40.074 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:18:39 smithi069 ceph-mon[29472]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mds"}]: dispatch 2024-04-14T14:18:40.074 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:18:39 smithi069 ceph-mon[29472]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mds"}]': finished 2024-04-14T14:18:40.074 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:18:39 smithi069 ceph-mon[29472]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw"}]: dispatch 2024-04-14T14:18:40.074 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:18:39 smithi069 ceph-mon[29472]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"}]: dispatch 2024-04-14T14:18:40.074 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:18:39 smithi069 ceph-mon[29472]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"}]': finished 2024-04-14T14:18:40.074 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:18:39 smithi069 ceph-mon[29472]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-14T14:18:40.074 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:18:39 smithi069 ceph-mon[29472]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter"}]: dispatch 2024-04-14T14:18:40.074 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:18:39 smithi069 ceph-mon[29472]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter"}]': finished 2024-04-14T14:18:40.074 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:18:39 smithi069 ceph-mon[29472]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.iscsi"}]: dispatch 2024-04-14T14:18:40.074 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:18:39 smithi069 ceph-mon[29472]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nfs"}]: dispatch 2024-04-14T14:18:40.074 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:18:39 smithi069 ceph-mon[29472]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.nfs"}]': finished 2024-04-14T14:18:40.075 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:18:39 smithi069 ceph-mon[29472]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]: dispatch 2024-04-14T14:18:40.075 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:18:39 smithi069 ceph-mon[29472]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]': finished 2024-04-14T14:18:40.075 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:18:39 smithi069 ceph-mon[29472]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-14T14:18:40.075 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:18:39 smithi069 ceph-mon[29472]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-14T14:18:40.075 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:18:39 smithi069 ceph-mon[29472]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-14T14:18:40.075 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:18:39 smithi069 ceph-mon[29472]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-14T14:18:40.075 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:18:39 smithi069 ceph-mon[29472]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-14T14:18:40.075 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:18:39 smithi069 ceph-mon[24818]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-14T14:18:40.075 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:18:39 smithi069 ceph-mon[24818]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' cmd=[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]: dispatch 2024-04-14T14:18:40.075 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:18:39 smithi069 ceph-mon[24818]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' cmd='[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]': finished 2024-04-14T14:18:40.075 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:18:39 smithi069 ceph-mon[24818]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T14:18:40.076 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:18:39 smithi069 ceph-mon[24818]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-14T14:18:40.076 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:18:39 smithi069 ceph-mon[24818]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' 2024-04-14T14:18:40.076 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:18:39 smithi069 ceph-mon[24818]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T14:18:40.076 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:18:39 smithi069 ceph-mon[24818]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-14T14:18:40.076 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:18:39 smithi069 ceph-mon[24818]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' 2024-04-14T14:18:40.076 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:18:39 smithi069 ceph-mon[29472]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-14T14:18:40.076 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:18:39 smithi069 ceph-mon[29472]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' cmd=[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]: dispatch 2024-04-14T14:18:40.076 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:18:39 smithi069 ceph-mon[29472]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' cmd='[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]': finished 2024-04-14T14:18:40.076 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:18:39 smithi069 ceph-mon[29472]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T14:18:40.076 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:18:39 smithi069 ceph-mon[29472]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-14T14:18:40.076 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:18:39 smithi069 ceph-mon[29472]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' 2024-04-14T14:18:40.076 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:18:39 smithi069 ceph-mon[29472]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T14:18:40.076 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:18:39 smithi069 ceph-mon[29472]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-14T14:18:40.077 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:18:39 smithi069 ceph-mon[29472]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' 2024-04-14T14:18:40.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:18:39 smithi122 ceph-mon[25721]: from='client.15618 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T14:18:40.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:18:39 smithi122 ceph-mon[25721]: pgmap v59: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:18:40.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:18:39 smithi122 ceph-mon[25721]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' 2024-04-14T14:18:40.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:18:39 smithi122 ceph-mon[25721]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' 2024-04-14T14:18:40.096 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:18:39 smithi122 ceph-mon[25721]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T14:18:40.096 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:18:39 smithi122 ceph-mon[25721]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-14T14:18:40.096 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:18:39 smithi122 ceph-mon[25721]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' 2024-04-14T14:18:40.096 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:18:39 smithi122 ceph-mon[25721]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-04-14T14:18:40.096 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:18:39 smithi122 ceph-mon[25721]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-14T14:18:40.096 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:18:39 smithi122 ceph-mon[25721]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T14:18:40.096 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:18:39 smithi122 ceph-mon[25721]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T14:18:40.096 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:18:39 smithi122 ceph-mon[25721]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T14:18:40.096 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:18:39 smithi122 ceph-mon[25721]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T14:18:40.096 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:18:39 smithi122 ceph-mon[25721]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T14:18:40.096 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:18:39 smithi122 ceph-mon[25721]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T14:18:40.096 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:18:39 smithi122 ceph-mon[25721]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T14:18:40.096 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:18:39 smithi122 ceph-mon[25721]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T14:18:40.096 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:18:39 smithi122 ceph-mon[25721]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T14:18:40.096 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:18:39 smithi122 ceph-mon[25721]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T14:18:40.097 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:18:39 smithi122 ceph-mon[25721]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T14:18:40.097 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:18:39 smithi122 ceph-mon[25721]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T14:18:40.097 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:18:39 smithi122 ceph-mon[25721]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T14:18:40.097 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:18:39 smithi122 ceph-mon[25721]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T14:18:40.097 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:18:39 smithi122 ceph-mon[25721]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' 2024-04-14T14:18:40.097 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:18:39 smithi122 ceph-mon[25721]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]: dispatch 2024-04-14T14:18:40.097 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:18:39 smithi122 ceph-mon[25721]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]': finished 2024-04-14T14:18:40.097 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:18:39 smithi122 ceph-mon[25721]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-14T14:18:40.097 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:18:39 smithi122 ceph-mon[25721]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon"}]': finished 2024-04-14T14:18:40.097 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:18:39 smithi122 ceph-mon[25721]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]: dispatch 2024-04-14T14:18:40.097 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:18:39 smithi122 ceph-mon[25721]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]': finished 2024-04-14T14:18:40.097 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:18:39 smithi122 ceph-mon[25721]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd"}]: dispatch 2024-04-14T14:18:40.097 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:18:39 smithi122 ceph-mon[25721]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mds"}]: dispatch 2024-04-14T14:18:40.097 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:18:39 smithi122 ceph-mon[25721]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mds"}]': finished 2024-04-14T14:18:40.097 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:18:39 smithi122 ceph-mon[25721]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw"}]: dispatch 2024-04-14T14:18:40.098 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:18:39 smithi122 ceph-mon[25721]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"}]: dispatch 2024-04-14T14:18:40.098 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:18:39 smithi122 ceph-mon[25721]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"}]': finished 2024-04-14T14:18:40.098 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:18:39 smithi122 ceph-mon[25721]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-14T14:18:40.098 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:18:39 smithi122 ceph-mon[25721]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter"}]: dispatch 2024-04-14T14:18:40.098 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:18:39 smithi122 ceph-mon[25721]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter"}]': finished 2024-04-14T14:18:40.098 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:18:39 smithi122 ceph-mon[25721]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.iscsi"}]: dispatch 2024-04-14T14:18:40.098 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:18:39 smithi122 ceph-mon[25721]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nfs"}]: dispatch 2024-04-14T14:18:40.098 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:18:39 smithi122 ceph-mon[25721]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.nfs"}]': finished 2024-04-14T14:18:40.098 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:18:39 smithi122 ceph-mon[25721]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]: dispatch 2024-04-14T14:18:40.098 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:18:39 smithi122 ceph-mon[25721]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]': finished 2024-04-14T14:18:40.098 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:18:39 smithi122 ceph-mon[25721]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-14T14:18:40.098 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:18:39 smithi122 ceph-mon[25721]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-14T14:18:40.098 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:18:39 smithi122 ceph-mon[25721]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-14T14:18:40.098 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:18:39 smithi122 ceph-mon[25721]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-14T14:18:40.099 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:18:39 smithi122 ceph-mon[25721]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-14T14:18:40.099 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:18:39 smithi122 ceph-mon[25721]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-14T14:18:40.099 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:18:39 smithi122 ceph-mon[25721]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' cmd=[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]: dispatch 2024-04-14T14:18:40.099 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:18:39 smithi122 ceph-mon[25721]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' cmd='[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]': finished 2024-04-14T14:18:40.099 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:18:39 smithi122 ceph-mon[25721]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T14:18:40.099 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:18:39 smithi122 ceph-mon[25721]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-14T14:18:40.099 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:18:39 smithi122 ceph-mon[25721]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' 2024-04-14T14:18:40.099 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:18:39 smithi122 ceph-mon[25721]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T14:18:40.099 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:18:39 smithi122 ceph-mon[25721]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-14T14:18:40.099 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:18:39 smithi122 ceph-mon[25721]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' 2024-04-14T14:18:40.925 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:18:40 smithi069 ceph-mon[24818]: pgmap v60: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 994 B/s rd, 0 op/s 2024-04-14T14:18:40.925 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:18:40 smithi069 ceph-mon[24818]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-04-14T14:18:40.925 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:18:40 smithi069 ceph-mon[24818]: Upgrade: Finalizing container_image settings 2024-04-14T14:18:40.925 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:18:40 smithi069 ceph-mon[24818]: Upgrade: Complete! 2024-04-14T14:18:40.925 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:18:40 smithi069 ceph-mon[24818]: Health check cleared: CEPHADM_FAILED_DAEMON (was: 1 failed cephadm daemon(s)) 2024-04-14T14:18:40.925 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:18:40 smithi069 ceph-mon[24818]: Cluster is now healthy 2024-04-14T14:18:40.925 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:18:40 smithi069 ceph-mon[24818]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' 2024-04-14T14:18:40.926 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:18:40 smithi069 ceph-mon[29472]: pgmap v60: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 994 B/s rd, 0 op/s 2024-04-14T14:18:40.926 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:18:40 smithi069 ceph-mon[29472]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-04-14T14:18:40.926 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:18:40 smithi069 ceph-mon[29472]: Upgrade: Finalizing container_image settings 2024-04-14T14:18:40.926 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:18:40 smithi069 ceph-mon[29472]: Upgrade: Complete! 2024-04-14T14:18:40.926 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:18:40 smithi069 ceph-mon[29472]: Health check cleared: CEPHADM_FAILED_DAEMON (was: 1 failed cephadm daemon(s)) 2024-04-14T14:18:40.926 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:18:40 smithi069 ceph-mon[29472]: Cluster is now healthy 2024-04-14T14:18:40.926 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:18:40 smithi069 ceph-mon[29472]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' 2024-04-14T14:18:41.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:18:40 smithi122 ceph-mon[25721]: pgmap v60: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 994 B/s rd, 0 op/s 2024-04-14T14:18:41.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:18:40 smithi122 ceph-mon[25721]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-04-14T14:18:41.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:18:40 smithi122 ceph-mon[25721]: Upgrade: Finalizing container_image settings 2024-04-14T14:18:41.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:18:40 smithi122 ceph-mon[25721]: Upgrade: Complete! 2024-04-14T14:18:41.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:18:40 smithi122 ceph-mon[25721]: Health check cleared: CEPHADM_FAILED_DAEMON (was: 1 failed cephadm daemon(s)) 2024-04-14T14:18:41.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:18:40 smithi122 ceph-mon[25721]: Cluster is now healthy 2024-04-14T14:18:41.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:18:40 smithi122 ceph-mon[25721]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' 2024-04-14T14:18:43.067 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:18:42 smithi069 ceph-mon[24818]: pgmap v61: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 994 B/s rd, 0 op/s 2024-04-14T14:18:43.067 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:18:42 smithi069 ceph-mon[24818]: from='client.25417 -' entity='client.iscsi.foo.smithi069.yuumtr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T14:18:43.068 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:18:42 smithi069 ceph-mon[29472]: pgmap v61: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 994 B/s rd, 0 op/s 2024-04-14T14:18:43.068 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:18:42 smithi069 ceph-mon[29472]: from='client.25417 -' entity='client.iscsi.foo.smithi069.yuumtr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T14:18:43.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:18:42 smithi122 ceph-mon[25721]: pgmap v61: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 994 B/s rd, 0 op/s 2024-04-14T14:18:43.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:18:42 smithi122 ceph-mon[25721]: from='client.25417 -' entity='client.iscsi.foo.smithi069.yuumtr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T14:18:45.067 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:18:44 smithi069 ceph-mon[24818]: pgmap v62: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 994 B/s rd, 0 op/s 2024-04-14T14:18:45.067 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:18:44 smithi069 ceph-mon[29472]: pgmap v62: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 994 B/s rd, 0 op/s 2024-04-14T14:18:45.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:18:44 smithi122 ceph-mon[25721]: pgmap v62: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 994 B/s rd, 0 op/s 2024-04-14T14:18:46.317 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:18:45 smithi069 ceph-mon[24818]: pgmap v63: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 994 B/s rd, 0 op/s 2024-04-14T14:18:46.317 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:18:45 smithi069 ceph-mon[24818]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' 2024-04-14T14:18:46.317 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:18:45 smithi069 ceph-mon[24818]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T14:18:46.317 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:18:45 smithi069 ceph-mon[29472]: pgmap v63: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 994 B/s rd, 0 op/s 2024-04-14T14:18:46.317 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:18:45 smithi069 ceph-mon[29472]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' 2024-04-14T14:18:46.318 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:18:45 smithi069 ceph-mon[29472]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T14:18:46.345 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:18:45 smithi122 ceph-mon[25721]: pgmap v63: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 994 B/s rd, 0 op/s 2024-04-14T14:18:46.345 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:18:45 smithi122 ceph-mon[25721]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' 2024-04-14T14:18:46.345 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:18:45 smithi122 ceph-mon[25721]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T14:18:46.817 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 14 14:18:46 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-y[63038]: ::ffff:172.21.15.122 - - [14/Apr/2024:14:18:46] "GET /metrics HTTP/1.1" 200 37985 "" "Prometheus/2.43.0" 2024-04-14T14:18:48.317 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:18:47 smithi069 ceph-mon[24818]: pgmap v64: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 994 B/s rd, 0 op/s 2024-04-14T14:18:48.317 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:18:47 smithi069 ceph-mon[29472]: pgmap v64: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 994 B/s rd, 0 op/s 2024-04-14T14:18:48.345 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:18:47 smithi122 ceph-mon[25721]: pgmap v64: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 994 B/s rd, 0 op/s 2024-04-14T14:18:50.317 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:18:49 smithi069 ceph-mon[24818]: pgmap v65: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 994 B/s rd, 0 op/s 2024-04-14T14:18:50.317 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:18:49 smithi069 ceph-mon[29472]: pgmap v65: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 994 B/s rd, 0 op/s 2024-04-14T14:18:50.345 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:18:49 smithi122 ceph-mon[25721]: pgmap v65: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 994 B/s rd, 0 op/s 2024-04-14T14:18:53.067 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:18:52 smithi069 ceph-mon[24818]: pgmap v66: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:18:53.067 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:18:52 smithi069 ceph-mon[24818]: from='client.25417 -' entity='client.iscsi.foo.smithi069.yuumtr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T14:18:53.067 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:18:52 smithi069 ceph-mon[29472]: pgmap v66: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:18:53.068 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:18:52 smithi069 ceph-mon[29472]: from='client.25417 -' entity='client.iscsi.foo.smithi069.yuumtr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T14:18:53.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:18:52 smithi122 ceph-mon[25721]: pgmap v66: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:18:53.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:18:52 smithi122 ceph-mon[25721]: from='client.25417 -' entity='client.iscsi.foo.smithi069.yuumtr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T14:18:55.067 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:18:54 smithi069 ceph-mon[24818]: pgmap v67: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:18:55.067 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:18:54 smithi069 ceph-mon[29472]: pgmap v67: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:18:55.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:18:54 smithi122 ceph-mon[25721]: pgmap v67: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:18:56.817 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 14 14:18:56 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-y[63038]: ::ffff:172.21.15.122 - - [14/Apr/2024:14:18:56] "GET /metrics HTTP/1.1" 200 37985 "" "Prometheus/2.43.0" 2024-04-14T14:18:56.817 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:18:56 smithi069 ceph-mon[24818]: pgmap v68: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:18:56.818 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:18:56 smithi069 ceph-mon[29472]: pgmap v68: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:18:56.897 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:18:56 smithi122 ceph-mon[25721]: pgmap v68: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:18:58.976 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:18:58 smithi069 ceph-mon[24818]: pgmap v69: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:18:59.010 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:18:58 smithi069 ceph-mon[29472]: pgmap v69: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:18:59.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:18:58 smithi122 ceph-mon[25721]: pgmap v69: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:19:00.947 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:19:00 smithi069 ceph-mon[24818]: pgmap v70: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:19:00.947 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:19:00 smithi069 ceph-mon[24818]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T14:19:00.947 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:19:00 smithi069 ceph-mon[29472]: pgmap v70: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:19:00.947 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:19:00 smithi069 ceph-mon[29472]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T14:19:01.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:19:00 smithi122 ceph-mon[25721]: pgmap v70: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:19:01.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:19:00 smithi122 ceph-mon[25721]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T14:19:03.067 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:19:02 smithi069 ceph-mon[24818]: pgmap v71: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:19:03.067 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:19:02 smithi069 ceph-mon[24818]: from='client.25417 -' entity='client.iscsi.foo.smithi069.yuumtr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T14:19:03.067 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:19:02 smithi069 ceph-mon[29472]: pgmap v71: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:19:03.067 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:19:02 smithi069 ceph-mon[29472]: from='client.25417 -' entity='client.iscsi.foo.smithi069.yuumtr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T14:19:03.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:19:02 smithi122 ceph-mon[25721]: pgmap v71: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:19:03.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:19:02 smithi122 ceph-mon[25721]: from='client.25417 -' entity='client.iscsi.foo.smithi069.yuumtr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T14:19:05.067 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:19:04 smithi069 ceph-mon[24818]: pgmap v72: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:19:05.067 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:19:04 smithi069 ceph-mon[29472]: pgmap v72: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:19:05.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:19:04 smithi122 ceph-mon[25721]: pgmap v72: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:19:06.817 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:19:06 smithi069 ceph-mon[24818]: pgmap v73: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:19:06.818 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 14 14:19:06 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-y[63038]: ::ffff:172.21.15.122 - - [14/Apr/2024:14:19:06] "GET /metrics HTTP/1.1" 200 37987 "" "Prometheus/2.43.0" 2024-04-14T14:19:06.818 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:19:06 smithi069 ceph-mon[29472]: pgmap v73: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:19:07.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:19:06 smithi122 ceph-mon[25721]: pgmap v73: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:19:08.779 DEBUG:teuthology.orchestra.run.smithi069:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid dc326fc2-fa65-11ee-bc8e-c7b262605968 -e sha1=a971fd5f5719038c70b817bafacd6d5400b1d888 -- bash -c 'ceph versions | jq -e '"'"'.mgr | length == 1'"'"'' 2024-04-14T14:19:09.067 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:19:08 smithi069 ceph-mon[24818]: pgmap v74: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:19:09.067 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:19:08 smithi069 ceph-mon[29472]: pgmap v74: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:19:09.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:19:08 smithi122 ceph-mon[25721]: pgmap v74: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:19:09.766 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:19:09 smithi069 ceph-mon[24818]: from='client.25525 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T14:19:09.766 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:19:09 smithi069 ceph-mon[29472]: from='client.25525 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T14:19:10.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:19:09 smithi122 ceph-mon[25721]: from='client.25525 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T14:19:10.862 INFO:teuthology.orchestra.run.smithi069.stdout:true 2024-04-14T14:19:10.957 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:19:10 smithi069 ceph-mon[24818]: pgmap v75: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:19:10.958 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:19:10 smithi069 ceph-mon[29472]: pgmap v75: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:19:11.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:19:10 smithi122 ceph-mon[25721]: pgmap v75: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:19:11.513 DEBUG:teuthology.orchestra.run.smithi069:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid dc326fc2-fa65-11ee-bc8e-c7b262605968 -e sha1=a971fd5f5719038c70b817bafacd6d5400b1d888 -- bash -c 'ceph versions | jq -e '"'"'.mgr | keys'"'"' | grep $sha1' 2024-04-14T14:19:11.929 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:19:11 smithi069 ceph-mon[24818]: from='client.? 172.21.15.69:0/2582782853' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T14:19:11.930 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:19:11 smithi069 ceph-mon[29472]: from='client.? 172.21.15.69:0/2582782853' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T14:19:12.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:19:11 smithi122 ceph-mon[25721]: from='client.? 172.21.15.69:0/2582782853' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T14:19:12.690 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:19:12 smithi069 ceph-mon[24818]: pgmap v76: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:19:12.690 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:19:12 smithi069 ceph-mon[24818]: from='client.25417 -' entity='client.iscsi.foo.smithi069.yuumtr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T14:19:12.690 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:19:12 smithi069 ceph-mon[29472]: pgmap v76: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:19:12.690 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:19:12 smithi069 ceph-mon[29472]: from='client.25417 -' entity='client.iscsi.foo.smithi069.yuumtr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T14:19:13.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:19:12 smithi122 ceph-mon[25721]: pgmap v76: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:19:13.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:19:12 smithi122 ceph-mon[25721]: from='client.25417 -' entity='client.iscsi.foo.smithi069.yuumtr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T14:19:13.557 INFO:teuthology.orchestra.run.smithi069.stdout: "ceph version 19.0.0-2778-ga971fd5f (a971fd5f5719038c70b817bafacd6d5400b1d888) squid (dev)" 2024-04-14T14:19:13.817 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:19:13 smithi069 ceph-mon[24818]: from='client.? 172.21.15.69:0/3683338172' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T14:19:13.818 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:19:13 smithi069 ceph-mon[29472]: from='client.? 172.21.15.69:0/3683338172' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T14:19:14.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:19:13 smithi122 ceph-mon[25721]: from='client.? 172.21.15.69:0/3683338172' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T14:19:14.212 DEBUG:teuthology.orchestra.run.smithi069:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid dc326fc2-fa65-11ee-bc8e-c7b262605968 -e sha1=a971fd5f5719038c70b817bafacd6d5400b1d888 -- bash -c 'ceph versions | jq -e '"'"'.overall | length == 2'"'"'' 2024-04-14T14:19:14.816 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:19:14 smithi069 ceph-mon[24818]: pgmap v77: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:19:14.817 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:19:14 smithi069 ceph-mon[29472]: pgmap v77: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:19:15.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:19:14 smithi122 ceph-mon[25721]: pgmap v77: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:19:15.817 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:19:15 smithi069 ceph-mon[24818]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T14:19:15.817 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:19:15 smithi069 ceph-mon[29472]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T14:19:16.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:19:15 smithi122 ceph-mon[25721]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T14:19:16.483 INFO:teuthology.orchestra.run.smithi069.stdout:true 2024-04-14T14:19:16.785 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:19:16 smithi069 ceph-mon[24818]: pgmap v78: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:19:16.785 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:19:16 smithi069 ceph-mon[24818]: from='client.? 172.21.15.69:0/3115321642' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T14:19:16.785 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:19:16 smithi069 ceph-mon[29472]: pgmap v78: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:19:16.786 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:19:16 smithi069 ceph-mon[29472]: from='client.? 172.21.15.69:0/3115321642' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T14:19:16.786 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 14 14:19:16 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-y[63038]: ::ffff:172.21.15.122 - - [14/Apr/2024:14:19:16] "GET /metrics HTTP/1.1" 200 37996 "" "Prometheus/2.43.0" 2024-04-14T14:19:17.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:19:16 smithi122 ceph-mon[25721]: pgmap v78: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:19:17.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:19:16 smithi122 ceph-mon[25721]: from='client.? 172.21.15.69:0/3115321642' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T14:19:17.109 DEBUG:teuthology.orchestra.run.smithi069:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid dc326fc2-fa65-11ee-bc8e-c7b262605968 -e sha1=a971fd5f5719038c70b817bafacd6d5400b1d888 -- bash -c 'ceph orch upgrade check quay.ceph.io/ceph-ci/ceph:$sha1 | jq -e '"'"'.up_to_date | length == 2'"'"'' 2024-04-14T14:19:19.067 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:19:18 smithi069 ceph-mon[24818]: pgmap v79: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:19:19.067 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:19:18 smithi069 ceph-mon[29472]: pgmap v79: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:19:19.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:19:18 smithi122 ceph-mon[25721]: pgmap v79: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:19:20.740 INFO:teuthology.orchestra.run.smithi069.stdout:true 2024-04-14T14:19:20.969 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:19:20 smithi069 ceph-mon[24818]: pgmap v80: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:19:20.969 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:19:20 smithi069 ceph-mon[24818]: from='client.15648 -' entity='client.admin' cmd=[{"prefix": "orch upgrade check", "image": "quay.ceph.io/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T14:19:20.970 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:19:20 smithi069 ceph-mon[29472]: pgmap v80: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:19:20.970 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:19:20 smithi069 ceph-mon[29472]: from='client.15648 -' entity='client.admin' cmd=[{"prefix": "orch upgrade check", "image": "quay.ceph.io/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T14:19:21.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:19:20 smithi122 ceph-mon[25721]: pgmap v80: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:19:21.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:19:20 smithi122 ceph-mon[25721]: from='client.15648 -' entity='client.admin' cmd=[{"prefix": "orch upgrade check", "image": "quay.ceph.io/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T14:19:21.393 DEBUG:teuthology.orchestra.run.smithi069:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid dc326fc2-fa65-11ee-bc8e-c7b262605968 -e sha1=a971fd5f5719038c70b817bafacd6d5400b1d888 -- bash -c 'ceph orch upgrade status' 2024-04-14T14:19:22.932 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:19:22 smithi069 ceph-mon[24818]: pgmap v81: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:19:22.933 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:19:22 smithi069 ceph-mon[24818]: from='client.25417 -' entity='client.iscsi.foo.smithi069.yuumtr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T14:19:22.933 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:19:22 smithi069 ceph-mon[29472]: pgmap v81: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:19:22.933 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:19:22 smithi069 ceph-mon[29472]: from='client.25417 -' entity='client.iscsi.foo.smithi069.yuumtr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T14:19:23.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:19:22 smithi122 ceph-mon[25721]: pgmap v81: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:19:23.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:19:22 smithi122 ceph-mon[25721]: from='client.25417 -' entity='client.iscsi.foo.smithi069.yuumtr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T14:19:23.511 INFO:teuthology.orchestra.run.smithi069.stdout:{ 2024-04-14T14:19:23.511 INFO:teuthology.orchestra.run.smithi069.stdout: "target_image": null, 2024-04-14T14:19:23.511 INFO:teuthology.orchestra.run.smithi069.stdout: "in_progress": false, 2024-04-14T14:19:23.511 INFO:teuthology.orchestra.run.smithi069.stdout: "which": "", 2024-04-14T14:19:23.511 INFO:teuthology.orchestra.run.smithi069.stdout: "services_complete": [], 2024-04-14T14:19:23.512 INFO:teuthology.orchestra.run.smithi069.stdout: "progress": null, 2024-04-14T14:19:23.512 INFO:teuthology.orchestra.run.smithi069.stdout: "message": "", 2024-04-14T14:19:23.512 INFO:teuthology.orchestra.run.smithi069.stdout: "is_paused": false 2024-04-14T14:19:23.512 INFO:teuthology.orchestra.run.smithi069.stdout:} 2024-04-14T14:19:24.171 DEBUG:teuthology.orchestra.run.smithi069:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid dc326fc2-fa65-11ee-bc8e-c7b262605968 -e sha1=a971fd5f5719038c70b817bafacd6d5400b1d888 -- bash -c 'ceph health detail' 2024-04-14T14:19:24.742 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:19:24 smithi069 ceph-mon[24818]: pgmap v82: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:19:24.742 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:19:24 smithi069 ceph-mon[24818]: from='client.15654 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T14:19:24.742 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:19:24 smithi069 ceph-mon[29472]: pgmap v82: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:19:24.742 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:19:24 smithi069 ceph-mon[29472]: from='client.15654 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T14:19:25.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:19:24 smithi122 ceph-mon[25721]: pgmap v82: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:19:25.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:19:24 smithi122 ceph-mon[25721]: from='client.15654 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T14:19:26.265 INFO:teuthology.orchestra.run.smithi069.stdout:HEALTH_OK 2024-04-14T14:19:26.567 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 14 14:19:26 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-y[63038]: ::ffff:172.21.15.122 - - [14/Apr/2024:14:19:26] "GET /metrics HTTP/1.1" 200 37996 "" "Prometheus/2.43.0" 2024-04-14T14:19:26.886 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:19:26 smithi069 ceph-mon[24818]: pgmap v83: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:19:26.887 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:19:26 smithi069 ceph-mon[24818]: from='client.? 172.21.15.69:0/202872734' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-14T14:19:26.887 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:19:26 smithi069 ceph-mon[29472]: pgmap v83: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:19:26.887 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:19:26 smithi069 ceph-mon[29472]: from='client.? 172.21.15.69:0/202872734' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-14T14:19:26.913 DEBUG:teuthology.orchestra.run.smithi069:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid dc326fc2-fa65-11ee-bc8e-c7b262605968 -e sha1=a971fd5f5719038c70b817bafacd6d5400b1d888 -- bash -c 'ceph orch upgrade start --image quay.ceph.io/ceph-ci/ceph:$sha1 --daemon-types mon --hosts $(ceph orch ps | grep mgr.x | awk '"'"'{print $2}'"'"')' 2024-04-14T14:19:27.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:19:26 smithi122 ceph-mon[25721]: pgmap v83: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:19:27.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:19:26 smithi122 ceph-mon[25721]: from='client.? 172.21.15.69:0/202872734' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-14T14:19:28.817 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:19:28 smithi069 ceph-mon[24818]: pgmap v84: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:19:28.817 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:19:28 smithi069 ceph-mon[29472]: pgmap v84: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:19:29.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:19:28 smithi122 ceph-mon[25721]: pgmap v84: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:19:30.770 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:19:30 smithi069 ceph-mon[24818]: pgmap v85: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:19:30.771 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:19:30 smithi069 ceph-mon[24818]: from='client.15666 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T14:19:30.771 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:19:30 smithi069 ceph-mon[24818]: from='client.15672 -' entity='client.admin' cmd=[{"prefix": "orch upgrade start", "image": "quay.ceph.io/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888", "daemon_types": "mon", "hosts": "smithi122", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T14:19:30.771 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:19:30 smithi069 ceph-mon[24818]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T14:19:30.771 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:19:30 smithi069 ceph-mon[29472]: pgmap v85: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:19:30.771 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:19:30 smithi069 ceph-mon[29472]: from='client.15666 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T14:19:30.771 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:19:30 smithi069 ceph-mon[29472]: from='client.15672 -' entity='client.admin' cmd=[{"prefix": "orch upgrade start", "image": "quay.ceph.io/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888", "daemon_types": "mon", "hosts": "smithi122", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T14:19:30.771 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:19:30 smithi069 ceph-mon[29472]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T14:19:30.846 INFO:teuthology.orchestra.run.smithi069.stdout:Initiating upgrade to quay.ceph.io/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888 2024-04-14T14:19:31.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:19:30 smithi122 ceph-mon[25721]: pgmap v85: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:19:31.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:19:30 smithi122 ceph-mon[25721]: from='client.15666 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T14:19:31.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:19:30 smithi122 ceph-mon[25721]: from='client.15672 -' entity='client.admin' cmd=[{"prefix": "orch upgrade start", "image": "quay.ceph.io/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888", "daemon_types": "mon", "hosts": "smithi122", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T14:19:31.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:19:30 smithi122 ceph-mon[25721]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T14:19:31.398 DEBUG:teuthology.orchestra.run.smithi069:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid dc326fc2-fa65-11ee-bc8e-c7b262605968 -e sha1=a971fd5f5719038c70b817bafacd6d5400b1d888 -- bash -c 'while ceph orch upgrade status | jq '"'"'.in_progress'"'"' | grep true && ! ceph orch upgrade status | jq '"'"'.message'"'"' | grep Error ; do ceph orch ps ; ceph versions ; ceph orch upgrade status ; sleep 30 ; done' 2024-04-14T14:19:31.969 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:19:31 smithi069 ceph-mon[29472]: pgmap v86: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:19:31.969 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:19:31 smithi069 ceph-mon[29472]: Upgrade: Started with target quay.ceph.io/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888 2024-04-14T14:19:31.969 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:19:31 smithi069 ceph-mon[29472]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' 2024-04-14T14:19:31.969 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:19:31 smithi069 ceph-mon[29472]: from='client.25417 -' entity='client.iscsi.foo.smithi069.yuumtr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T14:19:31.969 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:19:31 smithi069 ceph-mon[29472]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T14:19:31.970 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:19:31 smithi069 ceph-mon[29472]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-14T14:19:31.970 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:19:31 smithi069 ceph-mon[29472]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' 2024-04-14T14:19:31.970 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:19:31 smithi069 ceph-mon[29472]: Upgrade: First pull of quay.ceph.io/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888 2024-04-14T14:19:31.970 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:19:31 smithi069 ceph-mon[24818]: pgmap v86: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:19:31.970 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:19:31 smithi069 ceph-mon[24818]: Upgrade: Started with target quay.ceph.io/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888 2024-04-14T14:19:31.970 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:19:31 smithi069 ceph-mon[24818]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' 2024-04-14T14:19:31.970 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:19:31 smithi069 ceph-mon[24818]: from='client.25417 -' entity='client.iscsi.foo.smithi069.yuumtr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T14:19:31.970 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:19:31 smithi069 ceph-mon[24818]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T14:19:31.970 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:19:31 smithi069 ceph-mon[24818]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-14T14:19:31.970 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:19:31 smithi069 ceph-mon[24818]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' 2024-04-14T14:19:31.970 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:19:31 smithi069 ceph-mon[24818]: Upgrade: First pull of quay.ceph.io/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888 2024-04-14T14:19:32.345 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:19:31 smithi122 ceph-mon[25721]: pgmap v86: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:19:32.345 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:19:31 smithi122 ceph-mon[25721]: Upgrade: Started with target quay.ceph.io/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888 2024-04-14T14:19:32.345 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:19:31 smithi122 ceph-mon[25721]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' 2024-04-14T14:19:32.345 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:19:31 smithi122 ceph-mon[25721]: from='client.25417 -' entity='client.iscsi.foo.smithi069.yuumtr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T14:19:32.345 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:19:31 smithi122 ceph-mon[25721]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T14:19:32.345 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:19:31 smithi122 ceph-mon[25721]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-14T14:19:32.345 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:19:31 smithi122 ceph-mon[25721]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' 2024-04-14T14:19:32.345 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:19:31 smithi122 ceph-mon[25721]: Upgrade: First pull of quay.ceph.io/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888 2024-04-14T14:19:33.425 INFO:teuthology.orchestra.run.smithi069.stdout:true 2024-04-14T14:19:33.967 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:19:33 smithi069 ceph-mon[24818]: pgmap v87: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:19:33.968 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:19:33 smithi069 ceph-mon[24818]: from='client.25576 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T14:19:33.968 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:19:33 smithi069 ceph-mon[29472]: pgmap v87: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:19:33.968 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:19:33 smithi069 ceph-mon[29472]: from='client.25576 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T14:19:34.103 INFO:teuthology.orchestra.run.smithi069.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-14T14:19:34.103 INFO:teuthology.orchestra.run.smithi069.stdout:alertmanager.a smithi069 *:9093,9094 running (79s) 74s ago 22m 16.5M - 0.25.0 c8568f914cd2 7524f106e7bd 2024-04-14T14:19:34.103 INFO:teuthology.orchestra.run.smithi069.stdout:grafana.a smithi122 *:3000 running (64s) 59s ago 22m 53.1M - 9.4.12 2bacad6d85d8 5685178e3eba 2024-04-14T14:19:34.103 INFO:teuthology.orchestra.run.smithi069.stdout:iscsi.foo.smithi069.yuumtr smithi069 running (2m) 74s ago 22m 44.5M - 3.5 e1d6a67b021e 38cc81ea8cfa 2024-04-14T14:19:34.103 INFO:teuthology.orchestra.run.smithi069.stdout:mgr.x smithi122 *:8443,9283,8765 running (2m) 59s ago 27m 450M - 19.0.0-2778-ga971fd5f b96c509eef84 e29977590b43 2024-04-14T14:19:34.103 INFO:teuthology.orchestra.run.smithi069.stdout:mgr.y smithi069 *:8443,9283,8765 running (11m) 74s ago 29m 504M - 19.0.0-2778-ga971fd5f b96c509eef84 7af3d29d0e5f 2024-04-14T14:19:34.103 INFO:teuthology.orchestra.run.smithi069.stdout:mon.a smithi069 running (29m) 74s ago 29m 78.2M 2048M 17.2.0 e1d6a67b021e d3c553f91913 2024-04-14T14:19:34.103 INFO:teuthology.orchestra.run.smithi069.stdout:mon.b smithi122 running (27m) 59s ago 27m 55.4M 2048M 17.2.0 e1d6a67b021e 3b766bfce4be 2024-04-14T14:19:34.103 INFO:teuthology.orchestra.run.smithi069.stdout:mon.c smithi069 running (27m) 74s ago 27m 57.4M 2048M 17.2.0 e1d6a67b021e 23c078b9d1e5 2024-04-14T14:19:34.103 INFO:teuthology.orchestra.run.smithi069.stdout:node-exporter.a smithi069 *:9100 running (118s) 74s ago 23m 8258k - 1.5.0 0da6a335fe13 0de0eb2171b9 2024-04-14T14:19:34.103 INFO:teuthology.orchestra.run.smithi069.stdout:node-exporter.b smithi122 *:9100 running (110s) 59s ago 23m 16.4M - 1.5.0 0da6a335fe13 67877bf02d8e 2024-04-14T14:19:34.104 INFO:teuthology.orchestra.run.smithi069.stdout:osd.0 smithi069 running (27m) 74s ago 27m 56.4M 2992M 17.2.0 e1d6a67b021e 46654654b6b6 2024-04-14T14:19:34.104 INFO:teuthology.orchestra.run.smithi069.stdout:osd.1 smithi069 running (26m) 74s ago 26m 54.5M 2992M 17.2.0 e1d6a67b021e cdba54a59609 2024-04-14T14:19:34.104 INFO:teuthology.orchestra.run.smithi069.stdout:osd.2 smithi069 running (26m) 74s ago 26m 50.9M 2992M 17.2.0 e1d6a67b021e bc37a1a9cba0 2024-04-14T14:19:34.104 INFO:teuthology.orchestra.run.smithi069.stdout:osd.3 smithi069 running (26m) 74s ago 26m 55.2M 2992M 17.2.0 e1d6a67b021e 0925ba3da801 2024-04-14T14:19:34.104 INFO:teuthology.orchestra.run.smithi069.stdout:osd.4 smithi122 running (25m) 59s ago 25m 56.0M 3248M 17.2.0 e1d6a67b021e c00975caf771 2024-04-14T14:19:34.104 INFO:teuthology.orchestra.run.smithi069.stdout:osd.5 smithi122 running (25m) 59s ago 25m 55.1M 3248M 17.2.0 e1d6a67b021e 5f52d217ceb8 2024-04-14T14:19:34.104 INFO:teuthology.orchestra.run.smithi069.stdout:osd.6 smithi122 running (24m) 59s ago 24m 51.8M 3248M 17.2.0 e1d6a67b021e a2acfec61475 2024-04-14T14:19:34.104 INFO:teuthology.orchestra.run.smithi069.stdout:osd.7 smithi122 running (24m) 59s ago 24m 56.8M 3248M 17.2.0 e1d6a67b021e 62411cd06801 2024-04-14T14:19:34.104 INFO:teuthology.orchestra.run.smithi069.stdout:prometheus.a smithi122 *:9095 running (94s) 59s ago 23m 55.4M - 2.43.0 a07b618ecd1d 5c1f41d3bbe6 2024-04-14T14:19:34.104 INFO:teuthology.orchestra.run.smithi069.stdout:rgw.foo.smithi069.wesedz smithi069 *:8000 running (22m) 74s ago 22m 77.8M - 17.2.0 e1d6a67b021e 16de491440f3 2024-04-14T14:19:34.104 INFO:teuthology.orchestra.run.smithi069.stdout:rgw.foo.smithi122.jiobyv smithi122 *:8000 running (22m) 59s ago 22m 74.9M - 17.2.0 e1d6a67b021e 6d16de445dba 2024-04-14T14:19:34.345 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:19:33 smithi122 ceph-mon[25721]: pgmap v87: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:19:34.345 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:19:33 smithi122 ceph-mon[25721]: from='client.25576 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T14:19:34.513 INFO:teuthology.orchestra.run.smithi069.stdout:{ 2024-04-14T14:19:34.513 INFO:teuthology.orchestra.run.smithi069.stdout: "mon": { 2024-04-14T14:19:34.513 INFO:teuthology.orchestra.run.smithi069.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 3 2024-04-14T14:19:34.513 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-14T14:19:34.513 INFO:teuthology.orchestra.run.smithi069.stdout: "mgr": { 2024-04-14T14:19:34.513 INFO:teuthology.orchestra.run.smithi069.stdout: "ceph version 19.0.0-2778-ga971fd5f (a971fd5f5719038c70b817bafacd6d5400b1d888) squid (dev)": 2 2024-04-14T14:19:34.513 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-14T14:19:34.513 INFO:teuthology.orchestra.run.smithi069.stdout: "osd": { 2024-04-14T14:19:34.513 INFO:teuthology.orchestra.run.smithi069.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 8 2024-04-14T14:19:34.513 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-14T14:19:34.513 INFO:teuthology.orchestra.run.smithi069.stdout: "mds": {}, 2024-04-14T14:19:34.514 INFO:teuthology.orchestra.run.smithi069.stdout: "rgw": { 2024-04-14T14:19:34.514 INFO:teuthology.orchestra.run.smithi069.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-04-14T14:19:34.514 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-14T14:19:34.514 INFO:teuthology.orchestra.run.smithi069.stdout: "overall": { 2024-04-14T14:19:34.514 INFO:teuthology.orchestra.run.smithi069.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 13, 2024-04-14T14:19:34.514 INFO:teuthology.orchestra.run.smithi069.stdout: "ceph version 19.0.0-2778-ga971fd5f (a971fd5f5719038c70b817bafacd6d5400b1d888) squid (dev)": 2 2024-04-14T14:19:34.514 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-14T14:19:34.514 INFO:teuthology.orchestra.run.smithi069.stdout:} 2024-04-14T14:19:34.844 INFO:teuthology.orchestra.run.smithi069.stdout:{ 2024-04-14T14:19:34.844 INFO:teuthology.orchestra.run.smithi069.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888", 2024-04-14T14:19:34.844 INFO:teuthology.orchestra.run.smithi069.stdout: "in_progress": true, 2024-04-14T14:19:34.844 INFO:teuthology.orchestra.run.smithi069.stdout: "which": "Upgrading daemons of type(s) mon on host(s) smithi122", 2024-04-14T14:19:34.844 INFO:teuthology.orchestra.run.smithi069.stdout: "services_complete": [], 2024-04-14T14:19:34.844 INFO:teuthology.orchestra.run.smithi069.stdout: "progress": "0/1 daemons upgraded", 2024-04-14T14:19:34.844 INFO:teuthology.orchestra.run.smithi069.stdout: "message": "Currently upgrading mon daemons", 2024-04-14T14:19:34.844 INFO:teuthology.orchestra.run.smithi069.stdout: "is_paused": false 2024-04-14T14:19:34.844 INFO:teuthology.orchestra.run.smithi069.stdout:} 2024-04-14T14:19:35.595 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:19:35 smithi122 ceph-mon[25721]: from='client.25582 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T14:19:35.596 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:19:35 smithi122 ceph-mon[25721]: from='client.25588 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T14:19:35.596 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:19:35 smithi122 ceph-mon[25721]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' 2024-04-14T14:19:35.596 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:19:35 smithi122 ceph-mon[25721]: Upgrade: Target is version 19.0.0-2778-ga971fd5f (squid) 2024-04-14T14:19:35.596 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:19:35 smithi122 ceph-mon[25721]: Upgrade: Target container is quay.ceph.io/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888, digests ['quay.ceph.io/ceph-ci/ceph@sha256:02f0a191d4d693952654c64f93cc9c0152602c0915251b3c94c2c4e32855c608'] 2024-04-14T14:19:35.596 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:19:35 smithi122 ceph-mon[25721]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-14T14:19:35.596 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:19:35 smithi122 ceph-mon[25721]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T14:19:35.596 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:19:35 smithi122 ceph-mon[25721]: Upgrade: Setting container_image for all mgr 2024-04-14T14:19:35.596 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:19:35 smithi122 ceph-mon[25721]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' 2024-04-14T14:19:35.596 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:19:35 smithi122 ceph-mon[25721]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' cmd=[{"prefix": "quorum_status"}]: dispatch 2024-04-14T14:19:35.596 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:19:35 smithi122 ceph-mon[25721]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' cmd=[{"prefix": "mon ok-to-stop", "ids": ["b"]}]: dispatch 2024-04-14T14:19:35.596 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:19:35 smithi122 ceph-mon[25721]: Upgrade: It appears safe to stop mon.b 2024-04-14T14:19:35.596 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:19:35 smithi122 ceph-mon[25721]: from='client.? 172.21.15.69:0/36691256' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T14:19:35.817 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:19:35 smithi069 ceph-mon[24818]: from='client.25582 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T14:19:35.817 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:19:35 smithi069 ceph-mon[24818]: from='client.25588 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T14:19:35.817 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:19:35 smithi069 ceph-mon[24818]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' 2024-04-14T14:19:35.817 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:19:35 smithi069 ceph-mon[24818]: Upgrade: Target is version 19.0.0-2778-ga971fd5f (squid) 2024-04-14T14:19:35.818 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:19:35 smithi069 ceph-mon[24818]: Upgrade: Target container is quay.ceph.io/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888, digests ['quay.ceph.io/ceph-ci/ceph@sha256:02f0a191d4d693952654c64f93cc9c0152602c0915251b3c94c2c4e32855c608'] 2024-04-14T14:19:35.818 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:19:35 smithi069 ceph-mon[24818]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-14T14:19:35.818 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:19:35 smithi069 ceph-mon[24818]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T14:19:35.818 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:19:35 smithi069 ceph-mon[24818]: Upgrade: Setting container_image for all mgr 2024-04-14T14:19:35.818 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:19:35 smithi069 ceph-mon[24818]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' 2024-04-14T14:19:35.818 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:19:35 smithi069 ceph-mon[24818]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' cmd=[{"prefix": "quorum_status"}]: dispatch 2024-04-14T14:19:35.818 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:19:35 smithi069 ceph-mon[24818]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' cmd=[{"prefix": "mon ok-to-stop", "ids": ["b"]}]: dispatch 2024-04-14T14:19:35.818 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:19:35 smithi069 ceph-mon[24818]: Upgrade: It appears safe to stop mon.b 2024-04-14T14:19:35.818 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:19:35 smithi069 ceph-mon[24818]: from='client.? 172.21.15.69:0/36691256' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T14:19:35.818 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:19:35 smithi069 ceph-mon[29472]: from='client.25582 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T14:19:35.819 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:19:35 smithi069 ceph-mon[29472]: from='client.25588 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T14:19:35.819 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:19:35 smithi069 ceph-mon[29472]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' 2024-04-14T14:19:35.819 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:19:35 smithi069 ceph-mon[29472]: Upgrade: Target is version 19.0.0-2778-ga971fd5f (squid) 2024-04-14T14:19:35.819 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:19:35 smithi069 ceph-mon[29472]: Upgrade: Target container is quay.ceph.io/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888, digests ['quay.ceph.io/ceph-ci/ceph@sha256:02f0a191d4d693952654c64f93cc9c0152602c0915251b3c94c2c4e32855c608'] 2024-04-14T14:19:35.819 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:19:35 smithi069 ceph-mon[29472]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-14T14:19:35.819 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:19:35 smithi069 ceph-mon[29472]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T14:19:35.819 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:19:35 smithi069 ceph-mon[29472]: Upgrade: Setting container_image for all mgr 2024-04-14T14:19:35.819 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:19:35 smithi069 ceph-mon[29472]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' 2024-04-14T14:19:35.819 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:19:35 smithi069 ceph-mon[29472]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' cmd=[{"prefix": "quorum_status"}]: dispatch 2024-04-14T14:19:35.819 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:19:35 smithi069 ceph-mon[29472]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' cmd=[{"prefix": "mon ok-to-stop", "ids": ["b"]}]: dispatch 2024-04-14T14:19:35.819 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:19:35 smithi069 ceph-mon[29472]: Upgrade: It appears safe to stop mon.b 2024-04-14T14:19:35.819 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:19:35 smithi069 ceph-mon[29472]: from='client.? 172.21.15.69:0/36691256' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T14:19:36.817 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 14 14:19:36 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-y[63038]: ::ffff:172.21.15.122 - - [14/Apr/2024:14:19:36] "GET /metrics HTTP/1.1" 200 37995 "" "Prometheus/2.43.0" 2024-04-14T14:19:36.817 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:19:36 smithi069 ceph-mon[24818]: pgmap v88: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:19:36.817 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:19:36 smithi069 ceph-mon[24818]: from='client.15702 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T14:19:36.818 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:19:36 smithi069 ceph-mon[24818]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' 2024-04-14T14:19:36.818 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:19:36 smithi069 ceph-mon[24818]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-14T14:19:36.818 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:19:36 smithi069 ceph-mon[24818]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-14T14:19:36.818 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:19:36 smithi069 ceph-mon[24818]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T14:19:36.818 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:19:36 smithi069 ceph-mon[29472]: pgmap v88: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:19:36.818 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:19:36 smithi069 ceph-mon[29472]: from='client.15702 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T14:19:36.818 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:19:36 smithi069 ceph-mon[29472]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' 2024-04-14T14:19:36.818 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:19:36 smithi069 ceph-mon[29472]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-14T14:19:36.818 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:19:36 smithi069 ceph-mon[29472]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-14T14:19:36.818 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:19:36 smithi069 ceph-mon[29472]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T14:19:36.858 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:19:36 smithi122 ceph-mon[25721]: pgmap v88: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:19:36.858 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:19:36 smithi122 ceph-mon[25721]: from='client.15702 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T14:19:36.858 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:19:36 smithi122 ceph-mon[25721]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' 2024-04-14T14:19:36.858 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:19:36 smithi122 ceph-mon[25721]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-14T14:19:36.859 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:19:36 smithi122 ceph-mon[25721]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-14T14:19:36.859 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:19:36 smithi122 ceph-mon[25721]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T14:19:37.699 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:19:37 smithi122 ceph-mon[25721]: Upgrade: Updating mon.b 2024-04-14T14:19:37.700 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:19:37 smithi122 ceph-mon[25721]: Deploying daemon mon.b on smithi122 2024-04-14T14:19:38.067 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:19:37 smithi069 ceph-mon[24818]: Upgrade: Updating mon.b 2024-04-14T14:19:38.067 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:19:37 smithi069 ceph-mon[24818]: Deploying daemon mon.b on smithi122 2024-04-14T14:19:38.067 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:19:37 smithi069 ceph-mon[29472]: Upgrade: Updating mon.b 2024-04-14T14:19:38.067 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:19:37 smithi069 ceph-mon[29472]: Deploying daemon mon.b on smithi122 2024-04-14T14:19:38.230 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:19:37 smithi122 systemd[1]: Stopping Ceph mon.b for dc326fc2-fa65-11ee-bc8e-c7b262605968... 2024-04-14T14:19:38.595 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:19:38 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mon-b[25717]: 2024-04-14T14:19:38.227+0000 7f7ad97a3700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.b -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-04-14T14:19:38.595 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:19:38 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mon-b[25717]: 2024-04-14T14:19:38.227+0000 7f7ad97a3700 -1 mon.b@2(peon) e3 *** Got Signal Terminated *** 2024-04-14T14:19:38.595 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:19:38 smithi122 podman[67832]: 2024-04-14 14:19:38.309131244 +0000 UTC m=+0.198325856 container died 3b766bfce4be5663d61ecbbff3cf9815245b7c7a653cd757cf48d86884426543 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mon-b, RELEASE=HEAD, build-date=2022-05-03T08:36:31.336870, architecture=x86_64, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, maintainer=Guillaume Abrioux , vcs-type=git, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, io.buildah.version=1.19.8, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., io.k8s.display-name=CentOS Stream 8, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, version=8, io.openshift.tags=base centos centos-stream, distribution-scope=public, com.redhat.component=centos-stream-container, io.openshift.expose-services=, CEPH_POINT_RELEASE=-17.2.0, GIT_BRANCH=HEAD, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, vendor=Red Hat, Inc., GIT_CLEAN=True, release=754, GIT_REPO=https://github.com/ceph/ceph-container.git, ceph=True, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, name=centos-stream, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb) 2024-04-14T14:19:38.938 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:19:38 smithi122 podman[67832]: 2024-04-14 14:19:38.703197502 +0000 UTC m=+0.592392114 container cleanup 3b766bfce4be5663d61ecbbff3cf9815245b7c7a653cd757cf48d86884426543 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mon-b, vendor=Red Hat, Inc., GIT_REPO=https://github.com/ceph/ceph-container.git, distribution-scope=public, architecture=x86_64, io.openshift.tags=base centos centos-stream, maintainer=Guillaume Abrioux , GIT_CLEAN=True, com.redhat.component=centos-stream-container, io.openshift.expose-services=, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, ceph=True, GIT_BRANCH=HEAD, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, io.k8s.display-name=CentOS Stream 8, release=754, build-date=2022-05-03T08:36:31.336870, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, io.buildah.version=1.19.8, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, vcs-type=git, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, CEPH_POINT_RELEASE=-17.2.0, RELEASE=HEAD, version=8, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, name=centos-stream, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb) 2024-04-14T14:19:38.939 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:19:38 smithi122 bash[67832]: ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mon-b 2024-04-14T14:19:39.245 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:19:38 smithi122 podman[67846]: 2024-04-14 14:19:38.937470614 +0000 UTC m=+0.629661954 container remove 3b766bfce4be5663d61ecbbff3cf9815245b7c7a653cd757cf48d86884426543 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mon-b, GIT_CLEAN=True, GIT_REPO=https://github.com/ceph/ceph-container.git, name=centos-stream, version=8, vcs-type=git, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, release=754, com.redhat.component=centos-stream-container, io.openshift.tags=base centos centos-stream, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, build-date=2022-05-03T08:36:31.336870, architecture=x86_64, ceph=True, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, GIT_BRANCH=HEAD, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., io.openshift.expose-services=, io.k8s.display-name=CentOS Stream 8, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, vendor=Red Hat, Inc., maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=-17.2.0, RELEASE=HEAD, io.buildah.version=1.19.8, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, distribution-scope=public) 2024-04-14T14:19:39.246 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:19:39 smithi122 systemd[1]: ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968@mon.b.service: Deactivated successfully. 2024-04-14T14:19:39.246 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:19:39 smithi122 systemd[1]: Stopped Ceph mon.b for dc326fc2-fa65-11ee-bc8e-c7b262605968. 2024-04-14T14:19:39.246 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:19:39 smithi122 systemd[1]: ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968@mon.b.service: Consumed 16.765s CPU time. 2024-04-14T14:19:39.562 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:19:39 smithi122 systemd[1]: Starting Ceph mon.b for dc326fc2-fa65-11ee-bc8e-c7b262605968... 2024-04-14T14:19:39.562 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:19:39 smithi122 podman[67979]: 2024-04-14 14:19:39.362926072 +0000 UTC m=+0.018086402 image pull b96c509eef841460b3271f5fdd331860da87b004524f4ed59414aa3680183b53 quay.ceph.io/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888 2024-04-14T14:19:39.562 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:19:39 smithi122 podman[67979]: 2024-04-14 14:19:39.463389836 +0000 UTC m=+0.118550167 container create 05f437a1a9a1d666fc75595e6dd186557ac67a8fcc56c8a07ac28426fa996267 (image=quay.ceph.io/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888, name=ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mon-b, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, RELEASE=main-a971fd5, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.build-date=20240403) 2024-04-14T14:19:39.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:19:39 smithi122 podman[67979]: 2024-04-14 14:19:39.674111334 +0000 UTC m=+0.329271666 container init 05f437a1a9a1d666fc75595e6dd186557ac67a8fcc56c8a07ac28426fa996267 (image=quay.ceph.io/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888, name=ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mon-b, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.license=GPLv2, ceph=True, RELEASE=main-a971fd5, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.build-date=20240403, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-14T14:19:39.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:19:39 smithi122 podman[67979]: 2024-04-14 14:19:39.677179208 +0000 UTC m=+0.332339539 container start 05f437a1a9a1d666fc75595e6dd186557ac67a8fcc56c8a07ac28426fa996267 (image=quay.ceph.io/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888, name=ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mon-b, GIT_CLEAN=True, org.label-schema.build-date=20240403, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, RELEASE=main-a971fd5, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD) 2024-04-14T14:19:39.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:19:39 smithi122 ceph-mon[67998]: set uid:gid to 167:167 (ceph:ceph) 2024-04-14T14:19:39.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:19:39 smithi122 ceph-mon[67998]: ceph version 19.0.0-2778-ga971fd5f (a971fd5f5719038c70b817bafacd6d5400b1d888) squid (dev), process ceph-mon, pid 2 2024-04-14T14:19:39.846 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:19:39 smithi122 ceph-mon[67998]: pidfile_write: ignore empty --pid-file 2024-04-14T14:19:39.846 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:19:39 smithi122 ceph-mon[67998]: load: jerasure load: lrc 2024-04-14T14:19:39.846 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:19:39 smithi122 ceph-mon[67998]: rocksdb: RocksDB version: 7.9.2 2024-04-14T14:19:39.846 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:19:39 smithi122 ceph-mon[67998]: rocksdb: Git sha 0 2024-04-14T14:19:39.846 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:19:39 smithi122 ceph-mon[67998]: rocksdb: Compile date 2024-04-06 20:28:24 2024-04-14T14:19:39.846 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:19:39 smithi122 ceph-mon[67998]: rocksdb: DB SUMMARY 2024-04-14T14:19:39.846 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:19:39 smithi122 ceph-mon[67998]: rocksdb: DB Session ID: NOCV3CGSGW6RP8WM3807 2024-04-14T14:19:39.846 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:19:39 smithi122 ceph-mon[67998]: rocksdb: CURRENT file: CURRENT 2024-04-14T14:19:39.846 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:19:39 smithi122 ceph-mon[67998]: rocksdb: IDENTITY file: IDENTITY 2024-04-14T14:19:39.846 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:19:39 smithi122 ceph-mon[67998]: rocksdb: MANIFEST file: MANIFEST-000009 size: 2816 Bytes 2024-04-14T14:19:39.846 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:19:39 smithi122 ceph-mon[67998]: rocksdb: SST files in /var/lib/ceph/mon/ceph-b/store.db dir, Total Num: 1, files: 000054.sst 2024-04-14T14:19:39.846 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:19:39 smithi122 ceph-mon[67998]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-b/store.db: 000052.log size: 1717733 ; 2024-04-14T14:19:39.846 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:19:39 smithi122 ceph-mon[67998]: rocksdb: Options.error_if_exists: 0 2024-04-14T14:19:39.846 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:19:39 smithi122 ceph-mon[67998]: rocksdb: Options.create_if_missing: 0 2024-04-14T14:19:39.846 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:19:39 smithi122 ceph-mon[67998]: rocksdb: Options.paranoid_checks: 1 2024-04-14T14:19:39.847 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:19:39 smithi122 ceph-mon[67998]: rocksdb: Options.flush_verify_memtable_count: 1 2024-04-14T14:19:39.847 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:19:39 smithi122 ceph-mon[67998]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-04-14T14:19:39.847 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:19:39 smithi122 ceph-mon[67998]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-04-14T14:19:39.847 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:19:39 smithi122 ceph-mon[67998]: rocksdb: Options.env: 0x563535bb3c60 2024-04-14T14:19:39.847 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:19:39 smithi122 ceph-mon[67998]: rocksdb: Options.fs: PosixFileSystem 2024-04-14T14:19:39.847 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:19:39 smithi122 ceph-mon[67998]: rocksdb: Options.info_log: 0x56353775fbe0 2024-04-14T14:19:39.847 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:19:39 smithi122 ceph-mon[67998]: rocksdb: Options.max_file_opening_threads: 16 2024-04-14T14:19:39.847 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:19:39 smithi122 ceph-mon[67998]: rocksdb: Options.statistics: (nil) 2024-04-14T14:19:39.848 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:19:39 smithi122 ceph-mon[67998]: rocksdb: Options.use_fsync: 0 2024-04-14T14:19:39.848 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:19:39 smithi122 ceph-mon[67998]: rocksdb: Options.max_log_file_size: 0 2024-04-14T14:19:39.848 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:19:39 smithi122 ceph-mon[67998]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-04-14T14:19:39.849 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:19:39 smithi122 ceph-mon[67998]: rocksdb: Options.log_file_time_to_roll: 0 2024-04-14T14:19:39.849 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:19:39 smithi122 ceph-mon[67998]: rocksdb: Options.keep_log_file_num: 1000 2024-04-14T14:19:39.849 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:19:39 smithi122 ceph-mon[67998]: rocksdb: Options.recycle_log_file_num: 0 2024-04-14T14:19:39.849 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:19:39 smithi122 ceph-mon[67998]: rocksdb: Options.allow_fallocate: 1 2024-04-14T14:19:39.849 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:19:39 smithi122 ceph-mon[67998]: rocksdb: Options.allow_mmap_reads: 0 2024-04-14T14:19:39.849 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:19:39 smithi122 ceph-mon[67998]: rocksdb: Options.allow_mmap_writes: 0 2024-04-14T14:19:39.849 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:19:39 smithi122 ceph-mon[67998]: rocksdb: Options.use_direct_reads: 0 2024-04-14T14:19:39.849 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:19:39 smithi122 ceph-mon[67998]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-04-14T14:19:39.849 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:19:39 smithi122 ceph-mon[67998]: rocksdb: Options.create_missing_column_families: 0 2024-04-14T14:19:39.849 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:19:39 smithi122 ceph-mon[67998]: rocksdb: Options.db_log_dir: 2024-04-14T14:19:39.849 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:19:39 smithi122 ceph-mon[67998]: rocksdb: Options.wal_dir: 2024-04-14T14:19:39.849 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:19:39 smithi122 ceph-mon[67998]: rocksdb: Options.table_cache_numshardbits: 6 2024-04-14T14:19:39.849 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:19:39 smithi122 ceph-mon[67998]: rocksdb: Options.WAL_ttl_seconds: 0 2024-04-14T14:19:39.849 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:19:39 smithi122 ceph-mon[67998]: rocksdb: Options.WAL_size_limit_MB: 0 2024-04-14T14:19:39.849 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:19:39 smithi122 ceph-mon[67998]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-04-14T14:19:39.850 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:19:39 smithi122 ceph-mon[67998]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-04-14T14:19:39.850 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:19:39 smithi122 ceph-mon[67998]: rocksdb: Options.is_fd_close_on_exec: 1 2024-04-14T14:19:39.850 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:19:39 smithi122 ceph-mon[67998]: rocksdb: Options.advise_random_on_open: 1 2024-04-14T14:19:39.850 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:19:39 smithi122 ceph-mon[67998]: rocksdb: Options.db_write_buffer_size: 0 2024-04-14T14:19:39.850 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:19:39 smithi122 ceph-mon[67998]: rocksdb: Options.write_buffer_manager: 0x563537730820 2024-04-14T14:19:39.850 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:19:39 smithi122 ceph-mon[67998]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-04-14T14:19:39.850 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:19:39 smithi122 ceph-mon[67998]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-04-14T14:19:39.850 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:19:39 smithi122 ceph-mon[67998]: rocksdb: Options.use_adaptive_mutex: 0 2024-04-14T14:19:39.850 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:19:39 smithi122 ceph-mon[67998]: rocksdb: Options.rate_limiter: (nil) 2024-04-14T14:19:39.850 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:19:39 smithi122 ceph-mon[67998]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-04-14T14:19:39.850 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:19:39 smithi122 ceph-mon[67998]: rocksdb: Options.wal_recovery_mode: 2 2024-04-14T14:19:39.850 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:19:39 smithi122 ceph-mon[67998]: rocksdb: Options.enable_thread_tracking: 0 2024-04-14T14:19:39.850 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:19:39 smithi122 ceph-mon[67998]: rocksdb: Options.enable_pipelined_write: 0 2024-04-14T14:19:39.850 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:19:39 smithi122 ceph-mon[67998]: rocksdb: Options.unordered_write: 0 2024-04-14T14:19:39.851 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:19:39 smithi122 ceph-mon[67998]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-04-14T14:19:39.851 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:19:39 smithi122 ceph-mon[67998]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-04-14T14:19:39.851 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:19:39 smithi122 ceph-mon[67998]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-04-14T14:19:39.851 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:19:39 smithi122 ceph-mon[67998]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-04-14T14:19:39.851 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:19:39 smithi122 ceph-mon[67998]: rocksdb: Options.row_cache: None 2024-04-14T14:19:39.851 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:19:39 smithi122 ceph-mon[67998]: rocksdb: Options.wal_filter: None 2024-04-14T14:19:39.851 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:19:39 smithi122 ceph-mon[67998]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-04-14T14:19:39.851 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:19:39 smithi122 ceph-mon[67998]: rocksdb: Options.allow_ingest_behind: 0 2024-04-14T14:19:39.851 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:19:39 smithi122 ceph-mon[67998]: rocksdb: Options.two_write_queues: 0 2024-04-14T14:19:39.851 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:19:39 smithi122 ceph-mon[67998]: rocksdb: Options.manual_wal_flush: 0 2024-04-14T14:19:39.851 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:19:39 smithi122 ceph-mon[67998]: rocksdb: Options.wal_compression: 0 2024-04-14T14:19:39.851 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:19:39 smithi122 ceph-mon[67998]: rocksdb: Options.atomic_flush: 0 2024-04-14T14:19:39.851 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:19:39 smithi122 ceph-mon[67998]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-04-14T14:19:39.851 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:19:39 smithi122 ceph-mon[67998]: rocksdb: Options.persist_stats_to_disk: 0 2024-04-14T14:19:39.851 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:19:39 smithi122 ceph-mon[67998]: rocksdb: Options.write_dbid_to_manifest: 0 2024-04-14T14:19:39.852 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:19:39 smithi122 ceph-mon[67998]: rocksdb: Options.log_readahead_size: 0 2024-04-14T14:19:39.852 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:19:39 smithi122 ceph-mon[67998]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-04-14T14:19:39.852 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:19:39 smithi122 ceph-mon[67998]: rocksdb: Options.best_efforts_recovery: 0 2024-04-14T14:19:39.852 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:19:39 smithi122 ceph-mon[67998]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-04-14T14:19:39.852 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:19:39 smithi122 ceph-mon[67998]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-04-14T14:19:39.852 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:19:39 smithi122 ceph-mon[67998]: rocksdb: Options.allow_data_in_errors: 0 2024-04-14T14:19:39.852 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:19:39 smithi122 ceph-mon[67998]: rocksdb: Options.db_host_id: __hostname__ 2024-04-14T14:19:39.852 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:19:39 smithi122 ceph-mon[67998]: rocksdb: Options.enforce_single_del_contracts: true 2024-04-14T14:19:39.852 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:19:39 smithi122 ceph-mon[67998]: rocksdb: Options.max_background_jobs: 2 2024-04-14T14:19:39.852 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:19:39 smithi122 ceph-mon[67998]: rocksdb: Options.max_background_compactions: -1 2024-04-14T14:19:39.852 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:19:39 smithi122 ceph-mon[67998]: rocksdb: Options.max_subcompactions: 1 2024-04-14T14:19:39.852 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:19:39 smithi122 ceph-mon[67998]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-04-14T14:19:39.852 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:19:39 smithi122 ceph-mon[67998]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-04-14T14:19:39.852 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:19:39 smithi122 ceph-mon[67998]: rocksdb: Options.delayed_write_rate : 16777216 2024-04-14T14:19:39.852 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:19:39 smithi122 ceph-mon[67998]: rocksdb: Options.max_total_wal_size: 0 2024-04-14T14:19:39.853 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:19:39 smithi122 ceph-mon[67998]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-04-14T14:19:39.853 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:19:39 smithi122 ceph-mon[67998]: rocksdb: Options.stats_dump_period_sec: 600 2024-04-14T14:19:39.853 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:19:39 smithi122 ceph-mon[67998]: rocksdb: Options.stats_persist_period_sec: 600 2024-04-14T14:19:39.853 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:19:39 smithi122 ceph-mon[67998]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-04-14T14:19:39.853 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:19:39 smithi122 ceph-mon[67998]: rocksdb: Options.max_open_files: -1 2024-04-14T14:19:39.853 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:19:39 smithi122 ceph-mon[67998]: rocksdb: Options.bytes_per_sync: 0 2024-04-14T14:19:39.853 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:19:39 smithi122 ceph-mon[67998]: rocksdb: Options.wal_bytes_per_sync: 0 2024-04-14T14:19:39.853 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:19:39 smithi122 ceph-mon[67998]: rocksdb: Options.strict_bytes_per_sync: 0 2024-04-14T14:19:39.853 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:19:39 smithi122 ceph-mon[67998]: rocksdb: Options.compaction_readahead_size: 0 2024-04-14T14:19:39.853 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:19:39 smithi122 ceph-mon[67998]: rocksdb: Options.max_background_flushes: -1 2024-04-14T14:19:39.853 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:19:39 smithi122 ceph-mon[67998]: rocksdb: Compression algorithms supported: 2024-04-14T14:19:39.853 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:19:39 smithi122 ceph-mon[67998]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-04-14T14:19:39.853 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:19:39 smithi122 ceph-mon[67998]: rocksdb: kZSTD supported: 0 2024-04-14T14:19:39.853 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:19:39 smithi122 ceph-mon[67998]: rocksdb: kXpressCompression supported: 0 2024-04-14T14:19:39.853 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:19:39 smithi122 ceph-mon[67998]: rocksdb: kLZ4HCCompression supported: 1 2024-04-14T14:19:39.854 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:19:39 smithi122 ceph-mon[67998]: rocksdb: kZlibCompression supported: 1 2024-04-14T14:19:39.854 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:19:39 smithi122 ceph-mon[67998]: rocksdb: kSnappyCompression supported: 1 2024-04-14T14:19:39.854 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:19:39 smithi122 ceph-mon[67998]: rocksdb: kLZ4Compression supported: 1 2024-04-14T14:19:39.854 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:19:39 smithi122 ceph-mon[67998]: rocksdb: kBZip2Compression supported: 0 2024-04-14T14:19:39.854 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:19:39 smithi122 ceph-mon[67998]: rocksdb: Fast CRC32 supported: Supported on x86 2024-04-14T14:19:39.854 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:19:39 smithi122 ceph-mon[67998]: rocksdb: DMutex implementation: pthread_mutex_t 2024-04-14T14:19:39.854 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:19:39 smithi122 ceph-mon[67998]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-b/store.db/MANIFEST-000009 2024-04-14T14:19:39.854 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:19:39 smithi122 ceph-mon[67998]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-04-14T14:19:39.854 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:19:39 smithi122 ceph-mon[67998]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-04-14T14:19:39.854 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:19:39 smithi122 ceph-mon[67998]: rocksdb: Options.merge_operator: 2024-04-14T14:19:39.854 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:19:39 smithi122 ceph-mon[67998]: rocksdb: Options.compaction_filter: None 2024-04-14T14:19:39.854 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:19:39 smithi122 ceph-mon[67998]: rocksdb: Options.compaction_filter_factory: None 2024-04-14T14:19:39.854 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:19:39 smithi122 ceph-mon[67998]: rocksdb: Options.sst_partitioner_factory: None 2024-04-14T14:19:39.855 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:19:39 smithi122 ceph-mon[67998]: rocksdb: Options.memtable_factory: SkipListFactory 2024-04-14T14:19:39.855 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:19:39 smithi122 ceph-mon[67998]: rocksdb: Options.table_factory: BlockBasedTable 2024-04-14T14:19:39.855 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:19:39 smithi122 ceph-mon[67998]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x56353775fdc0) 2024-04-14T14:19:39.855 INFO:journalctl@ceph.mon.b.smithi122.stdout: cache_index_and_filter_blocks: 1 2024-04-14T14:19:39.855 INFO:journalctl@ceph.mon.b.smithi122.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-04-14T14:19:39.855 INFO:journalctl@ceph.mon.b.smithi122.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-04-14T14:19:39.855 INFO:journalctl@ceph.mon.b.smithi122.stdout: pin_top_level_index_and_filter: 1 2024-04-14T14:19:39.855 INFO:journalctl@ceph.mon.b.smithi122.stdout: index_type: 0 2024-04-14T14:19:39.855 INFO:journalctl@ceph.mon.b.smithi122.stdout: data_block_index_type: 0 2024-04-14T14:19:39.855 INFO:journalctl@ceph.mon.b.smithi122.stdout: index_shortening: 1 2024-04-14T14:19:39.855 INFO:journalctl@ceph.mon.b.smithi122.stdout: data_block_hash_table_util_ratio: 0.750000 2024-04-14T14:19:39.855 INFO:journalctl@ceph.mon.b.smithi122.stdout: checksum: 4 2024-04-14T14:19:39.855 INFO:journalctl@ceph.mon.b.smithi122.stdout: no_block_cache: 0 2024-04-14T14:19:39.855 INFO:journalctl@ceph.mon.b.smithi122.stdout: block_cache: 0x563537763090 2024-04-14T14:19:39.856 INFO:journalctl@ceph.mon.b.smithi122.stdout: block_cache_name: BinnedLRUCache 2024-04-14T14:19:39.856 INFO:journalctl@ceph.mon.b.smithi122.stdout: block_cache_options: 2024-04-14T14:19:39.856 INFO:journalctl@ceph.mon.b.smithi122.stdout: capacity : 536870912 2024-04-14T14:19:39.856 INFO:journalctl@ceph.mon.b.smithi122.stdout: num_shard_bits : 4 2024-04-14T14:19:39.856 INFO:journalctl@ceph.mon.b.smithi122.stdout: strict_capacity_limit : 0 2024-04-14T14:19:39.856 INFO:journalctl@ceph.mon.b.smithi122.stdout: high_pri_pool_ratio: 0.000 2024-04-14T14:19:39.856 INFO:journalctl@ceph.mon.b.smithi122.stdout: block_cache_compressed: (nil) 2024-04-14T14:19:39.856 INFO:journalctl@ceph.mon.b.smithi122.stdout: persistent_cache: (nil) 2024-04-14T14:19:39.856 INFO:journalctl@ceph.mon.b.smithi122.stdout: block_size: 4096 2024-04-14T14:19:39.856 INFO:journalctl@ceph.mon.b.smithi122.stdout: block_size_deviation: 10 2024-04-14T14:19:39.856 INFO:journalctl@ceph.mon.b.smithi122.stdout: block_restart_interval: 16 2024-04-14T14:19:39.856 INFO:journalctl@ceph.mon.b.smithi122.stdout: index_block_restart_interval: 1 2024-04-14T14:19:39.857 INFO:journalctl@ceph.mon.b.smithi122.stdout: metadata_block_size: 4096 2024-04-14T14:19:39.857 INFO:journalctl@ceph.mon.b.smithi122.stdout: partition_filters: 0 2024-04-14T14:19:39.857 INFO:journalctl@ceph.mon.b.smithi122.stdout: use_delta_encoding: 1 2024-04-14T14:19:39.857 INFO:journalctl@ceph.mon.b.smithi122.stdout: filter_policy: bloomfilter 2024-04-14T14:19:39.857 INFO:journalctl@ceph.mon.b.smithi122.stdout: whole_key_filtering: 1 2024-04-14T14:19:39.857 INFO:journalctl@ceph.mon.b.smithi122.stdout: verify_compression: 0 2024-04-14T14:19:39.857 INFO:journalctl@ceph.mon.b.smithi122.stdout: read_amp_bytes_per_bit: 0 2024-04-14T14:19:39.857 INFO:journalctl@ceph.mon.b.smithi122.stdout: format_version: 5 2024-04-14T14:19:39.857 INFO:journalctl@ceph.mon.b.smithi122.stdout: enable_index_compression: 1 2024-04-14T14:19:39.857 INFO:journalctl@ceph.mon.b.smithi122.stdout: block_align: 0 2024-04-14T14:19:39.857 INFO:journalctl@ceph.mon.b.smithi122.stdout: max_auto_readahead_size: 262144 2024-04-14T14:19:39.857 INFO:journalctl@ceph.mon.b.smithi122.stdout: prepopulate_block_cache: 0 2024-04-14T14:19:39.857 INFO:journalctl@ceph.mon.b.smithi122.stdout: initial_auto_readahead_size: 8192 2024-04-14T14:19:39.857 INFO:journalctl@ceph.mon.b.smithi122.stdout: num_file_reads_for_auto_readahead: 2 2024-04-14T14:19:39.857 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:19:39 smithi122 ceph-mon[67998]: rocksdb: Options.write_buffer_size: 33554432 2024-04-14T14:19:39.858 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:19:39 smithi122 ceph-mon[67998]: rocksdb: Options.max_write_buffer_number: 2 2024-04-14T14:19:39.858 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:19:39 smithi122 ceph-mon[67998]: rocksdb: Options.compression: NoCompression 2024-04-14T14:19:39.858 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:19:39 smithi122 ceph-mon[67998]: rocksdb: Options.bottommost_compression: Disabled 2024-04-14T14:19:39.858 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:19:39 smithi122 ceph-mon[67998]: rocksdb: Options.prefix_extractor: nullptr 2024-04-14T14:19:39.858 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:19:39 smithi122 ceph-mon[67998]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-04-14T14:19:39.858 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:19:39 smithi122 ceph-mon[67998]: rocksdb: Options.num_levels: 7 2024-04-14T14:19:39.858 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:19:39 smithi122 ceph-mon[67998]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-04-14T14:19:39.858 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:19:39 smithi122 ceph-mon[67998]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-04-14T14:19:39.858 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:19:39 smithi122 ceph-mon[67998]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-04-14T14:19:39.858 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:19:39 smithi122 ceph-mon[67998]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-04-14T14:19:39.858 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:19:39 smithi122 ceph-mon[67998]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-04-14T14:19:39.858 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:19:39 smithi122 ceph-mon[67998]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-04-14T14:19:39.858 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:19:39 smithi122 ceph-mon[67998]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-04-14T14:19:39.858 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:19:39 smithi122 ceph-mon[67998]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-04-14T14:19:39.859 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:19:39 smithi122 ceph-mon[67998]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-04-14T14:19:39.859 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:19:39 smithi122 ceph-mon[67998]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-04-14T14:19:39.859 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:19:39 smithi122 ceph-mon[67998]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-04-14T14:19:39.859 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:19:39 smithi122 ceph-mon[67998]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-04-14T14:19:39.859 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:19:39 smithi122 ceph-mon[67998]: rocksdb: Options.compression_opts.window_bits: -14 2024-04-14T14:19:39.859 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:19:39 smithi122 ceph-mon[67998]: rocksdb: Options.compression_opts.level: 32767 2024-04-14T14:19:39.859 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:19:39 smithi122 ceph-mon[67998]: rocksdb: Options.compression_opts.strategy: 0 2024-04-14T14:19:39.859 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:19:39 smithi122 ceph-mon[67998]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-04-14T14:19:39.859 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:19:39 smithi122 ceph-mon[67998]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-04-14T14:19:39.859 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:19:39 smithi122 ceph-mon[67998]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-04-14T14:19:39.859 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:19:39 smithi122 ceph-mon[67998]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-04-14T14:19:39.860 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:19:39 smithi122 ceph-mon[67998]: rocksdb: Options.compression_opts.enabled: false 2024-04-14T14:19:39.860 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:19:39 smithi122 ceph-mon[67998]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-04-14T14:19:39.860 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:19:39 smithi122 ceph-mon[67998]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-04-14T14:19:39.860 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:19:39 smithi122 ceph-mon[67998]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-04-14T14:19:39.860 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:19:39 smithi122 ceph-mon[67998]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-04-14T14:19:39.860 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:19:39 smithi122 ceph-mon[67998]: rocksdb: Options.target_file_size_base: 67108864 2024-04-14T14:19:39.860 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:19:39 smithi122 ceph-mon[67998]: rocksdb: Options.target_file_size_multiplier: 1 2024-04-14T14:19:39.860 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:19:39 smithi122 ceph-mon[67998]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-04-14T14:19:39.860 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:19:39 smithi122 ceph-mon[67998]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-04-14T14:19:39.860 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:19:39 smithi122 ceph-mon[67998]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-04-14T14:19:39.860 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:19:39 smithi122 ceph-mon[67998]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-04-14T14:19:39.860 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:19:39 smithi122 ceph-mon[67998]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-04-14T14:19:39.860 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:19:39 smithi122 ceph-mon[67998]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-04-14T14:19:39.860 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:19:39 smithi122 ceph-mon[67998]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-04-14T14:19:39.860 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:19:39 smithi122 ceph-mon[67998]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-04-14T14:19:39.861 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:19:39 smithi122 ceph-mon[67998]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-04-14T14:19:39.861 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:19:39 smithi122 ceph-mon[67998]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-04-14T14:19:39.861 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:19:39 smithi122 ceph-mon[67998]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-04-14T14:19:39.861 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:19:39 smithi122 ceph-mon[67998]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-04-14T14:19:39.861 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:19:39 smithi122 ceph-mon[67998]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-04-14T14:19:39.861 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:19:39 smithi122 ceph-mon[67998]: rocksdb: Options.arena_block_size: 1048576 2024-04-14T14:19:39.861 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:19:39 smithi122 ceph-mon[67998]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-04-14T14:19:39.861 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:19:39 smithi122 ceph-mon[67998]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-04-14T14:19:39.861 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:19:39 smithi122 ceph-mon[67998]: rocksdb: Options.disable_auto_compactions: 0 2024-04-14T14:19:39.861 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:19:39 smithi122 ceph-mon[67998]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-04-14T14:19:39.861 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:19:39 smithi122 ceph-mon[67998]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-04-14T14:19:39.861 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:19:39 smithi122 ceph-mon[67998]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-04-14T14:19:39.861 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:19:39 smithi122 ceph-mon[67998]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-04-14T14:19:39.861 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:19:39 smithi122 ceph-mon[67998]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-04-14T14:19:39.861 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:19:39 smithi122 ceph-mon[67998]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-04-14T14:19:39.862 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:19:39 smithi122 ceph-mon[67998]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-04-14T14:19:39.862 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:19:39 smithi122 ceph-mon[67998]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-04-14T14:19:39.862 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:19:39 smithi122 ceph-mon[67998]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-04-14T14:19:39.862 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:19:39 smithi122 ceph-mon[67998]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-04-14T14:19:39.862 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:19:39 smithi122 ceph-mon[67998]: rocksdb: Options.table_properties_collectors: CompactOnDeletionCollector (Sliding window size = 32768 Deletion trigger = 16384 Deletion ratio = 0); 2024-04-14T14:19:39.862 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:19:39 smithi122 ceph-mon[67998]: rocksdb: Options.inplace_update_support: 0 2024-04-14T14:19:39.862 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:19:39 smithi122 ceph-mon[67998]: rocksdb: Options.inplace_update_num_locks: 10000 2024-04-14T14:19:39.862 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:19:39 smithi122 ceph-mon[67998]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-04-14T14:19:39.862 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:19:39 smithi122 ceph-mon[67998]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-04-14T14:19:39.862 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:19:39 smithi122 ceph-mon[67998]: rocksdb: Options.memtable_huge_page_size: 0 2024-04-14T14:19:39.862 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:19:39 smithi122 ceph-mon[67998]: rocksdb: Options.bloom_locality: 0 2024-04-14T14:19:39.862 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:19:39 smithi122 ceph-mon[67998]: rocksdb: Options.max_successive_merges: 0 2024-04-14T14:19:39.862 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:19:39 smithi122 ceph-mon[67998]: rocksdb: Options.optimize_filters_for_hits: 0 2024-04-14T14:19:39.862 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:19:39 smithi122 ceph-mon[67998]: rocksdb: Options.paranoid_file_checks: 0 2024-04-14T14:19:39.862 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:19:39 smithi122 ceph-mon[67998]: rocksdb: Options.force_consistency_checks: 1 2024-04-14T14:19:39.863 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:19:39 smithi122 ceph-mon[67998]: rocksdb: Options.report_bg_io_stats: 0 2024-04-14T14:19:39.863 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:19:39 smithi122 ceph-mon[67998]: rocksdb: Options.ttl: 2592000 2024-04-14T14:19:39.863 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:19:39 smithi122 ceph-mon[67998]: rocksdb: Options.periodic_compaction_seconds: 0 2024-04-14T14:19:39.863 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:19:39 smithi122 ceph-mon[67998]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-04-14T14:19:39.863 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:19:39 smithi122 ceph-mon[67998]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-04-14T14:19:39.863 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:19:39 smithi122 ceph-mon[67998]: rocksdb: Options.enable_blob_files: false 2024-04-14T14:19:39.863 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:19:39 smithi122 ceph-mon[67998]: rocksdb: Options.min_blob_size: 0 2024-04-14T14:19:39.863 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:19:39 smithi122 ceph-mon[67998]: rocksdb: Options.blob_file_size: 268435456 2024-04-14T14:19:39.863 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:19:39 smithi122 ceph-mon[67998]: rocksdb: Options.blob_compression_type: NoCompression 2024-04-14T14:19:39.863 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:19:39 smithi122 ceph-mon[67998]: rocksdb: Options.enable_blob_garbage_collection: false 2024-04-14T14:19:39.863 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:19:39 smithi122 ceph-mon[67998]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-04-14T14:19:39.863 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:19:39 smithi122 ceph-mon[67998]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-04-14T14:19:39.863 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:19:39 smithi122 ceph-mon[67998]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-04-14T14:19:39.863 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:19:39 smithi122 ceph-mon[67998]: rocksdb: Options.blob_file_starting_level: 0 2024-04-14T14:19:39.863 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:19:39 smithi122 ceph-mon[67998]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-04-14T14:19:39.864 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:19:39 smithi122 ceph-mon[67998]: rocksdb: [table/block_based/block_based_table_reader.cc:721] At least one SST file opened without unique ID to verify: 54.sst 2024-04-14T14:19:39.864 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:19:39 smithi122 ceph-mon[67998]: rocksdb: [db/version_set.cc:4390] More existing levels in DB than needed. max_bytes_for_level_multiplier may not be guaranteed. 2024-04-14T14:19:39.864 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:19:39 smithi122 ceph-mon[67998]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-b/store.db/MANIFEST-000009 succeeded,manifest_file_number is 9, next_file_number is 56, last_sequence is 28770, log_number is 52,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2024-04-14T14:19:39.864 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:19:39 smithi122 ceph-mon[67998]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 52 2024-04-14T14:19:39.864 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:19:39 smithi122 ceph-mon[67998]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: 1068778e-f7d8-49e5-b49e-6136684706ae 2024-04-14T14:19:39.864 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:19:39 smithi122 ceph-mon[67998]: rocksdb: EVENT_LOG_v1 {"time_micros": 1713104379706541, "job": 1, "event": "recovery_started", "wal_files": [52]} 2024-04-14T14:19:39.864 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:19:39 smithi122 ceph-mon[67998]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #52 mode 2 2024-04-14T14:19:39.864 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:19:39 smithi122 ceph-mon[67998]: rocksdb: EVENT_LOG_v1 {"time_micros": 1713104379711775, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 57, "file_size": 1088860, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 28776, "largest_seqno": 29956, "table_properties": {"data_size": 1084156, "index_size": 2512, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 1157, "raw_key_size": 11052, "raw_average_key_size": 24, "raw_value_size": 1074676, "raw_average_value_size": 2382, "num_data_blocks": 113, "num_entries": 451, "num_filter_entries": 451, "num_deletions": 7, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[CompactOnDeletionCollector]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1713104379, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "1068778e-f7d8-49e5-b49e-6136684706ae", "db_session_id": "NOCV3CGSGW6RP8WM3807", "orig_file_number": 57, "seqno_to_time_mapping": "N/A"}} 2024-04-14T14:19:39.864 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:19:39 smithi122 ceph-mon[67998]: rocksdb: EVENT_LOG_v1 {"time_micros": 1713104379711864, "job": 1, "event": "recovery_finished"} 2024-04-14T14:19:39.864 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:19:39 smithi122 ceph-mon[67998]: rocksdb: [db/version_set.cc:5047] Creating manifest 59 2024-04-14T14:19:39.864 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:19:39 smithi122 ceph-mon[67998]: rocksdb: [db/version_set.cc:4390] More existing levels in DB than needed. max_bytes_for_level_multiplier may not be guaranteed. 2024-04-14T14:19:39.864 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:19:39 smithi122 ceph-mon[67998]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-b/store.db/000052.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-04-14T14:19:39.864 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:19:39 smithi122 ceph-mon[67998]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x563537852000 2024-04-14T14:19:39.864 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:19:39 smithi122 ceph-mon[67998]: rocksdb: DB pointer 0x56353783c000 2024-04-14T14:19:39.864 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:19:39 smithi122 ceph-mon[67998]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-04-14T14:19:39.865 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:19:39 smithi122 ceph-mon[67998]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-04-14T14:19:39.865 INFO:journalctl@ceph.mon.b.smithi122.stdout: ** DB Stats ** 2024-04-14T14:19:39.865 INFO:journalctl@ceph.mon.b.smithi122.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-04-14T14:19:39.865 INFO:journalctl@ceph.mon.b.smithi122.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-04-14T14:19:39.865 INFO:journalctl@ceph.mon.b.smithi122.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-04-14T14:19:39.865 INFO:journalctl@ceph.mon.b.smithi122.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-04-14T14:19:39.865 INFO:journalctl@ceph.mon.b.smithi122.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-04-14T14:19:39.865 INFO:journalctl@ceph.mon.b.smithi122.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-04-14T14:19:39.865 INFO:journalctl@ceph.mon.b.smithi122.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-04-14T14:19:39.865 INFO:journalctl@ceph.mon.b.smithi122.stdout: 2024-04-14T14:19:39.865 INFO:journalctl@ceph.mon.b.smithi122.stdout: ** Compaction Stats [default] ** 2024-04-14T14:19:39.865 INFO:journalctl@ceph.mon.b.smithi122.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-04-14T14:19:39.865 INFO:journalctl@ceph.mon.b.smithi122.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-04-14T14:19:39.865 INFO:journalctl@ceph.mon.b.smithi122.stdout: L0 1/0 1.04 MB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 415.0 0.00 0.00 1 0.003 0 0 0.0 0.0 2024-04-14T14:19:39.865 INFO:journalctl@ceph.mon.b.smithi122.stdout: L6 1/0 9.03 MB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 0.0 0.0 2024-04-14T14:19:39.866 INFO:journalctl@ceph.mon.b.smithi122.stdout: Sum 2/0 10.07 MB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 415.0 0.00 0.00 1 0.003 0 0 0.0 0.0 2024-04-14T14:19:39.866 INFO:journalctl@ceph.mon.b.smithi122.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 415.0 0.00 0.00 1 0.003 0 0 0.0 0.0 2024-04-14T14:19:39.866 INFO:journalctl@ceph.mon.b.smithi122.stdout: 2024-04-14T14:19:39.866 INFO:journalctl@ceph.mon.b.smithi122.stdout: ** Compaction Stats [default] ** 2024-04-14T14:19:39.866 INFO:journalctl@ceph.mon.b.smithi122.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-04-14T14:19:39.866 INFO:journalctl@ceph.mon.b.smithi122.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-04-14T14:19:39.866 INFO:journalctl@ceph.mon.b.smithi122.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 415.0 0.00 0.00 1 0.003 0 0 0.0 0.0 2024-04-14T14:19:39.866 INFO:journalctl@ceph.mon.b.smithi122.stdout: 2024-04-14T14:19:39.866 INFO:journalctl@ceph.mon.b.smithi122.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-04-14T14:19:39.866 INFO:journalctl@ceph.mon.b.smithi122.stdout: 2024-04-14T14:19:39.867 INFO:journalctl@ceph.mon.b.smithi122.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-04-14T14:19:39.867 INFO:journalctl@ceph.mon.b.smithi122.stdout: Flush(GB): cumulative 0.001, interval 0.001 2024-04-14T14:19:39.867 INFO:journalctl@ceph.mon.b.smithi122.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-04-14T14:19:39.867 INFO:journalctl@ceph.mon.b.smithi122.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-04-14T14:19:39.867 INFO:journalctl@ceph.mon.b.smithi122.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-04-14T14:19:39.867 INFO:journalctl@ceph.mon.b.smithi122.stdout: AddFile(Keys): cumulative 0, interval 0 2024-04-14T14:19:39.867 INFO:journalctl@ceph.mon.b.smithi122.stdout: Cumulative compaction: 0.00 GB write, 130.93 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-04-14T14:19:39.867 INFO:journalctl@ceph.mon.b.smithi122.stdout: Interval compaction: 0.00 GB write, 130.93 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-04-14T14:19:39.867 INFO:journalctl@ceph.mon.b.smithi122.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-04-14T14:19:39.867 INFO:journalctl@ceph.mon.b.smithi122.stdout: Block cache BinnedLRUCache@0x563537763090#2 capacity: 512.00 MB usage: 3.86 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 1.1e-05 secs_since: 0 2024-04-14T14:19:39.867 INFO:journalctl@ceph.mon.b.smithi122.stdout: Block cache entry stats(count,size,portion): FilterBlock(1,1.28 KB,0.000244379%) IndexBlock(1,2.58 KB,0.000491738%) Misc(1,0.00 KB,0%) 2024-04-14T14:19:39.867 INFO:journalctl@ceph.mon.b.smithi122.stdout: 2024-04-14T14:19:39.867 INFO:journalctl@ceph.mon.b.smithi122.stdout: ** File Read Latency Histogram By Level [default] ** 2024-04-14T14:19:39.867 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:19:39 smithi122 ceph-mon[67998]: starting mon.b rank 2 at public addrs [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] at bind addrs [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] mon_data /var/lib/ceph/mon/ceph-b fsid dc326fc2-fa65-11ee-bc8e-c7b262605968 2024-04-14T14:19:39.867 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:19:39 smithi122 ceph-mon[67998]: mon.b@-1(???) e3 preinit fsid dc326fc2-fa65-11ee-bc8e-c7b262605968 2024-04-14T14:19:39.868 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:19:39 smithi122 ceph-mon[67998]: mon.b@-1(???).mds e1 new map 2024-04-14T14:19:39.868 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:19:39 smithi122 ceph-mon[67998]: mon.b@-1(???).mds e1 print_map 2024-04-14T14:19:39.868 INFO:journalctl@ceph.mon.b.smithi122.stdout: e1 2024-04-14T14:19:39.868 INFO:journalctl@ceph.mon.b.smithi122.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-04-14T14:19:39.868 INFO:journalctl@ceph.mon.b.smithi122.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-14T14:19:39.868 INFO:journalctl@ceph.mon.b.smithi122.stdout: legacy client fscid: -1 2024-04-14T14:19:39.868 INFO:journalctl@ceph.mon.b.smithi122.stdout: 2024-04-14T14:19:39.868 INFO:journalctl@ceph.mon.b.smithi122.stdout: No filesystems configured 2024-04-14T14:19:39.868 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:19:39 smithi122 ceph-mon[67998]: mon.b@-1(???).osd e110 crush map has features 3314933000854323200, adjusting msgr requires 2024-04-14T14:19:39.868 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:19:39 smithi122 ceph-mon[67998]: mon.b@-1(???).osd e110 crush map has features 432629239337189376, adjusting msgr requires 2024-04-14T14:19:39.868 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:19:39 smithi122 ceph-mon[67998]: mon.b@-1(???).osd e110 crush map has features 432629239337189376, adjusting msgr requires 2024-04-14T14:19:39.868 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:19:39 smithi122 ceph-mon[67998]: mon.b@-1(???).osd e110 crush map has features 432629239337189376, adjusting msgr requires 2024-04-14T14:19:39.868 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:19:39 smithi122 ceph-mon[67998]: mon.b@-1(???).paxosservice(auth 1..25) refresh upgraded, format 0 -> 3 2024-04-14T14:19:39.868 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:19:39 smithi122 bash[67979]: 05f437a1a9a1d666fc75595e6dd186557ac67a8fcc56c8a07ac28426fa996267 2024-04-14T14:19:39.868 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:19:39 smithi122 systemd[1]: Started Ceph mon.b for dc326fc2-fa65-11ee-bc8e-c7b262605968. 2024-04-14T14:19:40.317 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:19:39 smithi069 ceph-mon[24818]: pgmap v89: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:19:40.317 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:19:39 smithi069 ceph-mon[29472]: pgmap v89: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:19:40.345 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:19:39 smithi122 ceph-mon[67998]: pgmap v89: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:19:41.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:19:40 smithi122 ceph-mon[67998]: pgmap v90: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:19:41.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:19:40 smithi122 ceph-mon[67998]: mon.b calling monitor election 2024-04-14T14:19:41.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:19:40 smithi122 ceph-mon[67998]: mon.c calling monitor election 2024-04-14T14:19:41.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:19:40 smithi122 ceph-mon[67998]: mon.a calling monitor election 2024-04-14T14:19:41.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:19:40 smithi122 ceph-mon[67998]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-04-14T14:19:41.096 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:19:40 smithi122 ceph-mon[67998]: monmap e3: 3 mons at {a=[v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0],b=[v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0],c=[v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0]} 2024-04-14T14:19:41.096 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:19:40 smithi122 ceph-mon[67998]: fsmap 2024-04-14T14:19:41.096 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:19:40 smithi122 ceph-mon[67998]: osdmap e110: 8 total, 8 up, 8 in 2024-04-14T14:19:41.096 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:19:40 smithi122 ceph-mon[67998]: mgrmap e41: y(active, since 2m), standbys: x 2024-04-14T14:19:41.096 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:19:40 smithi122 ceph-mon[67998]: overall HEALTH_OK 2024-04-14T14:19:41.096 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:19:40 smithi122 ceph-mon[67998]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' 2024-04-14T14:19:41.096 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:19:40 smithi122 ceph-mon[67998]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' 2024-04-14T14:19:41.317 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:19:40 smithi069 ceph-mon[24818]: pgmap v90: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:19:41.317 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:19:40 smithi069 ceph-mon[24818]: mon.b calling monitor election 2024-04-14T14:19:41.317 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:19:40 smithi069 ceph-mon[24818]: mon.c calling monitor election 2024-04-14T14:19:41.317 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:19:40 smithi069 ceph-mon[24818]: mon.a calling monitor election 2024-04-14T14:19:41.317 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:19:40 smithi069 ceph-mon[24818]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-04-14T14:19:41.317 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:19:40 smithi069 ceph-mon[24818]: monmap e3: 3 mons at {a=[v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0],b=[v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0],c=[v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0]} 2024-04-14T14:19:41.318 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:19:40 smithi069 ceph-mon[24818]: fsmap 2024-04-14T14:19:41.318 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:19:40 smithi069 ceph-mon[24818]: osdmap e110: 8 total, 8 up, 8 in 2024-04-14T14:19:41.318 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:19:40 smithi069 ceph-mon[24818]: mgrmap e41: y(active, since 2m), standbys: x 2024-04-14T14:19:41.318 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:19:40 smithi069 ceph-mon[24818]: overall HEALTH_OK 2024-04-14T14:19:41.318 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:19:40 smithi069 ceph-mon[24818]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' 2024-04-14T14:19:41.318 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:19:40 smithi069 ceph-mon[24818]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' 2024-04-14T14:19:41.318 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:19:40 smithi069 ceph-mon[29472]: pgmap v90: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:19:41.318 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:19:40 smithi069 ceph-mon[29472]: mon.b calling monitor election 2024-04-14T14:19:41.318 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:19:40 smithi069 ceph-mon[29472]: mon.c calling monitor election 2024-04-14T14:19:41.318 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:19:40 smithi069 ceph-mon[29472]: mon.a calling monitor election 2024-04-14T14:19:41.318 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:19:40 smithi069 ceph-mon[29472]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-04-14T14:19:41.319 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:19:40 smithi069 ceph-mon[29472]: monmap e3: 3 mons at {a=[v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0],b=[v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0],c=[v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0]} 2024-04-14T14:19:41.319 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:19:40 smithi069 ceph-mon[29472]: fsmap 2024-04-14T14:19:41.319 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:19:40 smithi069 ceph-mon[29472]: osdmap e110: 8 total, 8 up, 8 in 2024-04-14T14:19:41.319 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:19:40 smithi069 ceph-mon[29472]: mgrmap e41: y(active, since 2m), standbys: x 2024-04-14T14:19:41.319 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:19:40 smithi069 ceph-mon[29472]: overall HEALTH_OK 2024-04-14T14:19:41.319 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:19:40 smithi069 ceph-mon[29472]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' 2024-04-14T14:19:41.319 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:19:40 smithi069 ceph-mon[29472]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' 2024-04-14T14:19:42.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:19:42 smithi122 ceph-mon[67998]: pgmap v91: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:19:42.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:19:42 smithi122 ceph-mon[67998]: from='client.25417 -' entity='client.iscsi.foo.smithi069.yuumtr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T14:19:43.067 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:19:42 smithi069 ceph-mon[24818]: pgmap v91: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:19:43.067 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:19:42 smithi069 ceph-mon[24818]: from='client.25417 -' entity='client.iscsi.foo.smithi069.yuumtr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T14:19:43.067 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:19:42 smithi069 ceph-mon[29472]: pgmap v91: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:19:43.067 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:19:42 smithi069 ceph-mon[29472]: from='client.25417 -' entity='client.iscsi.foo.smithi069.yuumtr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T14:19:45.067 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:19:44 smithi069 ceph-mon[24818]: pgmap v92: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:19:45.067 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:19:44 smithi069 ceph-mon[24818]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' 2024-04-14T14:19:45.067 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:19:44 smithi069 ceph-mon[24818]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' 2024-04-14T14:19:45.067 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 14 14:19:44 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-y[63038]: 2024-04-14T14:19:44.719+0000 7ff9794e6700 -1 mgr.server handle_report got status from non-daemon mon.b 2024-04-14T14:19:45.068 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:19:44 smithi069 ceph-mon[29472]: pgmap v92: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:19:45.068 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:19:44 smithi069 ceph-mon[29472]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' 2024-04-14T14:19:45.068 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:19:44 smithi069 ceph-mon[29472]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' 2024-04-14T14:19:45.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:19:44 smithi122 ceph-mon[67998]: pgmap v92: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:19:45.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:19:44 smithi122 ceph-mon[67998]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' 2024-04-14T14:19:45.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:19:44 smithi122 ceph-mon[67998]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' 2024-04-14T14:19:46.018 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:19:45 smithi122 ceph-mon[67998]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T14:19:46.018 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:19:45 smithi122 ceph-mon[67998]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' 2024-04-14T14:19:46.018 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:19:45 smithi122 ceph-mon[67998]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' 2024-04-14T14:19:46.067 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:19:45 smithi069 ceph-mon[24818]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T14:19:46.067 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:19:45 smithi069 ceph-mon[24818]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' 2024-04-14T14:19:46.067 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:19:45 smithi069 ceph-mon[24818]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' 2024-04-14T14:19:46.067 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:19:45 smithi069 ceph-mon[29472]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T14:19:46.067 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:19:45 smithi069 ceph-mon[29472]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' 2024-04-14T14:19:46.067 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:19:45 smithi069 ceph-mon[29472]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' 2024-04-14T14:19:46.817 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:19:46 smithi069 ceph-mon[24818]: pgmap v93: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:19:46.817 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 14 14:19:46 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-y[63038]: ::ffff:172.21.15.122 - - [14/Apr/2024:14:19:46] "GET /metrics HTTP/1.1" 200 38007 "" "Prometheus/2.43.0" 2024-04-14T14:19:46.817 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:19:46 smithi069 ceph-mon[29472]: pgmap v93: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:19:46.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:19:46 smithi122 ceph-mon[67998]: pgmap v93: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:19:48.067 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:19:47 smithi069 ceph-mon[29472]: pgmap v94: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:19:48.067 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:19:47 smithi069 ceph-mon[24818]: pgmap v94: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:19:48.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:19:47 smithi122 ceph-mon[67998]: pgmap v94: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:19:49.567 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:19:49 smithi069 ceph-mon[24818]: Detected new or changed devices on smithi122 2024-04-14T14:19:49.568 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:19:49 smithi069 ceph-mon[24818]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' 2024-04-14T14:19:49.568 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:19:49 smithi069 ceph-mon[24818]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' 2024-04-14T14:19:49.568 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:19:49 smithi069 ceph-mon[24818]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T14:19:49.568 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:19:49 smithi069 ceph-mon[24818]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-14T14:19:49.568 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:19:49 smithi069 ceph-mon[24818]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' 2024-04-14T14:19:49.568 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:19:49 smithi069 ceph-mon[24818]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-14T14:19:49.568 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:19:49 smithi069 ceph-mon[24818]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T14:19:49.568 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:19:49 smithi069 ceph-mon[24818]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T14:19:49.568 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:19:49 smithi069 ceph-mon[24818]: Upgrade: Setting container_image for all crash 2024-04-14T14:19:49.568 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:19:49 smithi069 ceph-mon[24818]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' 2024-04-14T14:19:49.568 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:19:49 smithi069 ceph-mon[24818]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T14:19:49.568 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:19:49 smithi069 ceph-mon[24818]: Upgrade: Setting container_image for all mds 2024-04-14T14:19:49.568 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:19:49 smithi069 ceph-mon[24818]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' 2024-04-14T14:19:49.568 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:19:49 smithi069 ceph-mon[24818]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T14:19:49.569 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:19:49 smithi069 ceph-mon[24818]: Upgrade: Setting container_image for all rbd-mirror 2024-04-14T14:19:49.569 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:19:49 smithi069 ceph-mon[24818]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' 2024-04-14T14:19:49.569 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:19:49 smithi069 ceph-mon[24818]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T14:19:49.569 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:19:49 smithi069 ceph-mon[24818]: Upgrade: Setting container_image for all cephfs-mirror 2024-04-14T14:19:49.569 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:19:49 smithi069 ceph-mon[24818]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' 2024-04-14T14:19:49.569 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:19:49 smithi069 ceph-mon[24818]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.b"}]: dispatch 2024-04-14T14:19:49.569 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:19:49 smithi069 ceph-mon[24818]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon.b"}]': finished 2024-04-14T14:19:49.569 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:19:49 smithi069 ceph-mon[24818]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T14:19:49.569 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:19:49 smithi069 ceph-mon[24818]: Upgrade: Setting container_image for all ceph-exporter 2024-04-14T14:19:49.569 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:19:49 smithi069 ceph-mon[24818]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' 2024-04-14T14:19:49.569 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:19:49 smithi069 ceph-mon[24818]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T14:19:49.569 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:19:49 smithi069 ceph-mon[24818]: Upgrade: Setting container_image for all nfs 2024-04-14T14:19:49.569 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:19:49 smithi069 ceph-mon[24818]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' 2024-04-14T14:19:49.569 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:19:49 smithi069 ceph-mon[24818]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T14:19:49.570 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:19:49 smithi069 ceph-mon[24818]: Upgrade: Setting container_image for all nvmeof 2024-04-14T14:19:49.570 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:19:49 smithi069 ceph-mon[24818]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' 2024-04-14T14:19:49.570 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:19:49 smithi069 ceph-mon[24818]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T14:19:49.570 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:19:49 smithi069 ceph-mon[24818]: Upgrade: Setting container_image for all node-exporter 2024-04-14T14:19:49.570 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:19:49 smithi069 ceph-mon[29472]: Detected new or changed devices on smithi122 2024-04-14T14:19:49.570 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:19:49 smithi069 ceph-mon[29472]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' 2024-04-14T14:19:49.570 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:19:49 smithi069 ceph-mon[29472]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' 2024-04-14T14:19:49.570 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:19:49 smithi069 ceph-mon[29472]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T14:19:49.570 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:19:49 smithi069 ceph-mon[29472]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-14T14:19:49.570 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:19:49 smithi069 ceph-mon[29472]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' 2024-04-14T14:19:49.571 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:19:49 smithi069 ceph-mon[29472]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-14T14:19:49.571 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:19:49 smithi069 ceph-mon[29472]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T14:19:49.571 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:19:49 smithi069 ceph-mon[29472]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T14:19:49.571 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:19:49 smithi069 ceph-mon[29472]: Upgrade: Setting container_image for all crash 2024-04-14T14:19:49.571 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:19:49 smithi069 ceph-mon[29472]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' 2024-04-14T14:19:49.571 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:19:49 smithi069 ceph-mon[29472]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T14:19:49.571 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:19:49 smithi069 ceph-mon[29472]: Upgrade: Setting container_image for all mds 2024-04-14T14:19:49.571 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:19:49 smithi069 ceph-mon[29472]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' 2024-04-14T14:19:49.571 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:19:49 smithi069 ceph-mon[29472]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T14:19:49.571 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:19:49 smithi069 ceph-mon[29472]: Upgrade: Setting container_image for all rbd-mirror 2024-04-14T14:19:49.571 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:19:49 smithi069 ceph-mon[29472]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' 2024-04-14T14:19:49.571 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:19:49 smithi069 ceph-mon[29472]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T14:19:49.571 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:19:49 smithi069 ceph-mon[29472]: Upgrade: Setting container_image for all cephfs-mirror 2024-04-14T14:19:49.571 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:19:49 smithi069 ceph-mon[29472]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' 2024-04-14T14:19:49.572 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:19:49 smithi069 ceph-mon[29472]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.b"}]: dispatch 2024-04-14T14:19:49.572 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:19:49 smithi069 ceph-mon[29472]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon.b"}]': finished 2024-04-14T14:19:49.572 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:19:49 smithi069 ceph-mon[29472]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T14:19:49.572 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:19:49 smithi069 ceph-mon[29472]: Upgrade: Setting container_image for all ceph-exporter 2024-04-14T14:19:49.572 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:19:49 smithi069 ceph-mon[29472]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' 2024-04-14T14:19:49.572 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:19:49 smithi069 ceph-mon[29472]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T14:19:49.572 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:19:49 smithi069 ceph-mon[29472]: Upgrade: Setting container_image for all nfs 2024-04-14T14:19:49.572 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:19:49 smithi069 ceph-mon[29472]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' 2024-04-14T14:19:49.572 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:19:49 smithi069 ceph-mon[29472]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T14:19:49.572 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:19:49 smithi069 ceph-mon[29472]: Upgrade: Setting container_image for all nvmeof 2024-04-14T14:19:49.572 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:19:49 smithi069 ceph-mon[29472]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' 2024-04-14T14:19:49.572 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:19:49 smithi069 ceph-mon[29472]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T14:19:49.572 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:19:49 smithi069 ceph-mon[29472]: Upgrade: Setting container_image for all node-exporter 2024-04-14T14:19:49.573 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:19:49 smithi069 ceph-mon[24818]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.b"}]: dispatch 2024-04-14T14:19:49.573 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:19:49 smithi069 ceph-mon[24818]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T14:19:49.573 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:19:49 smithi069 ceph-mon[24818]: Upgrade: Setting container_image for all prometheus 2024-04-14T14:19:49.573 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:19:49 smithi069 ceph-mon[24818]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.b"}]: dispatch 2024-04-14T14:19:49.573 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:19:49 smithi069 ceph-mon[24818]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T14:19:49.573 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:19:49 smithi069 ceph-mon[24818]: Upgrade: Setting container_image for all alertmanager 2024-04-14T14:19:49.573 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:19:49 smithi069 ceph-mon[24818]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.b"}]: dispatch 2024-04-14T14:19:49.573 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:19:49 smithi069 ceph-mon[24818]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T14:19:49.573 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:19:49 smithi069 ceph-mon[24818]: Upgrade: Setting container_image for all grafana 2024-04-14T14:19:49.573 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:19:49 smithi069 ceph-mon[24818]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.b"}]: dispatch 2024-04-14T14:19:49.573 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:19:49 smithi069 ceph-mon[24818]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T14:19:49.573 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:19:49 smithi069 ceph-mon[24818]: Upgrade: Setting container_image for all loki 2024-04-14T14:19:49.573 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:19:49 smithi069 ceph-mon[24818]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.b"}]: dispatch 2024-04-14T14:19:49.573 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:19:49 smithi069 ceph-mon[24818]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T14:19:49.574 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:19:49 smithi069 ceph-mon[24818]: Upgrade: Setting container_image for all promtail 2024-04-14T14:19:49.574 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:19:49 smithi069 ceph-mon[24818]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.b"}]: dispatch 2024-04-14T14:19:49.574 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:19:49 smithi069 ceph-mon[24818]: Upgrade: Finalizing container_image settings 2024-04-14T14:19:49.574 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:19:49 smithi069 ceph-mon[24818]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]: dispatch 2024-04-14T14:19:49.574 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:19:49 smithi069 ceph-mon[24818]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]': finished 2024-04-14T14:19:49.574 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:19:49 smithi069 ceph-mon[24818]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-14T14:19:49.574 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:19:49 smithi069 ceph-mon[24818]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon"}]': finished 2024-04-14T14:19:49.574 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:19:49 smithi069 ceph-mon[24818]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]: dispatch 2024-04-14T14:19:49.574 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:19:49 smithi069 ceph-mon[24818]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]': finished 2024-04-14T14:19:49.574 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:19:49 smithi069 ceph-mon[24818]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd"}]: dispatch 2024-04-14T14:19:49.574 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:19:49 smithi069 ceph-mon[24818]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mds"}]: dispatch 2024-04-14T14:19:49.574 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:19:49 smithi069 ceph-mon[24818]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mds"}]': finished 2024-04-14T14:19:49.574 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:19:49 smithi069 ceph-mon[24818]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw"}]: dispatch 2024-04-14T14:19:49.575 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:19:49 smithi069 ceph-mon[29472]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.b"}]: dispatch 2024-04-14T14:19:49.575 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:19:49 smithi069 ceph-mon[29472]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T14:19:49.575 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:19:49 smithi069 ceph-mon[29472]: Upgrade: Setting container_image for all prometheus 2024-04-14T14:19:49.575 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:19:49 smithi069 ceph-mon[29472]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.b"}]: dispatch 2024-04-14T14:19:49.575 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:19:49 smithi069 ceph-mon[29472]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T14:19:49.575 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:19:49 smithi069 ceph-mon[29472]: Upgrade: Setting container_image for all alertmanager 2024-04-14T14:19:49.575 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:19:49 smithi069 ceph-mon[29472]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.b"}]: dispatch 2024-04-14T14:19:49.575 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:19:49 smithi069 ceph-mon[29472]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T14:19:49.575 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:19:49 smithi069 ceph-mon[29472]: Upgrade: Setting container_image for all grafana 2024-04-14T14:19:49.575 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:19:49 smithi069 ceph-mon[29472]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.b"}]: dispatch 2024-04-14T14:19:49.575 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:19:49 smithi069 ceph-mon[29472]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T14:19:49.575 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:19:49 smithi069 ceph-mon[29472]: Upgrade: Setting container_image for all loki 2024-04-14T14:19:49.576 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:19:49 smithi069 ceph-mon[29472]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.b"}]: dispatch 2024-04-14T14:19:49.576 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:19:49 smithi069 ceph-mon[29472]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T14:19:49.576 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:19:49 smithi069 ceph-mon[29472]: Upgrade: Setting container_image for all promtail 2024-04-14T14:19:49.576 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:19:49 smithi069 ceph-mon[29472]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.b"}]: dispatch 2024-04-14T14:19:49.576 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:19:49 smithi069 ceph-mon[29472]: Upgrade: Finalizing container_image settings 2024-04-14T14:19:49.576 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:19:49 smithi069 ceph-mon[29472]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]: dispatch 2024-04-14T14:19:49.576 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:19:49 smithi069 ceph-mon[29472]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]': finished 2024-04-14T14:19:49.576 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:19:49 smithi069 ceph-mon[29472]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-14T14:19:49.576 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:19:49 smithi069 ceph-mon[29472]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon"}]': finished 2024-04-14T14:19:49.576 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:19:49 smithi069 ceph-mon[29472]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]: dispatch 2024-04-14T14:19:49.576 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:19:49 smithi069 ceph-mon[29472]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]': finished 2024-04-14T14:19:49.576 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:19:49 smithi069 ceph-mon[29472]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd"}]: dispatch 2024-04-14T14:19:49.576 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:19:49 smithi069 ceph-mon[29472]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mds"}]: dispatch 2024-04-14T14:19:49.576 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:19:49 smithi069 ceph-mon[29472]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mds"}]': finished 2024-04-14T14:19:49.577 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:19:49 smithi069 ceph-mon[29472]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw"}]: dispatch 2024-04-14T14:19:49.577 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:19:49 smithi069 ceph-mon[24818]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"}]: dispatch 2024-04-14T14:19:49.577 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:19:49 smithi069 ceph-mon[24818]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"}]': finished 2024-04-14T14:19:49.577 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:19:49 smithi069 ceph-mon[24818]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-14T14:19:49.577 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:19:49 smithi069 ceph-mon[24818]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter"}]: dispatch 2024-04-14T14:19:49.577 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:19:49 smithi069 ceph-mon[24818]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter"}]': finished 2024-04-14T14:19:49.577 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:19:49 smithi069 ceph-mon[24818]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.iscsi"}]: dispatch 2024-04-14T14:19:49.577 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:19:49 smithi069 ceph-mon[24818]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nfs"}]: dispatch 2024-04-14T14:19:49.578 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:19:49 smithi069 ceph-mon[24818]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.nfs"}]': finished 2024-04-14T14:19:49.578 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:19:49 smithi069 ceph-mon[24818]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]: dispatch 2024-04-14T14:19:49.578 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:19:49 smithi069 ceph-mon[24818]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]': finished 2024-04-14T14:19:49.578 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:19:49 smithi069 ceph-mon[24818]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-14T14:19:49.578 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:19:49 smithi069 ceph-mon[24818]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-14T14:19:49.578 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:19:49 smithi069 ceph-mon[24818]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-14T14:19:49.578 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:19:49 smithi069 ceph-mon[24818]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-14T14:19:49.578 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:19:49 smithi069 ceph-mon[24818]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-14T14:19:49.578 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:19:49 smithi069 ceph-mon[24818]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-14T14:19:49.578 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:19:49 smithi069 ceph-mon[24818]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' cmd=[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]: dispatch 2024-04-14T14:19:49.578 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:19:49 smithi069 ceph-mon[24818]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' cmd='[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]': finished 2024-04-14T14:19:49.578 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:19:49 smithi069 ceph-mon[24818]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T14:19:49.578 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:19:49 smithi069 ceph-mon[24818]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-14T14:19:49.578 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:19:49 smithi069 ceph-mon[24818]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' 2024-04-14T14:19:49.579 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:19:49 smithi069 ceph-mon[24818]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T14:19:49.579 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:19:49 smithi069 ceph-mon[24818]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-14T14:19:49.579 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:19:49 smithi069 ceph-mon[24818]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' 2024-04-14T14:19:49.579 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:19:49 smithi069 ceph-mon[29472]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"}]: dispatch 2024-04-14T14:19:49.579 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:19:49 smithi069 ceph-mon[29472]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"}]': finished 2024-04-14T14:19:49.579 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:19:49 smithi069 ceph-mon[29472]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-14T14:19:49.579 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:19:49 smithi069 ceph-mon[29472]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter"}]: dispatch 2024-04-14T14:19:49.579 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:19:49 smithi069 ceph-mon[29472]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter"}]': finished 2024-04-14T14:19:49.579 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:19:49 smithi069 ceph-mon[29472]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.iscsi"}]: dispatch 2024-04-14T14:19:49.579 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:19:49 smithi069 ceph-mon[29472]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nfs"}]: dispatch 2024-04-14T14:19:49.580 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:19:49 smithi069 ceph-mon[29472]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.nfs"}]': finished 2024-04-14T14:19:49.580 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:19:49 smithi069 ceph-mon[29472]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]: dispatch 2024-04-14T14:19:49.580 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:19:49 smithi069 ceph-mon[29472]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]': finished 2024-04-14T14:19:49.580 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:19:49 smithi069 ceph-mon[29472]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-14T14:19:49.580 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:19:49 smithi069 ceph-mon[29472]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-14T14:19:49.580 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:19:49 smithi069 ceph-mon[29472]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-14T14:19:49.580 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:19:49 smithi069 ceph-mon[29472]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-14T14:19:49.580 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:19:49 smithi069 ceph-mon[29472]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-14T14:19:49.580 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:19:49 smithi069 ceph-mon[29472]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-14T14:19:49.580 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:19:49 smithi069 ceph-mon[29472]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' cmd=[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]: dispatch 2024-04-14T14:19:49.580 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:19:49 smithi069 ceph-mon[29472]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' cmd='[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]': finished 2024-04-14T14:19:49.580 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:19:49 smithi069 ceph-mon[29472]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T14:19:49.581 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:19:49 smithi069 ceph-mon[29472]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-14T14:19:49.581 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:19:49 smithi069 ceph-mon[29472]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' 2024-04-14T14:19:49.581 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:19:49 smithi069 ceph-mon[29472]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T14:19:49.581 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:19:49 smithi069 ceph-mon[29472]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-14T14:19:49.581 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:19:49 smithi069 ceph-mon[29472]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' 2024-04-14T14:19:49.846 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:19:49 smithi122 ceph-mon[67998]: Detected new or changed devices on smithi122 2024-04-14T14:19:49.846 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:19:49 smithi122 ceph-mon[67998]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' 2024-04-14T14:19:49.846 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:19:49 smithi122 ceph-mon[67998]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' 2024-04-14T14:19:49.846 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:19:49 smithi122 ceph-mon[67998]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T14:19:49.846 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:19:49 smithi122 ceph-mon[67998]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-14T14:19:49.846 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:19:49 smithi122 ceph-mon[67998]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' 2024-04-14T14:19:49.846 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:19:49 smithi122 ceph-mon[67998]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-14T14:19:49.847 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:19:49 smithi122 ceph-mon[67998]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T14:19:49.847 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:19:49 smithi122 ceph-mon[67998]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T14:19:49.847 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:19:49 smithi122 ceph-mon[67998]: Upgrade: Setting container_image for all crash 2024-04-14T14:19:49.847 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:19:49 smithi122 ceph-mon[67998]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' 2024-04-14T14:19:49.847 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:19:49 smithi122 ceph-mon[67998]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T14:19:49.847 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:19:49 smithi122 ceph-mon[67998]: Upgrade: Setting container_image for all mds 2024-04-14T14:19:49.847 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:19:49 smithi122 ceph-mon[67998]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' 2024-04-14T14:19:49.847 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:19:49 smithi122 ceph-mon[67998]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T14:19:49.847 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:19:49 smithi122 ceph-mon[67998]: Upgrade: Setting container_image for all rbd-mirror 2024-04-14T14:19:49.847 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:19:49 smithi122 ceph-mon[67998]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' 2024-04-14T14:19:49.847 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:19:49 smithi122 ceph-mon[67998]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T14:19:49.847 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:19:49 smithi122 ceph-mon[67998]: Upgrade: Setting container_image for all cephfs-mirror 2024-04-14T14:19:49.847 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:19:49 smithi122 ceph-mon[67998]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' 2024-04-14T14:19:49.848 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:19:49 smithi122 ceph-mon[67998]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.b"}]: dispatch 2024-04-14T14:19:49.848 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:19:49 smithi122 ceph-mon[67998]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon.b"}]': finished 2024-04-14T14:19:49.848 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:19:49 smithi122 ceph-mon[67998]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T14:19:49.848 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:19:49 smithi122 ceph-mon[67998]: Upgrade: Setting container_image for all ceph-exporter 2024-04-14T14:19:49.848 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:19:49 smithi122 ceph-mon[67998]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' 2024-04-14T14:19:49.848 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:19:49 smithi122 ceph-mon[67998]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T14:19:49.848 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:19:49 smithi122 ceph-mon[67998]: Upgrade: Setting container_image for all nfs 2024-04-14T14:19:49.848 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:19:49 smithi122 ceph-mon[67998]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' 2024-04-14T14:19:49.848 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:19:49 smithi122 ceph-mon[67998]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T14:19:49.848 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:19:49 smithi122 ceph-mon[67998]: Upgrade: Setting container_image for all nvmeof 2024-04-14T14:19:49.848 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:19:49 smithi122 ceph-mon[67998]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' 2024-04-14T14:19:49.848 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:19:49 smithi122 ceph-mon[67998]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T14:19:49.849 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:19:49 smithi122 ceph-mon[67998]: Upgrade: Setting container_image for all node-exporter 2024-04-14T14:19:49.849 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:19:49 smithi122 ceph-mon[67998]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.b"}]: dispatch 2024-04-14T14:19:49.849 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:19:49 smithi122 ceph-mon[67998]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T14:19:49.849 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:19:49 smithi122 ceph-mon[67998]: Upgrade: Setting container_image for all prometheus 2024-04-14T14:19:49.849 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:19:49 smithi122 ceph-mon[67998]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.b"}]: dispatch 2024-04-14T14:19:49.849 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:19:49 smithi122 ceph-mon[67998]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T14:19:49.849 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:19:49 smithi122 ceph-mon[67998]: Upgrade: Setting container_image for all alertmanager 2024-04-14T14:19:49.849 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:19:49 smithi122 ceph-mon[67998]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.b"}]: dispatch 2024-04-14T14:19:49.849 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:19:49 smithi122 ceph-mon[67998]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T14:19:49.849 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:19:49 smithi122 ceph-mon[67998]: Upgrade: Setting container_image for all grafana 2024-04-14T14:19:49.849 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:19:49 smithi122 ceph-mon[67998]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.b"}]: dispatch 2024-04-14T14:19:49.849 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:19:49 smithi122 ceph-mon[67998]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T14:19:49.849 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:19:49 smithi122 ceph-mon[67998]: Upgrade: Setting container_image for all loki 2024-04-14T14:19:49.850 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:19:49 smithi122 ceph-mon[67998]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.b"}]: dispatch 2024-04-14T14:19:49.850 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:19:49 smithi122 ceph-mon[67998]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T14:19:49.850 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:19:49 smithi122 ceph-mon[67998]: Upgrade: Setting container_image for all promtail 2024-04-14T14:19:49.850 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:19:49 smithi122 ceph-mon[67998]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.b"}]: dispatch 2024-04-14T14:19:49.850 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:19:49 smithi122 ceph-mon[67998]: Upgrade: Finalizing container_image settings 2024-04-14T14:19:49.850 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:19:49 smithi122 ceph-mon[67998]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]: dispatch 2024-04-14T14:19:49.850 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:19:49 smithi122 ceph-mon[67998]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]': finished 2024-04-14T14:19:49.850 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:19:49 smithi122 ceph-mon[67998]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-14T14:19:49.850 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:19:49 smithi122 ceph-mon[67998]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon"}]': finished 2024-04-14T14:19:49.850 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:19:49 smithi122 ceph-mon[67998]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]: dispatch 2024-04-14T14:19:49.850 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:19:49 smithi122 ceph-mon[67998]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]': finished 2024-04-14T14:19:49.850 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:19:49 smithi122 ceph-mon[67998]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd"}]: dispatch 2024-04-14T14:19:49.850 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:19:49 smithi122 ceph-mon[67998]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mds"}]: dispatch 2024-04-14T14:19:49.851 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:19:49 smithi122 ceph-mon[67998]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mds"}]': finished 2024-04-14T14:19:49.851 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:19:49 smithi122 ceph-mon[67998]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw"}]: dispatch 2024-04-14T14:19:49.851 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:19:49 smithi122 ceph-mon[67998]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"}]: dispatch 2024-04-14T14:19:49.851 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:19:49 smithi122 ceph-mon[67998]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"}]': finished 2024-04-14T14:19:49.851 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:19:49 smithi122 ceph-mon[67998]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-14T14:19:49.851 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:19:49 smithi122 ceph-mon[67998]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter"}]: dispatch 2024-04-14T14:19:49.851 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:19:49 smithi122 ceph-mon[67998]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter"}]': finished 2024-04-14T14:19:49.851 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:19:49 smithi122 ceph-mon[67998]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.iscsi"}]: dispatch 2024-04-14T14:19:49.851 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:19:49 smithi122 ceph-mon[67998]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nfs"}]: dispatch 2024-04-14T14:19:49.851 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:19:49 smithi122 ceph-mon[67998]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.nfs"}]': finished 2024-04-14T14:19:49.851 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:19:49 smithi122 ceph-mon[67998]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]: dispatch 2024-04-14T14:19:49.851 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:19:49 smithi122 ceph-mon[67998]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]': finished 2024-04-14T14:19:49.851 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:19:49 smithi122 ceph-mon[67998]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-14T14:19:49.852 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:19:49 smithi122 ceph-mon[67998]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-14T14:19:49.852 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:19:49 smithi122 ceph-mon[67998]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-14T14:19:49.852 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:19:49 smithi122 ceph-mon[67998]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-14T14:19:49.852 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:19:49 smithi122 ceph-mon[67998]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-14T14:19:49.852 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:19:49 smithi122 ceph-mon[67998]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-14T14:19:49.852 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:19:49 smithi122 ceph-mon[67998]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' cmd=[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]: dispatch 2024-04-14T14:19:49.852 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:19:49 smithi122 ceph-mon[67998]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' cmd='[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]': finished 2024-04-14T14:19:49.852 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:19:49 smithi122 ceph-mon[67998]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T14:19:49.852 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:19:49 smithi122 ceph-mon[67998]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-14T14:19:49.852 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:19:49 smithi122 ceph-mon[67998]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' 2024-04-14T14:19:49.852 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:19:49 smithi122 ceph-mon[67998]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T14:19:49.852 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:19:49 smithi122 ceph-mon[67998]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-14T14:19:49.852 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:19:49 smithi122 ceph-mon[67998]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' 2024-04-14T14:19:50.817 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:19:50 smithi069 ceph-mon[24818]: pgmap v95: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:19:50.817 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:19:50 smithi069 ceph-mon[24818]: Upgrade: Complete! 2024-04-14T14:19:50.817 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:19:50 smithi069 ceph-mon[24818]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' 2024-04-14T14:19:50.818 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:19:50 smithi069 ceph-mon[29472]: pgmap v95: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:19:50.818 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:19:50 smithi069 ceph-mon[29472]: Upgrade: Complete! 2024-04-14T14:19:50.818 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:19:50 smithi069 ceph-mon[29472]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' 2024-04-14T14:19:50.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:19:50 smithi122 ceph-mon[67998]: pgmap v95: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:19:50.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:19:50 smithi122 ceph-mon[67998]: Upgrade: Complete! 2024-04-14T14:19:50.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:19:50 smithi122 ceph-mon[67998]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' 2024-04-14T14:19:53.067 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:19:52 smithi069 ceph-mon[24818]: pgmap v96: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:19:53.067 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:19:52 smithi069 ceph-mon[24818]: from='client.25417 -' entity='client.iscsi.foo.smithi069.yuumtr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T14:19:53.068 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:19:52 smithi069 ceph-mon[29472]: pgmap v96: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:19:53.068 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:19:52 smithi069 ceph-mon[29472]: from='client.25417 -' entity='client.iscsi.foo.smithi069.yuumtr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T14:19:53.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:19:52 smithi122 ceph-mon[67998]: pgmap v96: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:19:53.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:19:52 smithi122 ceph-mon[67998]: from='client.25417 -' entity='client.iscsi.foo.smithi069.yuumtr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T14:19:55.067 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:19:54 smithi069 ceph-mon[24818]: pgmap v97: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:19:55.067 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:19:54 smithi069 ceph-mon[29472]: pgmap v97: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:19:55.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:19:54 smithi122 ceph-mon[67998]: pgmap v97: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:19:56.818 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:19:56 smithi069 ceph-mon[24818]: pgmap v98: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:19:56.818 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:19:56 smithi069 ceph-mon[29472]: pgmap v98: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:19:56.818 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 14 14:19:56 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-y[63038]: ::ffff:172.21.15.122 - - [14/Apr/2024:14:19:56] "GET /metrics HTTP/1.1" 200 38007 "" "Prometheus/2.43.0" 2024-04-14T14:19:56.897 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:19:56 smithi122 ceph-mon[67998]: pgmap v98: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:19:59.067 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:19:58 smithi069 ceph-mon[24818]: pgmap v99: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:19:59.067 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:19:58 smithi069 ceph-mon[29472]: pgmap v99: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:19:59.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:19:58 smithi122 ceph-mon[67998]: pgmap v99: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:20:01.005 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:20:00 smithi069 ceph-mon[24818]: pgmap v100: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:20:01.005 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:20:00 smithi069 ceph-mon[24818]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T14:20:01.005 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:20:00 smithi069 ceph-mon[24818]: overall HEALTH_OK 2024-04-14T14:20:01.006 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:20:00 smithi069 ceph-mon[29472]: pgmap v100: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:20:01.006 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:20:00 smithi069 ceph-mon[29472]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T14:20:01.006 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:20:00 smithi069 ceph-mon[29472]: overall HEALTH_OK 2024-04-14T14:20:01.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:20:00 smithi122 ceph-mon[67998]: pgmap v100: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:20:01.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:20:00 smithi122 ceph-mon[67998]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T14:20:01.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:20:00 smithi122 ceph-mon[67998]: overall HEALTH_OK 2024-04-14T14:20:03.067 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:20:02 smithi069 ceph-mon[24818]: pgmap v101: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:20:03.068 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:20:02 smithi069 ceph-mon[24818]: from='client.25417 -' entity='client.iscsi.foo.smithi069.yuumtr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T14:20:03.068 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:20:02 smithi069 ceph-mon[29472]: pgmap v101: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:20:03.068 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:20:02 smithi069 ceph-mon[29472]: from='client.25417 -' entity='client.iscsi.foo.smithi069.yuumtr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T14:20:03.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:20:02 smithi122 ceph-mon[67998]: pgmap v101: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:20:03.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:20:02 smithi122 ceph-mon[67998]: from='client.25417 -' entity='client.iscsi.foo.smithi069.yuumtr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T14:20:05.067 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:20:04 smithi069 ceph-mon[24818]: pgmap v102: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:20:05.067 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:20:04 smithi069 ceph-mon[29472]: pgmap v102: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:20:05.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:20:04 smithi122 ceph-mon[67998]: pgmap v102: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:20:05.760 DEBUG:teuthology.orchestra.run.smithi069:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid dc326fc2-fa65-11ee-bc8e-c7b262605968 -e sha1=a971fd5f5719038c70b817bafacd6d5400b1d888 -- bash -c 'ceph orch ps' 2024-04-14T14:20:06.636 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 14 14:20:06 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-y[63038]: ::ffff:172.21.15.122 - - [14/Apr/2024:14:20:06] "GET /metrics HTTP/1.1" 200 38004 "" "Prometheus/2.43.0" 2024-04-14T14:20:07.059 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:20:06 smithi069 ceph-mon[24818]: pgmap v103: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:20:07.059 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:20:06 smithi069 ceph-mon[24818]: from='client.15708 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T14:20:07.059 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:20:06 smithi069 ceph-mon[29472]: pgmap v103: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:20:07.059 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:20:06 smithi069 ceph-mon[29472]: from='client.15708 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T14:20:07.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:20:06 smithi122 ceph-mon[67998]: pgmap v103: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:20:07.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:20:06 smithi122 ceph-mon[67998]: from='client.15708 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T14:20:07.778 INFO:teuthology.orchestra.run.smithi069.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-14T14:20:07.778 INFO:teuthology.orchestra.run.smithi069.stdout:alertmanager.a smithi069 *:9093,9094 running (112s) 107s ago 23m 16.5M - 0.25.0 c8568f914cd2 7524f106e7bd 2024-04-14T14:20:07.778 INFO:teuthology.orchestra.run.smithi069.stdout:grafana.a smithi122 *:3000 running (98s) 23s ago 23m 75.7M - 9.4.12 2bacad6d85d8 5685178e3eba 2024-04-14T14:20:07.778 INFO:teuthology.orchestra.run.smithi069.stdout:iscsi.foo.smithi069.yuumtr smithi069 running (2m) 107s ago 22m 44.5M - 3.5 e1d6a67b021e 38cc81ea8cfa 2024-04-14T14:20:07.778 INFO:teuthology.orchestra.run.smithi069.stdout:mgr.x smithi122 *:8443,9283,8765 running (2m) 23s ago 28m 451M - 19.0.0-2778-ga971fd5f b96c509eef84 e29977590b43 2024-04-14T14:20:07.778 INFO:teuthology.orchestra.run.smithi069.stdout:mgr.y smithi069 *:8443,9283,8765 running (12m) 107s ago 29m 504M - 19.0.0-2778-ga971fd5f b96c509eef84 7af3d29d0e5f 2024-04-14T14:20:07.778 INFO:teuthology.orchestra.run.smithi069.stdout:mon.a smithi069 running (29m) 107s ago 29m 78.2M 2048M 17.2.0 e1d6a67b021e d3c553f91913 2024-04-14T14:20:07.779 INFO:teuthology.orchestra.run.smithi069.stdout:mon.b smithi122 running (28s) 23s ago 28m 17.2M 2048M 19.0.0-2778-ga971fd5f b96c509eef84 05f437a1a9a1 2024-04-14T14:20:07.779 INFO:teuthology.orchestra.run.smithi069.stdout:mon.c smithi069 running (28m) 107s ago 28m 57.4M 2048M 17.2.0 e1d6a67b021e 23c078b9d1e5 2024-04-14T14:20:07.779 INFO:teuthology.orchestra.run.smithi069.stdout:node-exporter.a smithi069 *:9100 running (2m) 107s ago 24m 8258k - 1.5.0 0da6a335fe13 0de0eb2171b9 2024-04-14T14:20:07.779 INFO:teuthology.orchestra.run.smithi069.stdout:node-exporter.b smithi122 *:9100 running (2m) 23s ago 24m 16.1M - 1.5.0 0da6a335fe13 67877bf02d8e 2024-04-14T14:20:07.779 INFO:teuthology.orchestra.run.smithi069.stdout:osd.0 smithi069 running (27m) 107s ago 27m 56.4M 2992M 17.2.0 e1d6a67b021e 46654654b6b6 2024-04-14T14:20:07.779 INFO:teuthology.orchestra.run.smithi069.stdout:osd.1 smithi069 running (27m) 107s ago 27m 54.5M 2992M 17.2.0 e1d6a67b021e cdba54a59609 2024-04-14T14:20:07.779 INFO:teuthology.orchestra.run.smithi069.stdout:osd.2 smithi069 running (26m) 107s ago 26m 50.9M 2992M 17.2.0 e1d6a67b021e bc37a1a9cba0 2024-04-14T14:20:07.779 INFO:teuthology.orchestra.run.smithi069.stdout:osd.3 smithi069 running (26m) 107s ago 26m 55.2M 2992M 17.2.0 e1d6a67b021e 0925ba3da801 2024-04-14T14:20:07.779 INFO:teuthology.orchestra.run.smithi069.stdout:osd.4 smithi122 running (26m) 23s ago 26m 56.1M 3248M 17.2.0 e1d6a67b021e c00975caf771 2024-04-14T14:20:07.779 INFO:teuthology.orchestra.run.smithi069.stdout:osd.5 smithi122 running (25m) 23s ago 25m 55.2M 3248M 17.2.0 e1d6a67b021e 5f52d217ceb8 2024-04-14T14:20:07.779 INFO:teuthology.orchestra.run.smithi069.stdout:osd.6 smithi122 running (25m) 23s ago 25m 52.2M 3248M 17.2.0 e1d6a67b021e a2acfec61475 2024-04-14T14:20:07.779 INFO:teuthology.orchestra.run.smithi069.stdout:osd.7 smithi122 running (25m) 23s ago 25m 57.0M 3248M 17.2.0 e1d6a67b021e 62411cd06801 2024-04-14T14:20:07.780 INFO:teuthology.orchestra.run.smithi069.stdout:prometheus.a smithi122 *:9095 running (2m) 23s ago 24m 65.6M - 2.43.0 a07b618ecd1d 5c1f41d3bbe6 2024-04-14T14:20:07.780 INFO:teuthology.orchestra.run.smithi069.stdout:rgw.foo.smithi069.wesedz smithi069 *:8000 running (22m) 107s ago 22m 77.8M - 17.2.0 e1d6a67b021e 16de491440f3 2024-04-14T14:20:07.780 INFO:teuthology.orchestra.run.smithi069.stdout:rgw.foo.smithi122.jiobyv smithi122 *:8000 running (22m) 23s ago 22m 75.4M - 17.2.0 e1d6a67b021e 6d16de445dba 2024-04-14T14:20:08.381 DEBUG:teuthology.orchestra.run.smithi069:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid dc326fc2-fa65-11ee-bc8e-c7b262605968 -e sha1=a971fd5f5719038c70b817bafacd6d5400b1d888 -- bash -c 'ceph versions | jq -e '"'"'.mon | length == 2'"'"'' 2024-04-14T14:20:08.936 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:20:08 smithi069 ceph-mon[24818]: pgmap v104: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:20:08.937 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:20:08 smithi069 ceph-mon[29472]: pgmap v104: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:20:09.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:20:08 smithi122 ceph-mon[67998]: pgmap v104: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:20:09.817 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:20:09 smithi069 ceph-mon[24818]: from='client.15714 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T14:20:09.817 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:20:09 smithi069 ceph-mon[29472]: from='client.15714 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T14:20:10.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:20:09 smithi122 ceph-mon[67998]: from='client.15714 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T14:20:10.450 INFO:teuthology.orchestra.run.smithi069.stdout:true 2024-04-14T14:20:10.797 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:20:10 smithi069 ceph-mon[24818]: pgmap v105: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:20:10.797 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:20:10 smithi069 ceph-mon[24818]: from='client.? 172.21.15.69:0/2786585618' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T14:20:10.798 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:20:10 smithi069 ceph-mon[29472]: pgmap v105: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:20:10.798 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:20:10 smithi069 ceph-mon[29472]: from='client.? 172.21.15.69:0/2786585618' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T14:20:11.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:20:10 smithi122 ceph-mon[67998]: pgmap v105: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:20:11.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:20:10 smithi122 ceph-mon[67998]: from='client.? 172.21.15.69:0/2786585618' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T14:20:11.142 DEBUG:teuthology.orchestra.run.smithi069:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid dc326fc2-fa65-11ee-bc8e-c7b262605968 -e sha1=a971fd5f5719038c70b817bafacd6d5400b1d888 -- bash -c 'ceph orch upgrade status' 2024-04-14T14:20:13.067 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:20:12 smithi069 ceph-mon[24818]: pgmap v106: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:20:13.067 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:20:12 smithi069 ceph-mon[24818]: from='client.25417 -' entity='client.iscsi.foo.smithi069.yuumtr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T14:20:13.068 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:20:12 smithi069 ceph-mon[29472]: pgmap v106: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:20:13.068 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:20:12 smithi069 ceph-mon[29472]: from='client.25417 -' entity='client.iscsi.foo.smithi069.yuumtr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T14:20:13.088 INFO:teuthology.orchestra.run.smithi069.stdout:{ 2024-04-14T14:20:13.088 INFO:teuthology.orchestra.run.smithi069.stdout: "target_image": null, 2024-04-14T14:20:13.088 INFO:teuthology.orchestra.run.smithi069.stdout: "in_progress": false, 2024-04-14T14:20:13.088 INFO:teuthology.orchestra.run.smithi069.stdout: "which": "", 2024-04-14T14:20:13.088 INFO:teuthology.orchestra.run.smithi069.stdout: "services_complete": [], 2024-04-14T14:20:13.089 INFO:teuthology.orchestra.run.smithi069.stdout: "progress": null, 2024-04-14T14:20:13.089 INFO:teuthology.orchestra.run.smithi069.stdout: "message": "", 2024-04-14T14:20:13.089 INFO:teuthology.orchestra.run.smithi069.stdout: "is_paused": false 2024-04-14T14:20:13.089 INFO:teuthology.orchestra.run.smithi069.stdout:} 2024-04-14T14:20:13.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:20:12 smithi122 ceph-mon[67998]: pgmap v106: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:20:13.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:20:12 smithi122 ceph-mon[67998]: from='client.25417 -' entity='client.iscsi.foo.smithi069.yuumtr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T14:20:13.610 DEBUG:teuthology.orchestra.run.smithi069:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid dc326fc2-fa65-11ee-bc8e-c7b262605968 -e sha1=a971fd5f5719038c70b817bafacd6d5400b1d888 -- bash -c 'ceph health detail' 2024-04-14T14:20:14.941 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:20:14 smithi069 ceph-mon[24818]: pgmap v107: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:20:14.942 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:20:14 smithi069 ceph-mon[24818]: from='client.25624 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T14:20:14.942 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:20:14 smithi069 ceph-mon[29472]: pgmap v107: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:20:14.942 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:20:14 smithi069 ceph-mon[29472]: from='client.25624 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T14:20:15.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:20:14 smithi122 ceph-mon[67998]: pgmap v107: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:20:15.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:20:14 smithi122 ceph-mon[67998]: from='client.25624 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T14:20:15.620 INFO:teuthology.orchestra.run.smithi069.stdout:HEALTH_OK 2024-04-14T14:20:15.942 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:20:15 smithi069 ceph-mon[24818]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T14:20:15.942 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:20:15 smithi069 ceph-mon[24818]: from='client.? 172.21.15.69:0/2331195271' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-14T14:20:15.942 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:20:15 smithi069 ceph-mon[29472]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T14:20:15.942 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:20:15 smithi069 ceph-mon[29472]: from='client.? 172.21.15.69:0/2331195271' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-14T14:20:16.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:20:15 smithi122 ceph-mon[67998]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T14:20:16.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:20:15 smithi122 ceph-mon[67998]: from='client.? 172.21.15.69:0/2331195271' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-14T14:20:16.233 DEBUG:teuthology.orchestra.run.smithi069:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid dc326fc2-fa65-11ee-bc8e-c7b262605968 -e sha1=a971fd5f5719038c70b817bafacd6d5400b1d888 -- bash -c 'ceph orch upgrade start --image quay.ceph.io/ceph-ci/ceph:$sha1 --daemon-types mon --hosts $(ceph orch ps | grep mgr.y | awk '"'"'{print $2}'"'"')' 2024-04-14T14:20:16.787 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 14 14:20:16 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-y[63038]: ::ffff:172.21.15.122 - - [14/Apr/2024:14:20:16] "GET /metrics HTTP/1.1" 200 38006 "" "Prometheus/2.43.0" 2024-04-14T14:20:16.787 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:20:16 smithi069 ceph-mon[24818]: pgmap v108: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:20:16.787 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:20:16 smithi069 ceph-mon[29472]: pgmap v108: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:20:17.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:20:16 smithi122 ceph-mon[67998]: pgmap v108: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:20:19.067 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:20:18 smithi069 ceph-mon[24818]: pgmap v109: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:20:19.067 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:20:18 smithi069 ceph-mon[29472]: pgmap v109: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:20:19.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:20:18 smithi122 ceph-mon[67998]: pgmap v109: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:20:20.051 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:20:19 smithi069 ceph-mon[24818]: from='client.15738 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T14:20:20.051 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:20:19 smithi069 ceph-mon[24818]: from='client.15744 -' entity='client.admin' cmd=[{"prefix": "orch upgrade start", "image": "quay.ceph.io/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888", "daemon_types": "mon", "hosts": "smithi069", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T14:20:20.052 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:20:19 smithi069 ceph-mon[29472]: from='client.15738 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T14:20:20.052 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:20:19 smithi069 ceph-mon[29472]: from='client.15744 -' entity='client.admin' cmd=[{"prefix": "orch upgrade start", "image": "quay.ceph.io/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888", "daemon_types": "mon", "hosts": "smithi069", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T14:20:20.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:20:19 smithi122 ceph-mon[67998]: from='client.15738 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T14:20:20.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:20:19 smithi122 ceph-mon[67998]: from='client.15744 -' entity='client.admin' cmd=[{"prefix": "orch upgrade start", "image": "quay.ceph.io/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888", "daemon_types": "mon", "hosts": "smithi069", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T14:20:20.110 INFO:teuthology.orchestra.run.smithi069.stdout:Initiating upgrade to quay.ceph.io/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888 2024-04-14T14:20:20.797 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:20:20 smithi069 ceph-mon[29472]: pgmap v110: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:20:20.797 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:20:20 smithi069 ceph-mon[29472]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' 2024-04-14T14:20:20.798 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:20:20 smithi069 ceph-mon[29472]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T14:20:20.798 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:20:20 smithi069 ceph-mon[29472]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-14T14:20:20.798 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:20:20 smithi069 ceph-mon[29472]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' 2024-04-14T14:20:20.798 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:20:20 smithi069 ceph-mon[24818]: pgmap v110: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:20:20.798 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:20:20 smithi069 ceph-mon[24818]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' 2024-04-14T14:20:20.798 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:20:20 smithi069 ceph-mon[24818]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T14:20:20.798 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:20:20 smithi069 ceph-mon[24818]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-14T14:20:20.798 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:20:20 smithi069 ceph-mon[24818]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' 2024-04-14T14:20:20.821 DEBUG:teuthology.orchestra.run.smithi069:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid dc326fc2-fa65-11ee-bc8e-c7b262605968 -e sha1=a971fd5f5719038c70b817bafacd6d5400b1d888 -- bash -c 'while ceph orch upgrade status | jq '"'"'.in_progress'"'"' | grep true && ! ceph orch upgrade status | jq '"'"'.message'"'"' | grep Error ; do ceph orch ps ; ceph versions ; ceph orch upgrade status ; sleep 30 ; done' 2024-04-14T14:20:21.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:20:20 smithi122 ceph-mon[67998]: pgmap v110: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:20:21.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:20:20 smithi122 ceph-mon[67998]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' 2024-04-14T14:20:21.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:20:20 smithi122 ceph-mon[67998]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T14:20:21.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:20:20 smithi122 ceph-mon[67998]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-14T14:20:21.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:20:20 smithi122 ceph-mon[67998]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' 2024-04-14T14:20:22.025 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:20:21 smithi069 ceph-mon[24818]: Upgrade: Started with target quay.ceph.io/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888 2024-04-14T14:20:22.025 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:20:21 smithi069 ceph-mon[24818]: Upgrade: First pull of quay.ceph.io/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888 2024-04-14T14:20:22.025 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:20:21 smithi069 ceph-mon[29472]: Upgrade: Started with target quay.ceph.io/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888 2024-04-14T14:20:22.025 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:20:21 smithi069 ceph-mon[29472]: Upgrade: First pull of quay.ceph.io/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888 2024-04-14T14:20:22.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:20:21 smithi122 ceph-mon[67998]: Upgrade: Started with target quay.ceph.io/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888 2024-04-14T14:20:22.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:20:21 smithi122 ceph-mon[67998]: Upgrade: First pull of quay.ceph.io/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888 2024-04-14T14:20:22.888 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:20:22 smithi069 ceph-mon[24818]: pgmap v111: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:20:22.888 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:20:22 smithi069 ceph-mon[24818]: from='client.25417 -' entity='client.iscsi.foo.smithi069.yuumtr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T14:20:22.889 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:20:22 smithi069 ceph-mon[24818]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' 2024-04-14T14:20:22.889 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:20:22 smithi069 ceph-mon[24818]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-14T14:20:22.889 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:20:22 smithi069 ceph-mon[24818]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T14:20:22.889 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:20:22 smithi069 ceph-mon[24818]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' 2024-04-14T14:20:22.889 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:20:22 smithi069 ceph-mon[24818]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' cmd=[{"prefix": "quorum_status"}]: dispatch 2024-04-14T14:20:22.889 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:20:22 smithi069 ceph-mon[24818]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' cmd=[{"prefix": "mon ok-to-stop", "ids": ["a"]}]: dispatch 2024-04-14T14:20:22.890 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:20:22 smithi069 ceph-mon[29472]: pgmap v111: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:20:22.890 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:20:22 smithi069 ceph-mon[29472]: from='client.25417 -' entity='client.iscsi.foo.smithi069.yuumtr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T14:20:22.890 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:20:22 smithi069 ceph-mon[29472]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' 2024-04-14T14:20:22.890 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:20:22 smithi069 ceph-mon[29472]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-14T14:20:22.890 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:20:22 smithi069 ceph-mon[29472]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T14:20:22.890 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:20:22 smithi069 ceph-mon[29472]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' 2024-04-14T14:20:22.890 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:20:22 smithi069 ceph-mon[29472]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' cmd=[{"prefix": "quorum_status"}]: dispatch 2024-04-14T14:20:22.890 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:20:22 smithi069 ceph-mon[29472]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' cmd=[{"prefix": "mon ok-to-stop", "ids": ["a"]}]: dispatch 2024-04-14T14:20:23.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:20:22 smithi122 ceph-mon[67998]: pgmap v111: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:20:23.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:20:22 smithi122 ceph-mon[67998]: from='client.25417 -' entity='client.iscsi.foo.smithi069.yuumtr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T14:20:23.096 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:20:22 smithi122 ceph-mon[67998]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' 2024-04-14T14:20:23.096 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:20:22 smithi122 ceph-mon[67998]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-14T14:20:23.096 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:20:22 smithi122 ceph-mon[67998]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T14:20:23.096 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:20:22 smithi122 ceph-mon[67998]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' 2024-04-14T14:20:23.096 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:20:22 smithi122 ceph-mon[67998]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' cmd=[{"prefix": "quorum_status"}]: dispatch 2024-04-14T14:20:23.096 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:20:22 smithi122 ceph-mon[67998]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' cmd=[{"prefix": "mon ok-to-stop", "ids": ["a"]}]: dispatch 2024-04-14T14:20:23.609 INFO:teuthology.orchestra.run.smithi069.stdout:true 2024-04-14T14:20:23.748 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:20:23 smithi069 ceph-mon[24818]: Upgrade: Target is version 19.0.0-2778-ga971fd5f (squid) 2024-04-14T14:20:23.748 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:20:23 smithi069 ceph-mon[24818]: Upgrade: Target container is quay.ceph.io/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888, digests ['quay.ceph.io/ceph-ci/ceph@sha256:02f0a191d4d693952654c64f93cc9c0152602c0915251b3c94c2c4e32855c608'] 2024-04-14T14:20:23.748 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:20:23 smithi069 ceph-mon[24818]: Upgrade: Setting container_image for all mgr 2024-04-14T14:20:23.748 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:20:23 smithi069 ceph-mon[24818]: Upgrade: It appears safe to stop mon.a 2024-04-14T14:20:23.748 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:20:23 smithi069 ceph-mon[29472]: Upgrade: Target is version 19.0.0-2778-ga971fd5f (squid) 2024-04-14T14:20:23.748 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:20:23 smithi069 ceph-mon[29472]: Upgrade: Target container is quay.ceph.io/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888, digests ['quay.ceph.io/ceph-ci/ceph@sha256:02f0a191d4d693952654c64f93cc9c0152602c0915251b3c94c2c4e32855c608'] 2024-04-14T14:20:23.748 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:20:23 smithi069 ceph-mon[29472]: Upgrade: Setting container_image for all mgr 2024-04-14T14:20:23.748 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:20:23 smithi069 ceph-mon[29472]: Upgrade: It appears safe to stop mon.a 2024-04-14T14:20:24.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:20:23 smithi122 ceph-mon[67998]: Upgrade: Target is version 19.0.0-2778-ga971fd5f (squid) 2024-04-14T14:20:24.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:20:23 smithi122 ceph-mon[67998]: Upgrade: Target container is quay.ceph.io/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888, digests ['quay.ceph.io/ceph-ci/ceph@sha256:02f0a191d4d693952654c64f93cc9c0152602c0915251b3c94c2c4e32855c608'] 2024-04-14T14:20:24.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:20:23 smithi122 ceph-mon[67998]: Upgrade: Setting container_image for all mgr 2024-04-14T14:20:24.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:20:23 smithi122 ceph-mon[67998]: Upgrade: It appears safe to stop mon.a 2024-04-14T14:20:24.292 INFO:teuthology.orchestra.run.smithi069.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-14T14:20:24.292 INFO:teuthology.orchestra.run.smithi069.stdout:alertmanager.a smithi069 *:9093,9094 running (2m) 2m ago 23m 16.5M - 0.25.0 c8568f914cd2 7524f106e7bd 2024-04-14T14:20:24.293 INFO:teuthology.orchestra.run.smithi069.stdout:grafana.a smithi122 *:3000 running (114s) 40s ago 23m 75.7M - 9.4.12 2bacad6d85d8 5685178e3eba 2024-04-14T14:20:24.293 INFO:teuthology.orchestra.run.smithi069.stdout:iscsi.foo.smithi069.yuumtr smithi069 running (3m) 2m ago 23m 44.5M - 3.5 e1d6a67b021e 38cc81ea8cfa 2024-04-14T14:20:24.293 INFO:teuthology.orchestra.run.smithi069.stdout:mgr.x smithi122 *:8443,9283,8765 running (3m) 40s ago 28m 451M - 19.0.0-2778-ga971fd5f b96c509eef84 e29977590b43 2024-04-14T14:20:24.294 INFO:teuthology.orchestra.run.smithi069.stdout:mgr.y smithi069 *:8443,9283,8765 running (12m) 2m ago 29m 504M - 19.0.0-2778-ga971fd5f b96c509eef84 7af3d29d0e5f 2024-04-14T14:20:24.294 INFO:teuthology.orchestra.run.smithi069.stdout:mon.a smithi069 running (29m) 2m ago 30m 78.2M 2048M 17.2.0 e1d6a67b021e d3c553f91913 2024-04-14T14:20:24.294 INFO:teuthology.orchestra.run.smithi069.stdout:mon.b smithi122 running (44s) 40s ago 28m 17.2M 2048M 19.0.0-2778-ga971fd5f b96c509eef84 05f437a1a9a1 2024-04-14T14:20:24.294 INFO:teuthology.orchestra.run.smithi069.stdout:mon.c smithi069 running (28m) 2m ago 28m 57.4M 2048M 17.2.0 e1d6a67b021e 23c078b9d1e5 2024-04-14T14:20:24.294 INFO:teuthology.orchestra.run.smithi069.stdout:node-exporter.a smithi069 *:9100 running (2m) 2m ago 24m 8258k - 1.5.0 0da6a335fe13 0de0eb2171b9 2024-04-14T14:20:24.294 INFO:teuthology.orchestra.run.smithi069.stdout:node-exporter.b smithi122 *:9100 running (2m) 40s ago 24m 16.1M - 1.5.0 0da6a335fe13 67877bf02d8e 2024-04-14T14:20:24.295 INFO:teuthology.orchestra.run.smithi069.stdout:osd.0 smithi069 running (28m) 2m ago 28m 56.4M 2992M 17.2.0 e1d6a67b021e 46654654b6b6 2024-04-14T14:20:24.295 INFO:teuthology.orchestra.run.smithi069.stdout:osd.1 smithi069 running (27m) 2m ago 27m 54.5M 2992M 17.2.0 e1d6a67b021e cdba54a59609 2024-04-14T14:20:24.295 INFO:teuthology.orchestra.run.smithi069.stdout:osd.2 smithi069 running (27m) 2m ago 27m 50.9M 2992M 17.2.0 e1d6a67b021e bc37a1a9cba0 2024-04-14T14:20:24.295 INFO:teuthology.orchestra.run.smithi069.stdout:osd.3 smithi069 running (26m) 2m ago 26m 55.2M 2992M 17.2.0 e1d6a67b021e 0925ba3da801 2024-04-14T14:20:24.295 INFO:teuthology.orchestra.run.smithi069.stdout:osd.4 smithi122 running (26m) 40s ago 26m 56.1M 3248M 17.2.0 e1d6a67b021e c00975caf771 2024-04-14T14:20:24.295 INFO:teuthology.orchestra.run.smithi069.stdout:osd.5 smithi122 running (26m) 40s ago 26m 55.2M 3248M 17.2.0 e1d6a67b021e 5f52d217ceb8 2024-04-14T14:20:24.296 INFO:teuthology.orchestra.run.smithi069.stdout:osd.6 smithi122 running (25m) 40s ago 25m 52.2M 3248M 17.2.0 e1d6a67b021e a2acfec61475 2024-04-14T14:20:24.296 INFO:teuthology.orchestra.run.smithi069.stdout:osd.7 smithi122 running (25m) 40s ago 25m 57.0M 3248M 17.2.0 e1d6a67b021e 62411cd06801 2024-04-14T14:20:24.296 INFO:teuthology.orchestra.run.smithi069.stdout:prometheus.a smithi122 *:9095 running (2m) 40s ago 24m 65.6M - 2.43.0 a07b618ecd1d 5c1f41d3bbe6 2024-04-14T14:20:24.296 INFO:teuthology.orchestra.run.smithi069.stdout:rgw.foo.smithi069.wesedz smithi069 *:8000 running (23m) 2m ago 23m 77.8M - 17.2.0 e1d6a67b021e 16de491440f3 2024-04-14T14:20:24.296 INFO:teuthology.orchestra.run.smithi069.stdout:rgw.foo.smithi122.jiobyv smithi122 *:8000 running (23m) 40s ago 23m 75.4M - 17.2.0 e1d6a67b021e 6d16de445dba 2024-04-14T14:20:24.706 INFO:teuthology.orchestra.run.smithi069.stdout:{ 2024-04-14T14:20:24.707 INFO:teuthology.orchestra.run.smithi069.stdout: "mon": { 2024-04-14T14:20:24.707 INFO:teuthology.orchestra.run.smithi069.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2, 2024-04-14T14:20:24.707 INFO:teuthology.orchestra.run.smithi069.stdout: "ceph version 19.0.0-2778-ga971fd5f (a971fd5f5719038c70b817bafacd6d5400b1d888) squid (dev)": 1 2024-04-14T14:20:24.707 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-14T14:20:24.707 INFO:teuthology.orchestra.run.smithi069.stdout: "mgr": { 2024-04-14T14:20:24.707 INFO:teuthology.orchestra.run.smithi069.stdout: "ceph version 19.0.0-2778-ga971fd5f (a971fd5f5719038c70b817bafacd6d5400b1d888) squid (dev)": 2 2024-04-14T14:20:24.707 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-14T14:20:24.707 INFO:teuthology.orchestra.run.smithi069.stdout: "osd": { 2024-04-14T14:20:24.707 INFO:teuthology.orchestra.run.smithi069.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 8 2024-04-14T14:20:24.707 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-14T14:20:24.707 INFO:teuthology.orchestra.run.smithi069.stdout: "mds": {}, 2024-04-14T14:20:24.707 INFO:teuthology.orchestra.run.smithi069.stdout: "rgw": { 2024-04-14T14:20:24.707 INFO:teuthology.orchestra.run.smithi069.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-04-14T14:20:24.708 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-14T14:20:24.708 INFO:teuthology.orchestra.run.smithi069.stdout: "overall": { 2024-04-14T14:20:24.708 INFO:teuthology.orchestra.run.smithi069.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 12, 2024-04-14T14:20:24.708 INFO:teuthology.orchestra.run.smithi069.stdout: "ceph version 19.0.0-2778-ga971fd5f (a971fd5f5719038c70b817bafacd6d5400b1d888) squid (dev)": 3 2024-04-14T14:20:24.708 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-14T14:20:24.708 INFO:teuthology.orchestra.run.smithi069.stdout:} 2024-04-14T14:20:25.042 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:20:24 smithi069 ceph-mon[24818]: pgmap v112: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:20:25.042 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:20:24 smithi069 ceph-mon[24818]: from='client.25648 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T14:20:25.042 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:20:24 smithi069 ceph-mon[24818]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' 2024-04-14T14:20:25.042 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:20:24 smithi069 ceph-mon[24818]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-14T14:20:25.042 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:20:24 smithi069 ceph-mon[24818]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-14T14:20:25.042 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:20:24 smithi069 ceph-mon[24818]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T14:20:25.043 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:20:24 smithi069 ceph-mon[29472]: pgmap v112: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:20:25.043 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:20:24 smithi069 ceph-mon[29472]: from='client.25648 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T14:20:25.043 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:20:24 smithi069 ceph-mon[29472]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' 2024-04-14T14:20:25.043 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:20:24 smithi069 ceph-mon[29472]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-14T14:20:25.043 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:20:24 smithi069 ceph-mon[29472]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-14T14:20:25.043 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:20:24 smithi069 ceph-mon[29472]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T14:20:25.056 INFO:teuthology.orchestra.run.smithi069.stdout:{ 2024-04-14T14:20:25.056 INFO:teuthology.orchestra.run.smithi069.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888", 2024-04-14T14:20:25.056 INFO:teuthology.orchestra.run.smithi069.stdout: "in_progress": true, 2024-04-14T14:20:25.056 INFO:teuthology.orchestra.run.smithi069.stdout: "which": "Upgrading daemons of type(s) mon on host(s) smithi069", 2024-04-14T14:20:25.056 INFO:teuthology.orchestra.run.smithi069.stdout: "services_complete": [], 2024-04-14T14:20:25.056 INFO:teuthology.orchestra.run.smithi069.stdout: "progress": "0/2 daemons upgraded", 2024-04-14T14:20:25.056 INFO:teuthology.orchestra.run.smithi069.stdout: "message": "Currently upgrading mon daemons", 2024-04-14T14:20:25.057 INFO:teuthology.orchestra.run.smithi069.stdout: "is_paused": false 2024-04-14T14:20:25.057 INFO:teuthology.orchestra.run.smithi069.stdout:} 2024-04-14T14:20:25.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:20:24 smithi122 ceph-mon[67998]: pgmap v112: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:20:25.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:20:24 smithi122 ceph-mon[67998]: from='client.25648 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T14:20:25.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:20:24 smithi122 ceph-mon[67998]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' 2024-04-14T14:20:25.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:20:24 smithi122 ceph-mon[67998]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-14T14:20:25.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:20:24 smithi122 ceph-mon[67998]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-14T14:20:25.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:20:24 smithi122 ceph-mon[67998]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T14:20:25.948 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:20:25 smithi069 ceph-mon[24818]: from='client.15756 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T14:20:25.948 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:20:25 smithi069 ceph-mon[24818]: from='client.15762 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T14:20:25.948 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:20:25 smithi069 ceph-mon[24818]: Upgrade: Updating mon.a 2024-04-14T14:20:25.948 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:20:25 smithi069 ceph-mon[24818]: Deploying daemon mon.a on smithi069 2024-04-14T14:20:25.948 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:20:25 smithi069 ceph-mon[24818]: from='client.? 172.21.15.69:0/4252399014' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T14:20:25.948 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:20:25 smithi069 ceph-mon[29472]: from='client.15756 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T14:20:25.948 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:20:25 smithi069 ceph-mon[29472]: from='client.15762 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T14:20:25.948 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:20:25 smithi069 ceph-mon[29472]: Upgrade: Updating mon.a 2024-04-14T14:20:25.948 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:20:25 smithi069 ceph-mon[29472]: Deploying daemon mon.a on smithi069 2024-04-14T14:20:25.948 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:20:25 smithi069 ceph-mon[29472]: from='client.? 172.21.15.69:0/4252399014' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T14:20:26.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:20:25 smithi122 ceph-mon[67998]: from='client.15756 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T14:20:26.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:20:25 smithi122 ceph-mon[67998]: from='client.15762 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T14:20:26.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:20:25 smithi122 ceph-mon[67998]: Upgrade: Updating mon.a 2024-04-14T14:20:26.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:20:25 smithi122 ceph-mon[67998]: Deploying daemon mon.a on smithi069 2024-04-14T14:20:26.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:20:25 smithi122 ceph-mon[67998]: from='client.? 172.21.15.69:0/4252399014' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T14:20:26.504 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:20:26 smithi069 systemd[1]: Stopping Ceph mon.a for dc326fc2-fa65-11ee-bc8e-c7b262605968... 2024-04-14T14:20:26.505 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:20:26 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mon-a[24814]: 2024-04-14T14:20:26.458+0000 7f3b05ab9700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-04-14T14:20:26.505 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:20:26 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mon-a[24814]: 2024-04-14T14:20:26.458+0000 7f3b05ab9700 -1 mon.a@0(leader) e3 *** Got Signal Terminated *** 2024-04-14T14:20:26.505 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 14 14:20:26 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-y[63038]: [14/Apr/2024:14:20:26] ENGINE Bus STOPPING 2024-04-14T14:20:26.505 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 14 14:20:26 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-y[63038]: [14/Apr/2024:14:20:26] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-04-14T14:20:26.505 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 14 14:20:26 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-y[63038]: [14/Apr/2024:14:20:26] ENGINE Bus STOPPED 2024-04-14T14:20:26.505 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 14 14:20:26 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-y[63038]: [14/Apr/2024:14:20:26] ENGINE Bus STARTING 2024-04-14T14:20:26.505 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 14 14:20:26 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-y[63038]: ::ffff:172.21.15.122 - - [14/Apr/2024:14:20:26] "GET /metrics HTTP/1.1" 200 38006 "" "Prometheus/2.43.0" 2024-04-14T14:20:26.817 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:20:26 smithi069 podman[83507]: 2024-04-14 14:20:26.632839319 +0000 UTC m=+0.259458934 container died d3c553f919132d9c83653799e05d39a186cec421066a3f156339d49d1a0966b4 (image=quay.io/ceph/ceph:v17.2.0, name=ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mon-a, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, version=8, maintainer=Guillaume Abrioux , com.redhat.license_terms=https://centos.org/legal/licensing-policy/, io.openshift.expose-services=, RELEASE=HEAD, release=754, name=centos-stream, vcs-type=git, architecture=x86_64, vendor=Red Hat, Inc., GIT_CLEAN=True, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., CEPH_POINT_RELEASE=-17.2.0, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, GIT_REPO=https://github.com/ceph/ceph-container.git, io.buildah.version=1.19.8, ceph=True, com.redhat.component=centos-stream-container, distribution-scope=public, GIT_BRANCH=HEAD, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, io.k8s.display-name=CentOS Stream 8, io.openshift.tags=base centos centos-stream, build-date=2022-05-03T08:36:31.336870, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754) 2024-04-14T14:20:26.817 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 14 14:20:26 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-y[63038]: [14/Apr/2024:14:20:26] ENGINE Serving on http://:::9283 2024-04-14T14:20:26.817 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 14 14:20:26 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-y[63038]: [14/Apr/2024:14:20:26] ENGINE Bus STARTED 2024-04-14T14:20:27.178 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:20:26 smithi069 podman[83507]: 2024-04-14 14:20:26.975320151 +0000 UTC m=+0.601939766 container cleanup d3c553f919132d9c83653799e05d39a186cec421066a3f156339d49d1a0966b4 (image=quay.io/ceph/ceph:v17.2.0, name=ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mon-a, GIT_BRANCH=HEAD, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, maintainer=Guillaume Abrioux , name=centos-stream, GIT_REPO=https://github.com/ceph/ceph-container.git, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, RELEASE=HEAD, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, com.redhat.component=centos-stream-container, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, io.openshift.expose-services=, io.k8s.display-name=CentOS Stream 8, vcs-type=git, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., release=754, distribution-scope=public, GIT_CLEAN=True, build-date=2022-05-03T08:36:31.336870, version=8, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, io.openshift.tags=base centos centos-stream, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, CEPH_POINT_RELEASE=-17.2.0, architecture=x86_64, io.buildah.version=1.19.8, ceph=True, vendor=Red Hat, Inc.) 2024-04-14T14:20:27.178 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:20:26 smithi069 bash[83507]: ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mon-a 2024-04-14T14:20:27.567 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:20:27 smithi069 podman[83532]: 2024-04-14 14:20:27.176870741 +0000 UTC m=+0.542844723 container remove d3c553f919132d9c83653799e05d39a186cec421066a3f156339d49d1a0966b4 (image=quay.io/ceph/ceph:v17.2.0, name=ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mon-a, com.redhat.component=centos-stream-container, build-date=2022-05-03T08:36:31.336870, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., io.openshift.tags=base centos centos-stream, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, maintainer=Guillaume Abrioux , version=8, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, CEPH_POINT_RELEASE=-17.2.0, GIT_BRANCH=HEAD, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, vcs-type=git, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, GIT_CLEAN=True, io.buildah.version=1.19.8, release=754, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, vendor=Red Hat, Inc., description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, distribution-scope=public, name=centos-stream, architecture=x86_64, io.k8s.display-name=CentOS Stream 8, RELEASE=HEAD, ceph=True, io.openshift.expose-services=, GIT_REPO=https://github.com/ceph/ceph-container.git) 2024-04-14T14:20:27.567 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:20:27 smithi069 systemd[1]: ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968@mon.a.service: Deactivated successfully. 2024-04-14T14:20:27.567 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:20:27 smithi069 systemd[1]: Stopped Ceph mon.a for dc326fc2-fa65-11ee-bc8e-c7b262605968. 2024-04-14T14:20:27.567 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:20:27 smithi069 systemd[1]: ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968@mon.a.service: Consumed 25.483s CPU time. 2024-04-14T14:20:27.854 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:20:27 smithi069 systemd[1]: Starting Ceph mon.a for dc326fc2-fa65-11ee-bc8e-c7b262605968... 2024-04-14T14:20:27.854 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:20:27 smithi069 podman[83661]: 2024-04-14 14:20:27.709078431 +0000 UTC m=+0.017576915 image pull b96c509eef841460b3271f5fdd331860da87b004524f4ed59414aa3680183b53 quay.ceph.io/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888 2024-04-14T14:20:27.854 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:20:27 smithi069 podman[83661]: 2024-04-14 14:20:27.853272947 +0000 UTC m=+0.161771453 container create 8bcd6b3d3846c97cfe8e4e3c5925c23bb2c673c01b9102dced6517351af8bc80 (image=quay.ceph.io/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888, name=ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mon-a, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, maintainer=Guillaume Abrioux , ceph=True, RELEASE=main-a971fd5, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240403, CEPH_POINT_RELEASE=, GIT_CLEAN=True, io.buildah.version=1.33.5, GIT_BRANCH=HEAD) 2024-04-14T14:20:28.390 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:20:28 smithi069 podman[83661]: 2024-04-14 14:20:28.106734461 +0000 UTC m=+0.415232925 container init 8bcd6b3d3846c97cfe8e4e3c5925c23bb2c673c01b9102dced6517351af8bc80 (image=quay.ceph.io/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888, name=ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mon-a, org.label-schema.build-date=20240403, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=main-a971fd5, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.vendor=CentOS, ceph=True, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0) 2024-04-14T14:20:28.390 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:20:28 smithi069 podman[83661]: 2024-04-14 14:20:28.109919124 +0000 UTC m=+0.418417593 container start 8bcd6b3d3846c97cfe8e4e3c5925c23bb2c673c01b9102dced6517351af8bc80 (image=quay.ceph.io/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888, name=ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mon-a, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240403, org.label-schema.vendor=CentOS, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, ceph=True, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, RELEASE=main-a971fd5, org.label-schema.schema-version=1.0) 2024-04-14T14:20:28.390 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:20:28 smithi069 ceph-mon[83680]: set uid:gid to 167:167 (ceph:ceph) 2024-04-14T14:20:28.390 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:20:28 smithi069 ceph-mon[83680]: ceph version 19.0.0-2778-ga971fd5f (a971fd5f5719038c70b817bafacd6d5400b1d888) squid (dev), process ceph-mon, pid 2 2024-04-14T14:20:28.390 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:20:28 smithi069 ceph-mon[83680]: pidfile_write: ignore empty --pid-file 2024-04-14T14:20:28.390 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:20:28 smithi069 ceph-mon[83680]: load: jerasure load: lrc 2024-04-14T14:20:28.390 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:20:28 smithi069 ceph-mon[83680]: rocksdb: RocksDB version: 7.9.2 2024-04-14T14:20:28.390 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:20:28 smithi069 ceph-mon[83680]: rocksdb: Git sha 0 2024-04-14T14:20:28.390 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:20:28 smithi069 ceph-mon[83680]: rocksdb: Compile date 2024-04-06 20:28:24 2024-04-14T14:20:28.390 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:20:28 smithi069 ceph-mon[83680]: rocksdb: DB SUMMARY 2024-04-14T14:20:28.390 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:20:28 smithi069 ceph-mon[83680]: rocksdb: DB Session ID: 6XM50FKVSR33ODMD9PIX 2024-04-14T14:20:28.390 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:20:28 smithi069 ceph-mon[83680]: rocksdb: CURRENT file: CURRENT 2024-04-14T14:20:28.391 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:20:28 smithi069 ceph-mon[83680]: rocksdb: IDENTITY file: IDENTITY 2024-04-14T14:20:28.391 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:20:28 smithi069 ceph-mon[83680]: rocksdb: MANIFEST file: MANIFEST-000015 size: 2891 Bytes 2024-04-14T14:20:28.391 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:20:28 smithi069 ceph-mon[83680]: rocksdb: SST files in /var/lib/ceph/mon/ceph-a/store.db dir, Total Num: 1, files: 000060.sst 2024-04-14T14:20:28.391 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:20:28 smithi069 ceph-mon[83680]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-a/store.db: 000058.log size: 2029467 ; 2024-04-14T14:20:28.391 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:20:28 smithi069 ceph-mon[83680]: rocksdb: Options.error_if_exists: 0 2024-04-14T14:20:28.391 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:20:28 smithi069 ceph-mon[83680]: rocksdb: Options.create_if_missing: 0 2024-04-14T14:20:28.391 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:20:28 smithi069 ceph-mon[83680]: rocksdb: Options.paranoid_checks: 1 2024-04-14T14:20:28.391 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:20:28 smithi069 ceph-mon[83680]: rocksdb: Options.flush_verify_memtable_count: 1 2024-04-14T14:20:28.391 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:20:28 smithi069 ceph-mon[83680]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-04-14T14:20:28.391 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:20:28 smithi069 ceph-mon[83680]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-04-14T14:20:28.391 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:20:28 smithi069 ceph-mon[83680]: rocksdb: Options.env: 0x562dfdc82c60 2024-04-14T14:20:28.391 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:20:28 smithi069 ceph-mon[83680]: rocksdb: Options.fs: PosixFileSystem 2024-04-14T14:20:28.391 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:20:28 smithi069 ceph-mon[83680]: rocksdb: Options.info_log: 0x562dff1a7be0 2024-04-14T14:20:28.391 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:20:28 smithi069 ceph-mon[83680]: rocksdb: Options.max_file_opening_threads: 16 2024-04-14T14:20:28.391 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:20:28 smithi069 ceph-mon[83680]: rocksdb: Options.statistics: (nil) 2024-04-14T14:20:28.393 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:20:28 smithi069 ceph-mon[83680]: rocksdb: Options.use_fsync: 0 2024-04-14T14:20:28.393 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:20:28 smithi069 ceph-mon[83680]: rocksdb: Options.max_log_file_size: 0 2024-04-14T14:20:28.393 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:20:28 smithi069 ceph-mon[83680]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-04-14T14:20:28.393 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:20:28 smithi069 ceph-mon[83680]: rocksdb: Options.log_file_time_to_roll: 0 2024-04-14T14:20:28.393 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:20:28 smithi069 ceph-mon[83680]: rocksdb: Options.keep_log_file_num: 1000 2024-04-14T14:20:28.393 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:20:28 smithi069 ceph-mon[83680]: rocksdb: Options.recycle_log_file_num: 0 2024-04-14T14:20:28.393 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:20:28 smithi069 ceph-mon[83680]: rocksdb: Options.allow_fallocate: 1 2024-04-14T14:20:28.393 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:20:28 smithi069 ceph-mon[83680]: rocksdb: Options.allow_mmap_reads: 0 2024-04-14T14:20:28.393 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:20:28 smithi069 ceph-mon[83680]: rocksdb: Options.allow_mmap_writes: 0 2024-04-14T14:20:28.393 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:20:28 smithi069 ceph-mon[83680]: rocksdb: Options.use_direct_reads: 0 2024-04-14T14:20:28.394 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:20:28 smithi069 ceph-mon[83680]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-04-14T14:20:28.394 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:20:28 smithi069 ceph-mon[83680]: rocksdb: Options.create_missing_column_families: 0 2024-04-14T14:20:28.394 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:20:28 smithi069 ceph-mon[83680]: rocksdb: Options.db_log_dir: 2024-04-14T14:20:28.394 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:20:28 smithi069 ceph-mon[83680]: rocksdb: Options.wal_dir: 2024-04-14T14:20:28.394 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:20:28 smithi069 ceph-mon[83680]: rocksdb: Options.table_cache_numshardbits: 6 2024-04-14T14:20:28.394 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:20:28 smithi069 ceph-mon[83680]: rocksdb: Options.WAL_ttl_seconds: 0 2024-04-14T14:20:28.394 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:20:28 smithi069 ceph-mon[83680]: rocksdb: Options.WAL_size_limit_MB: 0 2024-04-14T14:20:28.394 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:20:28 smithi069 ceph-mon[83680]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-04-14T14:20:28.394 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:20:28 smithi069 ceph-mon[83680]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-04-14T14:20:28.394 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:20:28 smithi069 ceph-mon[83680]: rocksdb: Options.is_fd_close_on_exec: 1 2024-04-14T14:20:28.394 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:20:28 smithi069 ceph-mon[83680]: rocksdb: Options.advise_random_on_open: 1 2024-04-14T14:20:28.394 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:20:28 smithi069 ceph-mon[83680]: rocksdb: Options.db_write_buffer_size: 0 2024-04-14T14:20:28.394 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:20:28 smithi069 ceph-mon[83680]: rocksdb: Options.write_buffer_manager: 0x562dff178820 2024-04-14T14:20:28.394 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:20:28 smithi069 ceph-mon[83680]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-04-14T14:20:28.394 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:20:28 smithi069 ceph-mon[83680]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-04-14T14:20:28.395 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:20:28 smithi069 ceph-mon[83680]: rocksdb: Options.use_adaptive_mutex: 0 2024-04-14T14:20:28.395 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:20:28 smithi069 ceph-mon[83680]: rocksdb: Options.rate_limiter: (nil) 2024-04-14T14:20:28.395 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:20:28 smithi069 ceph-mon[83680]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-04-14T14:20:28.395 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:20:28 smithi069 ceph-mon[83680]: rocksdb: Options.wal_recovery_mode: 2 2024-04-14T14:20:28.395 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:20:28 smithi069 ceph-mon[83680]: rocksdb: Options.enable_thread_tracking: 0 2024-04-14T14:20:28.395 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:20:28 smithi069 ceph-mon[83680]: rocksdb: Options.enable_pipelined_write: 0 2024-04-14T14:20:28.395 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:20:28 smithi069 ceph-mon[83680]: rocksdb: Options.unordered_write: 0 2024-04-14T14:20:28.395 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:20:28 smithi069 ceph-mon[83680]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-04-14T14:20:28.395 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:20:28 smithi069 ceph-mon[83680]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-04-14T14:20:28.395 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:20:28 smithi069 ceph-mon[83680]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-04-14T14:20:28.395 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:20:28 smithi069 ceph-mon[83680]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-04-14T14:20:28.395 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:20:28 smithi069 ceph-mon[83680]: rocksdb: Options.row_cache: None 2024-04-14T14:20:28.395 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:20:28 smithi069 ceph-mon[83680]: rocksdb: Options.wal_filter: None 2024-04-14T14:20:28.395 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:20:28 smithi069 ceph-mon[83680]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-04-14T14:20:28.395 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:20:28 smithi069 ceph-mon[83680]: rocksdb: Options.allow_ingest_behind: 0 2024-04-14T14:20:28.396 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:20:28 smithi069 ceph-mon[83680]: rocksdb: Options.two_write_queues: 0 2024-04-14T14:20:28.396 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:20:28 smithi069 ceph-mon[83680]: rocksdb: Options.manual_wal_flush: 0 2024-04-14T14:20:28.396 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:20:28 smithi069 ceph-mon[83680]: rocksdb: Options.wal_compression: 0 2024-04-14T14:20:28.396 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:20:28 smithi069 ceph-mon[83680]: rocksdb: Options.atomic_flush: 0 2024-04-14T14:20:28.396 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:20:28 smithi069 ceph-mon[83680]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-04-14T14:20:28.396 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:20:28 smithi069 ceph-mon[83680]: rocksdb: Options.persist_stats_to_disk: 0 2024-04-14T14:20:28.396 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:20:28 smithi069 ceph-mon[83680]: rocksdb: Options.write_dbid_to_manifest: 0 2024-04-14T14:20:28.396 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:20:28 smithi069 ceph-mon[83680]: rocksdb: Options.log_readahead_size: 0 2024-04-14T14:20:28.396 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:20:28 smithi069 ceph-mon[83680]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-04-14T14:20:28.396 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:20:28 smithi069 ceph-mon[83680]: rocksdb: Options.best_efforts_recovery: 0 2024-04-14T14:20:28.396 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:20:28 smithi069 ceph-mon[83680]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-04-14T14:20:28.396 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:20:28 smithi069 ceph-mon[83680]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-04-14T14:20:28.396 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:20:28 smithi069 ceph-mon[83680]: rocksdb: Options.allow_data_in_errors: 0 2024-04-14T14:20:28.396 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:20:28 smithi069 ceph-mon[83680]: rocksdb: Options.db_host_id: __hostname__ 2024-04-14T14:20:28.397 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:20:28 smithi069 ceph-mon[83680]: rocksdb: Options.enforce_single_del_contracts: true 2024-04-14T14:20:28.397 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:20:28 smithi069 ceph-mon[83680]: rocksdb: Options.max_background_jobs: 2 2024-04-14T14:20:28.397 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:20:28 smithi069 ceph-mon[83680]: rocksdb: Options.max_background_compactions: -1 2024-04-14T14:20:28.397 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:20:28 smithi069 ceph-mon[83680]: rocksdb: Options.max_subcompactions: 1 2024-04-14T14:20:28.397 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:20:28 smithi069 ceph-mon[83680]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-04-14T14:20:28.397 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:20:28 smithi069 ceph-mon[83680]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-04-14T14:20:28.397 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:20:28 smithi069 ceph-mon[83680]: rocksdb: Options.delayed_write_rate : 16777216 2024-04-14T14:20:28.397 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:20:28 smithi069 ceph-mon[83680]: rocksdb: Options.max_total_wal_size: 0 2024-04-14T14:20:28.397 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:20:28 smithi069 ceph-mon[83680]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-04-14T14:20:28.397 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:20:28 smithi069 ceph-mon[83680]: rocksdb: Options.stats_dump_period_sec: 600 2024-04-14T14:20:28.397 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:20:28 smithi069 ceph-mon[83680]: rocksdb: Options.stats_persist_period_sec: 600 2024-04-14T14:20:28.397 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:20:28 smithi069 ceph-mon[83680]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-04-14T14:20:28.397 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:20:28 smithi069 ceph-mon[83680]: rocksdb: Options.max_open_files: -1 2024-04-14T14:20:28.397 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:20:28 smithi069 ceph-mon[83680]: rocksdb: Options.bytes_per_sync: 0 2024-04-14T14:20:28.397 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:20:28 smithi069 ceph-mon[83680]: rocksdb: Options.wal_bytes_per_sync: 0 2024-04-14T14:20:28.398 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:20:28 smithi069 ceph-mon[83680]: rocksdb: Options.strict_bytes_per_sync: 0 2024-04-14T14:20:28.398 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:20:28 smithi069 ceph-mon[83680]: rocksdb: Options.compaction_readahead_size: 0 2024-04-14T14:20:28.398 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:20:28 smithi069 ceph-mon[83680]: rocksdb: Options.max_background_flushes: -1 2024-04-14T14:20:28.398 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:20:28 smithi069 ceph-mon[83680]: rocksdb: Compression algorithms supported: 2024-04-14T14:20:28.398 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:20:28 smithi069 ceph-mon[83680]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-04-14T14:20:28.398 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:20:28 smithi069 ceph-mon[83680]: rocksdb: kZSTD supported: 0 2024-04-14T14:20:28.398 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:20:28 smithi069 ceph-mon[83680]: rocksdb: kXpressCompression supported: 0 2024-04-14T14:20:28.398 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:20:28 smithi069 ceph-mon[83680]: rocksdb: kLZ4HCCompression supported: 1 2024-04-14T14:20:28.398 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:20:28 smithi069 ceph-mon[83680]: rocksdb: kZlibCompression supported: 1 2024-04-14T14:20:28.398 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:20:28 smithi069 ceph-mon[83680]: rocksdb: kSnappyCompression supported: 1 2024-04-14T14:20:28.398 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:20:28 smithi069 ceph-mon[83680]: rocksdb: kLZ4Compression supported: 1 2024-04-14T14:20:28.398 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:20:28 smithi069 ceph-mon[83680]: rocksdb: kBZip2Compression supported: 0 2024-04-14T14:20:28.398 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:20:28 smithi069 ceph-mon[83680]: rocksdb: Fast CRC32 supported: Supported on x86 2024-04-14T14:20:28.398 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:20:28 smithi069 ceph-mon[83680]: rocksdb: DMutex implementation: pthread_mutex_t 2024-04-14T14:20:28.399 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:20:28 smithi069 ceph-mon[83680]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000015 2024-04-14T14:20:28.399 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:20:28 smithi069 ceph-mon[83680]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-04-14T14:20:28.399 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:20:28 smithi069 ceph-mon[83680]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-04-14T14:20:28.399 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:20:28 smithi069 ceph-mon[83680]: rocksdb: Options.merge_operator: 2024-04-14T14:20:28.399 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:20:28 smithi069 ceph-mon[83680]: rocksdb: Options.compaction_filter: None 2024-04-14T14:20:28.399 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:20:28 smithi069 ceph-mon[83680]: rocksdb: Options.compaction_filter_factory: None 2024-04-14T14:20:28.399 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:20:28 smithi069 ceph-mon[83680]: rocksdb: Options.sst_partitioner_factory: None 2024-04-14T14:20:28.399 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:20:28 smithi069 ceph-mon[83680]: rocksdb: Options.memtable_factory: SkipListFactory 2024-04-14T14:20:28.399 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:20:28 smithi069 ceph-mon[83680]: rocksdb: Options.table_factory: BlockBasedTable 2024-04-14T14:20:28.399 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:20:28 smithi069 ceph-mon[83680]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x562dff1a7dc0) 2024-04-14T14:20:28.399 INFO:journalctl@ceph.mon.a.smithi069.stdout: cache_index_and_filter_blocks: 1 2024-04-14T14:20:28.399 INFO:journalctl@ceph.mon.a.smithi069.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-04-14T14:20:28.399 INFO:journalctl@ceph.mon.a.smithi069.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-04-14T14:20:28.399 INFO:journalctl@ceph.mon.a.smithi069.stdout: pin_top_level_index_and_filter: 1 2024-04-14T14:20:28.399 INFO:journalctl@ceph.mon.a.smithi069.stdout: index_type: 0 2024-04-14T14:20:28.400 INFO:journalctl@ceph.mon.a.smithi069.stdout: data_block_index_type: 0 2024-04-14T14:20:28.400 INFO:journalctl@ceph.mon.a.smithi069.stdout: index_shortening: 1 2024-04-14T14:20:28.400 INFO:journalctl@ceph.mon.a.smithi069.stdout: data_block_hash_table_util_ratio: 0.750000 2024-04-14T14:20:28.400 INFO:journalctl@ceph.mon.a.smithi069.stdout: checksum: 4 2024-04-14T14:20:28.400 INFO:journalctl@ceph.mon.a.smithi069.stdout: no_block_cache: 0 2024-04-14T14:20:28.400 INFO:journalctl@ceph.mon.a.smithi069.stdout: block_cache: 0x562dff1ab090 2024-04-14T14:20:28.400 INFO:journalctl@ceph.mon.a.smithi069.stdout: block_cache_name: BinnedLRUCache 2024-04-14T14:20:28.400 INFO:journalctl@ceph.mon.a.smithi069.stdout: block_cache_options: 2024-04-14T14:20:28.400 INFO:journalctl@ceph.mon.a.smithi069.stdout: capacity : 536870912 2024-04-14T14:20:28.400 INFO:journalctl@ceph.mon.a.smithi069.stdout: num_shard_bits : 4 2024-04-14T14:20:28.400 INFO:journalctl@ceph.mon.a.smithi069.stdout: strict_capacity_limit : 0 2024-04-14T14:20:28.400 INFO:journalctl@ceph.mon.a.smithi069.stdout: high_pri_pool_ratio: 0.000 2024-04-14T14:20:28.400 INFO:journalctl@ceph.mon.a.smithi069.stdout: block_cache_compressed: (nil) 2024-04-14T14:20:28.400 INFO:journalctl@ceph.mon.a.smithi069.stdout: persistent_cache: (nil) 2024-04-14T14:20:28.400 INFO:journalctl@ceph.mon.a.smithi069.stdout: block_size: 4096 2024-04-14T14:20:28.401 INFO:journalctl@ceph.mon.a.smithi069.stdout: block_size_deviation: 10 2024-04-14T14:20:28.401 INFO:journalctl@ceph.mon.a.smithi069.stdout: block_restart_interval: 16 2024-04-14T14:20:28.401 INFO:journalctl@ceph.mon.a.smithi069.stdout: index_block_restart_interval: 1 2024-04-14T14:20:28.401 INFO:journalctl@ceph.mon.a.smithi069.stdout: metadata_block_size: 4096 2024-04-14T14:20:28.401 INFO:journalctl@ceph.mon.a.smithi069.stdout: partition_filters: 0 2024-04-14T14:20:28.401 INFO:journalctl@ceph.mon.a.smithi069.stdout: use_delta_encoding: 1 2024-04-14T14:20:28.401 INFO:journalctl@ceph.mon.a.smithi069.stdout: filter_policy: bloomfilter 2024-04-14T14:20:28.401 INFO:journalctl@ceph.mon.a.smithi069.stdout: whole_key_filtering: 1 2024-04-14T14:20:28.401 INFO:journalctl@ceph.mon.a.smithi069.stdout: verify_compression: 0 2024-04-14T14:20:28.401 INFO:journalctl@ceph.mon.a.smithi069.stdout: read_amp_bytes_per_bit: 0 2024-04-14T14:20:28.401 INFO:journalctl@ceph.mon.a.smithi069.stdout: format_version: 5 2024-04-14T14:20:28.401 INFO:journalctl@ceph.mon.a.smithi069.stdout: enable_index_compression: 1 2024-04-14T14:20:28.401 INFO:journalctl@ceph.mon.a.smithi069.stdout: block_align: 0 2024-04-14T14:20:28.401 INFO:journalctl@ceph.mon.a.smithi069.stdout: max_auto_readahead_size: 262144 2024-04-14T14:20:28.401 INFO:journalctl@ceph.mon.a.smithi069.stdout: prepopulate_block_cache: 0 2024-04-14T14:20:28.402 INFO:journalctl@ceph.mon.a.smithi069.stdout: initial_auto_readahead_size: 8192 2024-04-14T14:20:28.402 INFO:journalctl@ceph.mon.a.smithi069.stdout: num_file_reads_for_auto_readahead: 2 2024-04-14T14:20:28.402 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:20:28 smithi069 ceph-mon[83680]: rocksdb: Options.write_buffer_size: 33554432 2024-04-14T14:20:28.402 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:20:28 smithi069 ceph-mon[83680]: rocksdb: Options.max_write_buffer_number: 2 2024-04-14T14:20:28.402 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:20:28 smithi069 ceph-mon[83680]: rocksdb: Options.compression: NoCompression 2024-04-14T14:20:28.402 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:20:28 smithi069 ceph-mon[83680]: rocksdb: Options.bottommost_compression: Disabled 2024-04-14T14:20:28.402 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:20:28 smithi069 ceph-mon[83680]: rocksdb: Options.prefix_extractor: nullptr 2024-04-14T14:20:28.402 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:20:28 smithi069 ceph-mon[83680]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-04-14T14:20:28.402 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:20:28 smithi069 ceph-mon[83680]: rocksdb: Options.num_levels: 7 2024-04-14T14:20:28.402 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:20:28 smithi069 ceph-mon[83680]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-04-14T14:20:28.402 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:20:28 smithi069 ceph-mon[83680]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-04-14T14:20:28.402 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:20:28 smithi069 ceph-mon[83680]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-04-14T14:20:28.402 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:20:28 smithi069 ceph-mon[83680]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-04-14T14:20:28.402 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:20:28 smithi069 ceph-mon[83680]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-04-14T14:20:28.403 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:20:28 smithi069 ceph-mon[83680]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-04-14T14:20:28.403 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:20:28 smithi069 ceph-mon[83680]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-04-14T14:20:28.403 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:20:28 smithi069 ceph-mon[83680]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-04-14T14:20:28.403 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:20:28 smithi069 ceph-mon[83680]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-04-14T14:20:28.403 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:20:28 smithi069 ceph-mon[83680]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-04-14T14:20:28.403 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:20:28 smithi069 ceph-mon[83680]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-04-14T14:20:28.403 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:20:28 smithi069 ceph-mon[83680]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-04-14T14:20:28.403 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:20:28 smithi069 ceph-mon[83680]: rocksdb: Options.compression_opts.window_bits: -14 2024-04-14T14:20:28.403 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:20:28 smithi069 ceph-mon[83680]: rocksdb: Options.compression_opts.level: 32767 2024-04-14T14:20:28.403 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:20:28 smithi069 ceph-mon[83680]: rocksdb: Options.compression_opts.strategy: 0 2024-04-14T14:20:28.403 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:20:28 smithi069 ceph-mon[83680]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-04-14T14:20:28.403 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:20:28 smithi069 ceph-mon[83680]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-04-14T14:20:28.403 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:20:28 smithi069 ceph-mon[83680]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-04-14T14:20:28.403 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:20:28 smithi069 ceph-mon[83680]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-04-14T14:20:28.403 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:20:28 smithi069 ceph-mon[83680]: rocksdb: Options.compression_opts.enabled: false 2024-04-14T14:20:28.404 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:20:28 smithi069 ceph-mon[83680]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-04-14T14:20:28.404 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:20:28 smithi069 ceph-mon[83680]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-04-14T14:20:28.404 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:20:28 smithi069 ceph-mon[83680]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-04-14T14:20:28.404 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:20:28 smithi069 ceph-mon[83680]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-04-14T14:20:28.404 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:20:28 smithi069 ceph-mon[83680]: rocksdb: Options.target_file_size_base: 67108864 2024-04-14T14:20:28.404 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:20:28 smithi069 ceph-mon[83680]: rocksdb: Options.target_file_size_multiplier: 1 2024-04-14T14:20:28.404 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:20:28 smithi069 ceph-mon[83680]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-04-14T14:20:28.404 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:20:28 smithi069 ceph-mon[83680]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-04-14T14:20:28.404 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:20:28 smithi069 ceph-mon[83680]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-04-14T14:20:28.404 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:20:28 smithi069 ceph-mon[83680]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-04-14T14:20:28.404 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:20:28 smithi069 ceph-mon[83680]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-04-14T14:20:28.404 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:20:28 smithi069 ceph-mon[83680]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-04-14T14:20:28.404 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:20:28 smithi069 ceph-mon[83680]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-04-14T14:20:28.404 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:20:28 smithi069 ceph-mon[83680]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-04-14T14:20:28.404 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:20:28 smithi069 ceph-mon[83680]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-04-14T14:20:28.405 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:20:28 smithi069 ceph-mon[83680]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-04-14T14:20:28.405 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:20:28 smithi069 ceph-mon[83680]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-04-14T14:20:28.405 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:20:28 smithi069 ceph-mon[83680]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-04-14T14:20:28.405 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:20:28 smithi069 ceph-mon[83680]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-04-14T14:20:28.405 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:20:28 smithi069 ceph-mon[83680]: rocksdb: Options.arena_block_size: 1048576 2024-04-14T14:20:28.405 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:20:28 smithi069 ceph-mon[83680]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-04-14T14:20:28.405 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:20:28 smithi069 ceph-mon[83680]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-04-14T14:20:28.405 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:20:28 smithi069 ceph-mon[83680]: rocksdb: Options.disable_auto_compactions: 0 2024-04-14T14:20:28.405 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:20:28 smithi069 ceph-mon[83680]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-04-14T14:20:28.405 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:20:28 smithi069 ceph-mon[83680]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-04-14T14:20:28.405 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:20:28 smithi069 ceph-mon[83680]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-04-14T14:20:28.405 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:20:28 smithi069 ceph-mon[83680]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-04-14T14:20:28.405 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:20:28 smithi069 ceph-mon[83680]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-04-14T14:20:28.405 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:20:28 smithi069 ceph-mon[83680]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-04-14T14:20:28.405 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:20:28 smithi069 ceph-mon[83680]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-04-14T14:20:28.406 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:20:28 smithi069 ceph-mon[83680]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-04-14T14:20:28.406 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:20:28 smithi069 ceph-mon[83680]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-04-14T14:20:28.406 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:20:28 smithi069 ceph-mon[83680]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-04-14T14:20:28.406 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:20:28 smithi069 ceph-mon[83680]: rocksdb: Options.table_properties_collectors: CompactOnDeletionCollector (Sliding window size = 32768 Deletion trigger = 16384 Deletion ratio = 0); 2024-04-14T14:20:28.406 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:20:28 smithi069 ceph-mon[83680]: rocksdb: Options.inplace_update_support: 0 2024-04-14T14:20:28.406 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:20:28 smithi069 ceph-mon[83680]: rocksdb: Options.inplace_update_num_locks: 10000 2024-04-14T14:20:28.406 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:20:28 smithi069 ceph-mon[83680]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-04-14T14:20:28.406 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:20:28 smithi069 ceph-mon[83680]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-04-14T14:20:28.406 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:20:28 smithi069 ceph-mon[83680]: rocksdb: Options.memtable_huge_page_size: 0 2024-04-14T14:20:28.406 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:20:28 smithi069 ceph-mon[83680]: rocksdb: Options.bloom_locality: 0 2024-04-14T14:20:28.406 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:20:28 smithi069 ceph-mon[83680]: rocksdb: Options.max_successive_merges: 0 2024-04-14T14:20:28.406 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:20:28 smithi069 ceph-mon[83680]: rocksdb: Options.optimize_filters_for_hits: 0 2024-04-14T14:20:28.406 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:20:28 smithi069 ceph-mon[83680]: rocksdb: Options.paranoid_file_checks: 0 2024-04-14T14:20:28.406 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:20:28 smithi069 ceph-mon[83680]: rocksdb: Options.force_consistency_checks: 1 2024-04-14T14:20:28.407 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:20:28 smithi069 ceph-mon[83680]: rocksdb: Options.report_bg_io_stats: 0 2024-04-14T14:20:28.407 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:20:28 smithi069 ceph-mon[83680]: rocksdb: Options.ttl: 2592000 2024-04-14T14:20:28.407 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:20:28 smithi069 ceph-mon[83680]: rocksdb: Options.periodic_compaction_seconds: 0 2024-04-14T14:20:28.407 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:20:28 smithi069 ceph-mon[83680]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-04-14T14:20:28.407 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:20:28 smithi069 ceph-mon[83680]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-04-14T14:20:28.407 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:20:28 smithi069 ceph-mon[83680]: rocksdb: Options.enable_blob_files: false 2024-04-14T14:20:28.407 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:20:28 smithi069 ceph-mon[83680]: rocksdb: Options.min_blob_size: 0 2024-04-14T14:20:28.407 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:20:28 smithi069 ceph-mon[83680]: rocksdb: Options.blob_file_size: 268435456 2024-04-14T14:20:28.407 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:20:28 smithi069 ceph-mon[83680]: rocksdb: Options.blob_compression_type: NoCompression 2024-04-14T14:20:28.407 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:20:28 smithi069 ceph-mon[83680]: rocksdb: Options.enable_blob_garbage_collection: false 2024-04-14T14:20:28.407 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:20:28 smithi069 ceph-mon[83680]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-04-14T14:20:28.407 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:20:28 smithi069 ceph-mon[83680]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-04-14T14:20:28.407 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:20:28 smithi069 ceph-mon[83680]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-04-14T14:20:28.407 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:20:28 smithi069 ceph-mon[83680]: rocksdb: Options.blob_file_starting_level: 0 2024-04-14T14:20:28.407 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:20:28 smithi069 ceph-mon[83680]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-04-14T14:20:28.407 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:20:28 smithi069 ceph-mon[83680]: rocksdb: [table/block_based/block_based_table_reader.cc:721] At least one SST file opened without unique ID to verify: 60.sst 2024-04-14T14:20:28.408 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:20:28 smithi069 ceph-mon[83680]: rocksdb: [db/version_set.cc:4390] More existing levels in DB than needed. max_bytes_for_level_multiplier may not be guaranteed. 2024-04-14T14:20:28.408 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:20:28 smithi069 ceph-mon[83680]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000015 succeeded,manifest_file_number is 15, next_file_number is 62, last_sequence is 26787, log_number is 58,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2024-04-14T14:20:28.408 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:20:28 smithi069 ceph-mon[83680]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 58 2024-04-14T14:20:28.408 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:20:28 smithi069 ceph-mon[83680]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: 68dacaa5-8ecc-4064-9fea-cbab7d38d01a 2024-04-14T14:20:28.408 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:20:28 smithi069 ceph-mon[83680]: rocksdb: EVENT_LOG_v1 {"time_micros": 1713104428139650, "job": 1, "event": "recovery_started", "wal_files": [58]} 2024-04-14T14:20:28.408 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:20:28 smithi069 ceph-mon[83680]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #58 mode 2 2024-04-14T14:20:28.408 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:20:28 smithi069 ceph-mon[83680]: rocksdb: EVENT_LOG_v1 {"time_micros": 1713104428147216, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 63, "file_size": 1896090, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 26789, "largest_seqno": 28798, "table_properties": {"data_size": 1888434, "index_size": 4438, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 2181, "raw_key_size": 21153, "raw_average_key_size": 24, "raw_value_size": 1871003, "raw_average_value_size": 2193, "num_data_blocks": 200, "num_entries": 853, "num_filter_entries": 853, "num_deletions": 8, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[CompactOnDeletionCollector]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1713104428, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "68dacaa5-8ecc-4064-9fea-cbab7d38d01a", "db_session_id": "6XM50FKVSR33ODMD9PIX", "orig_file_number": 63, "seqno_to_time_mapping": "N/A"}} 2024-04-14T14:20:28.408 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:20:28 smithi069 ceph-mon[83680]: rocksdb: EVENT_LOG_v1 {"time_micros": 1713104428147317, "job": 1, "event": "recovery_finished"} 2024-04-14T14:20:28.408 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:20:28 smithi069 ceph-mon[83680]: rocksdb: [db/version_set.cc:5047] Creating manifest 65 2024-04-14T14:20:28.408 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:20:28 smithi069 ceph-mon[83680]: rocksdb: [db/version_set.cc:4390] More existing levels in DB than needed. max_bytes_for_level_multiplier may not be guaranteed. 2024-04-14T14:20:28.408 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:20:28 smithi069 ceph-mon[83680]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-a/store.db/000058.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-04-14T14:20:28.408 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:20:28 smithi069 ceph-mon[83680]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x562dff29a000 2024-04-14T14:20:28.408 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:20:28 smithi069 ceph-mon[83680]: rocksdb: DB pointer 0x562dff284000 2024-04-14T14:20:28.408 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:20:28 smithi069 ceph-mon[83680]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-04-14T14:20:28.409 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:20:28 smithi069 ceph-mon[83680]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-04-14T14:20:28.409 INFO:journalctl@ceph.mon.a.smithi069.stdout: ** DB Stats ** 2024-04-14T14:20:28.409 INFO:journalctl@ceph.mon.a.smithi069.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-04-14T14:20:28.409 INFO:journalctl@ceph.mon.a.smithi069.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-04-14T14:20:28.409 INFO:journalctl@ceph.mon.a.smithi069.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-04-14T14:20:28.409 INFO:journalctl@ceph.mon.a.smithi069.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-04-14T14:20:28.409 INFO:journalctl@ceph.mon.a.smithi069.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-04-14T14:20:28.409 INFO:journalctl@ceph.mon.a.smithi069.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-04-14T14:20:28.409 INFO:journalctl@ceph.mon.a.smithi069.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-04-14T14:20:28.409 INFO:journalctl@ceph.mon.a.smithi069.stdout: 2024-04-14T14:20:28.409 INFO:journalctl@ceph.mon.a.smithi069.stdout: ** Compaction Stats [default] ** 2024-04-14T14:20:28.409 INFO:journalctl@ceph.mon.a.smithi069.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-04-14T14:20:28.409 INFO:journalctl@ceph.mon.a.smithi069.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-04-14T14:20:28.409 INFO:journalctl@ceph.mon.a.smithi069.stdout: L0 1/0 1.81 MB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 530.9 0.00 0.00 1 0.003 0 0 0.0 0.0 2024-04-14T14:20:28.409 INFO:journalctl@ceph.mon.a.smithi069.stdout: L6 1/0 9.03 MB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 0.0 0.0 2024-04-14T14:20:28.410 INFO:journalctl@ceph.mon.a.smithi069.stdout: Sum 2/0 10.84 MB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 530.9 0.00 0.00 1 0.003 0 0 0.0 0.0 2024-04-14T14:20:28.410 INFO:journalctl@ceph.mon.a.smithi069.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 530.9 0.00 0.00 1 0.003 0 0 0.0 0.0 2024-04-14T14:20:28.410 INFO:journalctl@ceph.mon.a.smithi069.stdout: 2024-04-14T14:20:28.410 INFO:journalctl@ceph.mon.a.smithi069.stdout: ** Compaction Stats [default] ** 2024-04-14T14:20:28.410 INFO:journalctl@ceph.mon.a.smithi069.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-04-14T14:20:28.410 INFO:journalctl@ceph.mon.a.smithi069.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-04-14T14:20:28.410 INFO:journalctl@ceph.mon.a.smithi069.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 530.9 0.00 0.00 1 0.003 0 0 0.0 0.0 2024-04-14T14:20:28.410 INFO:journalctl@ceph.mon.a.smithi069.stdout: 2024-04-14T14:20:28.410 INFO:journalctl@ceph.mon.a.smithi069.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-04-14T14:20:28.410 INFO:journalctl@ceph.mon.a.smithi069.stdout: 2024-04-14T14:20:28.410 INFO:journalctl@ceph.mon.a.smithi069.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-04-14T14:20:28.410 INFO:journalctl@ceph.mon.a.smithi069.stdout: Flush(GB): cumulative 0.002, interval 0.002 2024-04-14T14:20:28.410 INFO:journalctl@ceph.mon.a.smithi069.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-04-14T14:20:28.410 INFO:journalctl@ceph.mon.a.smithi069.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-04-14T14:20:28.410 INFO:journalctl@ceph.mon.a.smithi069.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-04-14T14:20:28.411 INFO:journalctl@ceph.mon.a.smithi069.stdout: AddFile(Keys): cumulative 0, interval 0 2024-04-14T14:20:28.411 INFO:journalctl@ceph.mon.a.smithi069.stdout: Cumulative compaction: 0.00 GB write, 184.48 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-04-14T14:20:28.411 INFO:journalctl@ceph.mon.a.smithi069.stdout: Interval compaction: 0.00 GB write, 184.48 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-04-14T14:20:28.411 INFO:journalctl@ceph.mon.a.smithi069.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-04-14T14:20:28.411 INFO:journalctl@ceph.mon.a.smithi069.stdout: Block cache BinnedLRUCache@0x562dff1ab090#2 capacity: 512.00 MB usage: 6.86 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 9e-06 secs_since: 0 2024-04-14T14:20:28.411 INFO:journalctl@ceph.mon.a.smithi069.stdout: Block cache entry stats(count,size,portion): FilterBlock(1,2.28 KB,0.000435114%) IndexBlock(1,4.58 KB,0.000873208%) Misc(1,0.00 KB,0%) 2024-04-14T14:20:28.411 INFO:journalctl@ceph.mon.a.smithi069.stdout: 2024-04-14T14:20:28.411 INFO:journalctl@ceph.mon.a.smithi069.stdout: ** File Read Latency Histogram By Level [default] ** 2024-04-14T14:20:28.411 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:20:28 smithi069 ceph-mon[83680]: starting mon.a rank 0 at public addrs [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] at bind addrs [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon_data /var/lib/ceph/mon/ceph-a fsid dc326fc2-fa65-11ee-bc8e-c7b262605968 2024-04-14T14:20:28.411 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:20:28 smithi069 ceph-mon[83680]: mon.a@-1(???) e3 preinit fsid dc326fc2-fa65-11ee-bc8e-c7b262605968 2024-04-14T14:20:28.411 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:20:28 smithi069 ceph-mon[83680]: mon.a@-1(???).mds e1 new map 2024-04-14T14:20:28.411 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:20:28 smithi069 ceph-mon[83680]: mon.a@-1(???).mds e1 print_map 2024-04-14T14:20:28.411 INFO:journalctl@ceph.mon.a.smithi069.stdout: e1 2024-04-14T14:20:28.411 INFO:journalctl@ceph.mon.a.smithi069.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-04-14T14:20:28.411 INFO:journalctl@ceph.mon.a.smithi069.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-14T14:20:28.412 INFO:journalctl@ceph.mon.a.smithi069.stdout: legacy client fscid: -1 2024-04-14T14:20:28.412 INFO:journalctl@ceph.mon.a.smithi069.stdout: 2024-04-14T14:20:28.412 INFO:journalctl@ceph.mon.a.smithi069.stdout: No filesystems configured 2024-04-14T14:20:28.412 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:20:28 smithi069 ceph-mon[83680]: mon.a@-1(???).osd e110 crush map has features 3314933000854323200, adjusting msgr requires 2024-04-14T14:20:28.412 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:20:28 smithi069 ceph-mon[83680]: mon.a@-1(???).osd e110 crush map has features 432629239337189376, adjusting msgr requires 2024-04-14T14:20:28.412 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:20:28 smithi069 ceph-mon[83680]: mon.a@-1(???).osd e110 crush map has features 432629239337189376, adjusting msgr requires 2024-04-14T14:20:28.412 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:20:28 smithi069 ceph-mon[83680]: mon.a@-1(???).osd e110 crush map has features 432629239337189376, adjusting msgr requires 2024-04-14T14:20:28.412 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:20:28 smithi069 ceph-mon[83680]: mon.a@-1(???).paxosservice(auth 1..26) refresh upgraded, format 0 -> 3 2024-04-14T14:20:28.412 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:20:28 smithi069 bash[83661]: 8bcd6b3d3846c97cfe8e4e3c5925c23bb2c673c01b9102dced6517351af8bc80 2024-04-14T14:20:28.412 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:20:28 smithi069 systemd[1]: Started Ceph mon.a for dc326fc2-fa65-11ee-bc8e-c7b262605968. 2024-04-14T14:20:29.948 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Apr 14 14:20:29 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-x[61397]: ignoring --setuser ceph since I am not root 2024-04-14T14:20:29.948 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Apr 14 14:20:29 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-x[61397]: ignoring --setgroup ceph since I am not root 2024-04-14T14:20:29.948 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Apr 14 14:20:29 smithi122 ceph-mgr[61401]: -- 172.21.15.122:0/2767636149 <== mon.2 v2:172.21.15.122:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x5628fffda5a0 con 0x5628fffae800 2024-04-14T14:20:29.948 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:20:29 smithi122 ceph-mon[67998]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-14T14:20:29.949 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:20:29 smithi122 ceph-mon[67998]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-14T14:20:29.949 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:20:29 smithi122 ceph-mon[67998]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-14T14:20:29.949 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:20:29 smithi122 ceph-mon[67998]: pgmap v114: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:20:29.949 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:20:29 smithi122 ceph-mon[67998]: mon.a calling monitor election 2024-04-14T14:20:29.949 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:20:29 smithi122 ceph-mon[67998]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-04-14T14:20:29.949 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:20:29 smithi122 ceph-mon[67998]: monmap epoch 3 2024-04-14T14:20:29.949 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:20:29 smithi122 ceph-mon[67998]: fsid dc326fc2-fa65-11ee-bc8e-c7b262605968 2024-04-14T14:20:29.949 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:20:29 smithi122 ceph-mon[67998]: last_changed 2024-04-14T13:51:50.965882+0000 2024-04-14T14:20:29.949 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:20:29 smithi122 ceph-mon[67998]: created 2024-04-14T13:50:18.166086+0000 2024-04-14T14:20:29.949 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:20:29 smithi122 ceph-mon[67998]: min_mon_release 17 (quincy) 2024-04-14T14:20:29.949 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:20:29 smithi122 ceph-mon[67998]: election_strategy: 1 2024-04-14T14:20:29.949 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:20:29 smithi122 ceph-mon[67998]: 0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2024-04-14T14:20:29.949 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:20:29 smithi122 ceph-mon[67998]: 1: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2024-04-14T14:20:29.950 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:20:29 smithi122 ceph-mon[67998]: 2: [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] mon.b 2024-04-14T14:20:29.950 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:20:29 smithi122 ceph-mon[67998]: fsmap 2024-04-14T14:20:29.950 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:20:29 smithi122 ceph-mon[67998]: osdmap e110: 8 total, 8 up, 8 in 2024-04-14T14:20:29.950 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:20:29 smithi122 ceph-mon[67998]: mgrmap e41: y(active, since 3m), standbys: x 2024-04-14T14:20:29.950 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:20:29 smithi122 ceph-mon[67998]: overall HEALTH_OK 2024-04-14T14:20:29.950 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:20:29 smithi122 ceph-mon[67998]: from='mgr.15339 ' entity='' 2024-04-14T14:20:29.950 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:20:29 smithi122 ceph-mon[67998]: mgrmap e42: y(active, since 3m), standbys: x 2024-04-14T14:20:30.067 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 14 14:20:29 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-y[63038]: ignoring --setuser ceph since I am not root 2024-04-14T14:20:30.067 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 14 14:20:29 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-y[63038]: ignoring --setgroup ceph since I am not root 2024-04-14T14:20:30.067 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 14 14:20:29 smithi069 ceph-mgr[63042]: -- 172.21.15.69:0/3612085749 <== mon.0 v2:172.21.15.69:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55be1bbac5a0 con 0x55be1bb81000 2024-04-14T14:20:30.067 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:20:29 smithi069 ceph-mon[29472]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-14T14:20:30.068 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:20:29 smithi069 ceph-mon[29472]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-14T14:20:30.068 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:20:29 smithi069 ceph-mon[29472]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-14T14:20:30.068 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:20:29 smithi069 ceph-mon[29472]: pgmap v114: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:20:30.068 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:20:29 smithi069 ceph-mon[29472]: mon.a calling monitor election 2024-04-14T14:20:30.068 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:20:29 smithi069 ceph-mon[29472]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-04-14T14:20:30.068 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:20:29 smithi069 ceph-mon[29472]: monmap epoch 3 2024-04-14T14:20:30.068 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:20:29 smithi069 ceph-mon[29472]: fsid dc326fc2-fa65-11ee-bc8e-c7b262605968 2024-04-14T14:20:30.068 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:20:29 smithi069 ceph-mon[29472]: last_changed 2024-04-14T13:51:50.965882+0000 2024-04-14T14:20:30.068 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:20:29 smithi069 ceph-mon[29472]: created 2024-04-14T13:50:18.166086+0000 2024-04-14T14:20:30.068 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:20:29 smithi069 ceph-mon[29472]: min_mon_release 17 (quincy) 2024-04-14T14:20:30.068 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:20:29 smithi069 ceph-mon[29472]: election_strategy: 1 2024-04-14T14:20:30.068 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:20:29 smithi069 ceph-mon[29472]: 0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2024-04-14T14:20:30.068 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:20:29 smithi069 ceph-mon[29472]: 1: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2024-04-14T14:20:30.068 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:20:29 smithi069 ceph-mon[29472]: 2: [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] mon.b 2024-04-14T14:20:30.069 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:20:29 smithi069 ceph-mon[29472]: fsmap 2024-04-14T14:20:30.069 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:20:29 smithi069 ceph-mon[29472]: osdmap e110: 8 total, 8 up, 8 in 2024-04-14T14:20:30.069 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:20:29 smithi069 ceph-mon[29472]: mgrmap e41: y(active, since 3m), standbys: x 2024-04-14T14:20:30.069 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:20:29 smithi069 ceph-mon[29472]: overall HEALTH_OK 2024-04-14T14:20:30.069 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:20:29 smithi069 ceph-mon[29472]: from='mgr.15339 ' entity='' 2024-04-14T14:20:30.069 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:20:29 smithi069 ceph-mon[29472]: mgrmap e42: y(active, since 3m), standbys: x 2024-04-14T14:20:30.069 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:20:29 smithi069 ceph-mon[83680]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-14T14:20:30.069 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:20:29 smithi069 ceph-mon[83680]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-14T14:20:30.069 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:20:29 smithi069 ceph-mon[83680]: from='mgr.15339 172.21.15.69:0/1161558199' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-14T14:20:30.069 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:20:29 smithi069 ceph-mon[83680]: pgmap v114: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:20:30.069 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:20:29 smithi069 ceph-mon[83680]: mon.a calling monitor election 2024-04-14T14:20:30.069 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:20:29 smithi069 ceph-mon[83680]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-04-14T14:20:30.069 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:20:29 smithi069 ceph-mon[83680]: monmap epoch 3 2024-04-14T14:20:30.070 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:20:29 smithi069 ceph-mon[83680]: fsid dc326fc2-fa65-11ee-bc8e-c7b262605968 2024-04-14T14:20:30.070 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:20:29 smithi069 ceph-mon[83680]: last_changed 2024-04-14T13:51:50.965882+0000 2024-04-14T14:20:30.070 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:20:29 smithi069 ceph-mon[83680]: created 2024-04-14T13:50:18.166086+0000 2024-04-14T14:20:30.070 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:20:29 smithi069 ceph-mon[83680]: min_mon_release 17 (quincy) 2024-04-14T14:20:30.070 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:20:29 smithi069 ceph-mon[83680]: election_strategy: 1 2024-04-14T14:20:30.070 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:20:29 smithi069 ceph-mon[83680]: 0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2024-04-14T14:20:30.070 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:20:29 smithi069 ceph-mon[83680]: 1: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2024-04-14T14:20:30.070 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:20:29 smithi069 ceph-mon[83680]: 2: [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] mon.b 2024-04-14T14:20:30.070 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:20:29 smithi069 ceph-mon[83680]: fsmap 2024-04-14T14:20:30.070 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:20:29 smithi069 ceph-mon[83680]: osdmap e110: 8 total, 8 up, 8 in 2024-04-14T14:20:30.070 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:20:29 smithi069 ceph-mon[83680]: mgrmap e41: y(active, since 3m), standbys: x 2024-04-14T14:20:30.070 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:20:29 smithi069 ceph-mon[83680]: overall HEALTH_OK 2024-04-14T14:20:30.070 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:20:29 smithi069 ceph-mon[83680]: from='mgr.15339 ' entity='' 2024-04-14T14:20:30.070 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:20:29 smithi069 ceph-mon[83680]: mgrmap e42: y(active, since 3m), standbys: x 2024-04-14T14:20:30.274 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Apr 14 14:20:30 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-x[61397]: 2024-04-14T14:20:30.173+0000 7f555ac38200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-14T14:20:30.567 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 14 14:20:30 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-y[63038]: 2024-04-14T14:20:30.141+0000 7fdd822c2200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-14T14:20:30.567 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 14 14:20:30 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-y[63038]: 2024-04-14T14:20:30.235+0000 7fdd822c2200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-14T14:20:30.595 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Apr 14 14:20:30 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-x[61397]: 2024-04-14T14:20:30.271+0000 7f555ac38200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-14T14:20:31.168 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Apr 14 14:20:30 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-x[61397]: 2024-04-14T14:20:30.931+0000 7f555ac38200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-14T14:20:31.168 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Apr 14 14:20:31 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-x[61397]: 2024-04-14T14:20:31.023+0000 7f555ac38200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-14T14:20:31.168 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Apr 14 14:20:31 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-x[61397]: 2024-04-14T14:20:31.165+0000 7f555ac38200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-14T14:20:31.211 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 14 14:20:30 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-y[63038]: 2024-04-14T14:20:30.878+0000 7fdd822c2200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-14T14:20:31.211 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 14 14:20:30 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-y[63038]: 2024-04-14T14:20:30.971+0000 7fdd822c2200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-14T14:20:31.211 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 14 14:20:31 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-y[63038]: 2024-04-14T14:20:31.112+0000 7fdd822c2200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-14T14:20:31.211 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 14 14:20:31 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-y[63038]: 2024-04-14T14:20:31.208+0000 7fdd822c2200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-14T14:20:31.563 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 14 14:20:31 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-y[63038]: 2024-04-14T14:20:31.292+0000 7fdd822c2200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-14T14:20:31.580 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Apr 14 14:20:31 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-x[61397]: 2024-04-14T14:20:31.260+0000 7f555ac38200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-14T14:20:31.580 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Apr 14 14:20:31 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-x[61397]: 2024-04-14T14:20:31.343+0000 7f555ac38200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-14T14:20:31.817 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:20:31 smithi069 ceph-mon[83680]: mgrmap e43: y(active, since 3m), standbys: x 2024-04-14T14:20:31.818 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:20:31 smithi069 ceph-mon[29472]: mgrmap e43: y(active, since 3m), standbys: x 2024-04-14T14:20:31.818 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 14 14:20:31 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-y[63038]: 2024-04-14T14:20:31.559+0000 7fdd822c2200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-14T14:20:31.818 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 14 14:20:31 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-y[63038]: 2024-04-14T14:20:31.670+0000 7fdd822c2200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-14T14:20:31.818 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 14 14:20:31 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-y[63038]: 2024-04-14T14:20:31.757+0000 7fdd822c2200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-14T14:20:31.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:20:31 smithi122 ceph-mon[67998]: mgrmap e43: y(active, since 3m), standbys: x 2024-04-14T14:20:31.845 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Apr 14 14:20:31 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-x[61397]: 2024-04-14T14:20:31.606+0000 7f555ac38200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-14T14:20:31.845 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Apr 14 14:20:31 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-x[61397]: 2024-04-14T14:20:31.714+0000 7f555ac38200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-14T14:20:31.846 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Apr 14 14:20:31 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-x[61397]: 2024-04-14T14:20:31.799+0000 7f555ac38200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-14T14:20:32.317 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 14 14:20:31 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-y[63038]: 2024-04-14T14:20:31.852+0000 7fdd822c2200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-14T14:20:32.317 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 14 14:20:32 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-y[63038]: 2024-04-14T14:20:32.030+0000 7fdd822c2200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-14T14:20:32.345 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Apr 14 14:20:31 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-x[61397]: 2024-04-14T14:20:31.894+0000 7f555ac38200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-14T14:20:32.345 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Apr 14 14:20:32 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-x[61397]: 2024-04-14T14:20:32.068+0000 7f555ac38200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-14T14:20:33.317 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 14 14:20:32 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-y[63038]: 2024-04-14T14:20:32.991+0000 7fdd822c2200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-14T14:20:33.317 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 14 14:20:33 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-y[63038]: 2024-04-14T14:20:33.123+0000 7fdd822c2200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-14T14:20:33.345 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Apr 14 14:20:33 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-x[61397]: 2024-04-14T14:20:33.020+0000 7f555ac38200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-14T14:20:33.345 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Apr 14 14:20:33 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-x[61397]: 2024-04-14T14:20:33.149+0000 7f555ac38200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-14T14:20:33.766 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 14 14:20:33 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-y[63038]: 2024-04-14T14:20:33.467+0000 7fdd822c2200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-14T14:20:33.766 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 14 14:20:33 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-y[63038]: 2024-04-14T14:20:33.560+0000 7fdd822c2200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-14T14:20:33.766 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 14 14:20:33 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-y[63038]: 2024-04-14T14:20:33.764+0000 7fdd822c2200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-14T14:20:33.781 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Apr 14 14:20:33 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-x[61397]: 2024-04-14T14:20:33.487+0000 7f555ac38200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-14T14:20:33.781 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Apr 14 14:20:33 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-x[61397]: 2024-04-14T14:20:33.580+0000 7f555ac38200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-14T14:20:33.781 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Apr 14 14:20:33 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-x[61397]: 2024-04-14T14:20:33.779+0000 7f555ac38200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-14T14:20:34.067 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 14 14:20:33 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-y[63038]: 2024-04-14T14:20:33.955+0000 7fdd822c2200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-14T14:20:34.095 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Apr 14 14:20:33 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-x[61397]: 2024-04-14T14:20:33.970+0000 7f555ac38200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-14T14:20:34.514 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 14 14:20:34 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-y[63038]: 2024-04-14T14:20:34.250+0000 7fdd822c2200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-14T14:20:34.528 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Apr 14 14:20:34 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-x[61397]: 2024-04-14T14:20:34.269+0000 7f555ac38200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-14T14:20:34.818 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 14 14:20:34 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-y[63038]: 2024-04-14T14:20:34.512+0000 7fdd822c2200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-14T14:20:34.818 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 14 14:20:34 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-y[63038]: 2024-04-14T14:20:34.756+0000 7fdd822c2200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-14T14:20:34.845 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Apr 14 14:20:34 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-x[61397]: 2024-04-14T14:20:34.525+0000 7f555ac38200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-14T14:20:34.845 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Apr 14 14:20:34 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-x[61397]: 2024-04-14T14:20:34.764+0000 7f555ac38200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-14T14:20:35.845 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Apr 14 14:20:35 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-x[61397]: 2024-04-14T14:20:35.575+0000 7f555ac38200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-14T14:20:36.067 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 14 14:20:35 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-y[63038]: 2024-04-14T14:20:35.581+0000 7fdd822c2200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-14T14:20:36.466 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Apr 14 14:20:36 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-x[61397]: 2024-04-14T14:20:36.146+0000 7f555ac38200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-14T14:20:36.467 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Apr 14 14:20:36 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-x[61397]: 2024-04-14T14:20:36.244+0000 7f555ac38200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-14T14:20:36.467 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Apr 14 14:20:36 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-x[61397]: 2024-04-14T14:20:36.464+0000 7f555ac38200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-14T14:20:36.488 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 14 14:20:36 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-y[63038]: 2024-04-14T14:20:36.160+0000 7fdd822c2200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-14T14:20:36.488 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 14 14:20:36 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-y[63038]: 2024-04-14T14:20:36.259+0000 7fdd822c2200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-14T14:20:36.817 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 14 14:20:36 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-y[63038]: 2024-04-14T14:20:36.486+0000 7fdd822c2200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-14T14:20:36.817 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 14 14:20:36 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-y[63038]: [14/Apr/2024:14:20:36] ENGINE Bus STARTING 2024-04-14T14:20:36.817 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 14 14:20:36 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-y[63038]: CherryPy Checker: 2024-04-14T14:20:36.817 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 14 14:20:36 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-y[63038]: The Application mounted at '' has an empty config. 2024-04-14T14:20:36.817 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 14 14:20:36 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-y[63038]: 2024-04-14T14:20:36.818 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 14 14:20:36 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-y[63038]: [14/Apr/2024:14:20:36] ENGINE Serving on http://:::9283 2024-04-14T14:20:36.818 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 14 14:20:36 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-y[63038]: [14/Apr/2024:14:20:36] ENGINE Bus STARTED 2024-04-14T14:20:36.818 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:20:36 smithi069 ceph-mon[83680]: Standby manager daemon x restarted 2024-04-14T14:20:36.818 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:20:36 smithi069 ceph-mon[83680]: Standby manager daemon x started 2024-04-14T14:20:36.818 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:20:36 smithi069 ceph-mon[83680]: from='mgr.? 172.21.15.122:0/3578745082' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-04-14T14:20:36.818 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:20:36 smithi069 ceph-mon[83680]: from='mgr.? 172.21.15.122:0/3578745082' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-14T14:20:36.818 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:20:36 smithi069 ceph-mon[83680]: from='mgr.? 172.21.15.122:0/3578745082' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-04-14T14:20:36.818 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:20:36 smithi069 ceph-mon[83680]: from='mgr.? 172.21.15.122:0/3578745082' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-14T14:20:36.818 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:20:36 smithi069 ceph-mon[83680]: Active manager daemon y restarted 2024-04-14T14:20:36.819 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:20:36 smithi069 ceph-mon[83680]: Activating manager daemon y 2024-04-14T14:20:36.819 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:20:36 smithi069 ceph-mon[83680]: osdmap e111: 8 total, 8 up, 8 in 2024-04-14T14:20:36.819 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:20:36 smithi069 ceph-mon[83680]: mgrmap e44: y(active, starting, since 0.0075093s), standbys: x 2024-04-14T14:20:36.819 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:20:36 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-14T14:20:36.819 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:20:36 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-14T14:20:36.819 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:20:36 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-14T14:20:36.819 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:20:36 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-04-14T14:20:36.819 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:20:36 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-04-14T14:20:36.819 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:20:36 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-14T14:20:36.819 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:20:36 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-14T14:20:36.819 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:20:36 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-14T14:20:36.819 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:20:36 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-14T14:20:36.819 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:20:36 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-14T14:20:36.819 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:20:36 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-14T14:20:36.819 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:20:36 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-14T14:20:36.820 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:20:36 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-14T14:20:36.820 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:20:36 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-04-14T14:20:36.820 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:20:36 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-04-14T14:20:36.820 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:20:36 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-04-14T14:20:36.820 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:20:36 smithi069 ceph-mon[83680]: Manager daemon y is now available 2024-04-14T14:20:36.820 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:20:36 smithi069 ceph-mon[29472]: Standby manager daemon x restarted 2024-04-14T14:20:36.820 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:20:36 smithi069 ceph-mon[29472]: Standby manager daemon x started 2024-04-14T14:20:36.820 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:20:36 smithi069 ceph-mon[29472]: from='mgr.? 172.21.15.122:0/3578745082' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-04-14T14:20:36.820 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:20:36 smithi069 ceph-mon[29472]: from='mgr.? 172.21.15.122:0/3578745082' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-14T14:20:36.820 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:20:36 smithi069 ceph-mon[29472]: from='mgr.? 172.21.15.122:0/3578745082' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-04-14T14:20:36.820 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:20:36 smithi069 ceph-mon[29472]: from='mgr.? 172.21.15.122:0/3578745082' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-14T14:20:36.820 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:20:36 smithi069 ceph-mon[29472]: Active manager daemon y restarted 2024-04-14T14:20:36.821 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:20:36 smithi069 ceph-mon[29472]: Activating manager daemon y 2024-04-14T14:20:36.821 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:20:36 smithi069 ceph-mon[29472]: osdmap e111: 8 total, 8 up, 8 in 2024-04-14T14:20:36.821 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:20:36 smithi069 ceph-mon[29472]: mgrmap e44: y(active, starting, since 0.0075093s), standbys: x 2024-04-14T14:20:36.821 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:20:36 smithi069 ceph-mon[29472]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-14T14:20:36.821 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:20:36 smithi069 ceph-mon[29472]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-14T14:20:36.821 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:20:36 smithi069 ceph-mon[29472]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-14T14:20:36.821 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:20:36 smithi069 ceph-mon[29472]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-04-14T14:20:36.821 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:20:36 smithi069 ceph-mon[29472]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-04-14T14:20:36.821 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:20:36 smithi069 ceph-mon[29472]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-14T14:20:36.821 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:20:36 smithi069 ceph-mon[29472]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-14T14:20:36.821 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:20:36 smithi069 ceph-mon[29472]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-14T14:20:36.821 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:20:36 smithi069 ceph-mon[29472]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-14T14:20:36.821 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:20:36 smithi069 ceph-mon[29472]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-14T14:20:36.821 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:20:36 smithi069 ceph-mon[29472]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-14T14:20:36.821 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:20:36 smithi069 ceph-mon[29472]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-14T14:20:36.822 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:20:36 smithi069 ceph-mon[29472]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-14T14:20:36.822 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:20:36 smithi069 ceph-mon[29472]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-04-14T14:20:36.822 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:20:36 smithi069 ceph-mon[29472]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-04-14T14:20:36.822 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:20:36 smithi069 ceph-mon[29472]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-04-14T14:20:36.822 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:20:36 smithi069 ceph-mon[29472]: Manager daemon y is now available 2024-04-14T14:20:36.845 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Apr 14 14:20:36 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-x[61397]: [14/Apr/2024:14:20:36] ENGINE Bus STARTING 2024-04-14T14:20:36.845 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Apr 14 14:20:36 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-x[61397]: CherryPy Checker: 2024-04-14T14:20:36.845 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Apr 14 14:20:36 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-x[61397]: The Application mounted at '' has an empty config. 2024-04-14T14:20:36.845 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Apr 14 14:20:36 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-x[61397]: 2024-04-14T14:20:36.845 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Apr 14 14:20:36 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-x[61397]: [14/Apr/2024:14:20:36] ENGINE Serving on http://:::9283 2024-04-14T14:20:36.845 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Apr 14 14:20:36 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-x[61397]: [14/Apr/2024:14:20:36] ENGINE Bus STARTED 2024-04-14T14:20:36.846 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:20:36 smithi122 ceph-mon[67998]: Standby manager daemon x restarted 2024-04-14T14:20:36.846 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:20:36 smithi122 ceph-mon[67998]: Standby manager daemon x started 2024-04-14T14:20:36.846 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:20:36 smithi122 ceph-mon[67998]: from='mgr.? 172.21.15.122:0/3578745082' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-04-14T14:20:36.846 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:20:36 smithi122 ceph-mon[67998]: from='mgr.? 172.21.15.122:0/3578745082' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-14T14:20:36.846 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:20:36 smithi122 ceph-mon[67998]: from='mgr.? 172.21.15.122:0/3578745082' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-04-14T14:20:36.846 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:20:36 smithi122 ceph-mon[67998]: from='mgr.? 172.21.15.122:0/3578745082' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-14T14:20:36.846 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:20:36 smithi122 ceph-mon[67998]: Active manager daemon y restarted 2024-04-14T14:20:36.846 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:20:36 smithi122 ceph-mon[67998]: Activating manager daemon y 2024-04-14T14:20:36.846 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:20:36 smithi122 ceph-mon[67998]: osdmap e111: 8 total, 8 up, 8 in 2024-04-14T14:20:36.846 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:20:36 smithi122 ceph-mon[67998]: mgrmap e44: y(active, starting, since 0.0075093s), standbys: x 2024-04-14T14:20:36.846 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:20:36 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-14T14:20:36.846 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:20:36 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-14T14:20:36.846 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:20:36 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-14T14:20:36.847 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:20:36 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-04-14T14:20:36.847 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:20:36 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-04-14T14:20:36.847 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:20:36 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-14T14:20:36.847 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:20:36 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-14T14:20:36.847 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:20:36 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-14T14:20:36.847 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:20:36 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-14T14:20:36.847 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:20:36 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-14T14:20:36.847 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:20:36 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-14T14:20:36.847 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:20:36 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-14T14:20:36.847 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:20:36 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-14T14:20:36.847 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:20:36 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-04-14T14:20:36.847 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:20:36 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-04-14T14:20:36.847 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:20:36 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-04-14T14:20:36.847 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:20:36 smithi122 ceph-mon[67998]: Manager daemon y is now available 2024-04-14T14:20:37.809 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 14 14:20:37 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-y[63038]: 2024-04-14T14:20:37.500+0000 7fdd38467700 -1 mgr.server handle_report got status from non-daemon mon.a 2024-04-14T14:20:37.810 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:20:37 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T14:20:37.810 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:20:37 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-04-14T14:20:37.810 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:20:37 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-04-14T14:20:37.810 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:20:37 smithi069 ceph-mon[83680]: mgrmap e45: y(active, since 1.01156s), standbys: x 2024-04-14T14:20:37.810 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:20:37 smithi069 ceph-mon[29472]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T14:20:37.810 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:20:37 smithi069 ceph-mon[29472]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-04-14T14:20:37.810 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:20:37 smithi069 ceph-mon[29472]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-04-14T14:20:37.811 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:20:37 smithi069 ceph-mon[29472]: mgrmap e45: y(active, since 1.01156s), standbys: x 2024-04-14T14:20:37.814 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:20:37 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T14:20:37.814 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:20:37 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-04-14T14:20:37.815 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:20:37 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-04-14T14:20:37.815 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:20:37 smithi122 ceph-mon[67998]: mgrmap e45: y(active, since 1.01156s), standbys: x 2024-04-14T14:20:38.817 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:20:38 smithi069 ceph-mon[83680]: [14/Apr/2024:14:20:36] ENGINE Bus STARTING 2024-04-14T14:20:38.817 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:20:38 smithi069 ceph-mon[83680]: [14/Apr/2024:14:20:37] ENGINE Serving on http://172.21.15.69:8765 2024-04-14T14:20:38.817 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:20:38 smithi069 ceph-mon[83680]: [14/Apr/2024:14:20:37] ENGINE Serving on https://172.21.15.69:7150 2024-04-14T14:20:38.817 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:20:38 smithi069 ceph-mon[83680]: [14/Apr/2024:14:20:37] ENGINE Bus STARTED 2024-04-14T14:20:38.817 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:20:38 smithi069 ceph-mon[83680]: from='client.25417 -' entity='client.iscsi.foo.smithi069.yuumtr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T14:20:38.818 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:20:38 smithi069 ceph-mon[83680]: pgmap v3: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail 2024-04-14T14:20:38.818 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:20:38 smithi069 ceph-mon[29472]: [14/Apr/2024:14:20:36] ENGINE Bus STARTING 2024-04-14T14:20:38.818 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:20:38 smithi069 ceph-mon[29472]: [14/Apr/2024:14:20:37] ENGINE Serving on http://172.21.15.69:8765 2024-04-14T14:20:38.818 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:20:38 smithi069 ceph-mon[29472]: [14/Apr/2024:14:20:37] ENGINE Serving on https://172.21.15.69:7150 2024-04-14T14:20:38.818 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:20:38 smithi069 ceph-mon[29472]: [14/Apr/2024:14:20:37] ENGINE Bus STARTED 2024-04-14T14:20:38.818 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:20:38 smithi069 ceph-mon[29472]: from='client.25417 -' entity='client.iscsi.foo.smithi069.yuumtr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T14:20:38.818 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:20:38 smithi069 ceph-mon[29472]: pgmap v3: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail 2024-04-14T14:20:38.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:20:38 smithi122 ceph-mon[67998]: [14/Apr/2024:14:20:36] ENGINE Bus STARTING 2024-04-14T14:20:38.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:20:38 smithi122 ceph-mon[67998]: [14/Apr/2024:14:20:37] ENGINE Serving on http://172.21.15.69:8765 2024-04-14T14:20:38.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:20:38 smithi122 ceph-mon[67998]: [14/Apr/2024:14:20:37] ENGINE Serving on https://172.21.15.69:7150 2024-04-14T14:20:38.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:20:38 smithi122 ceph-mon[67998]: [14/Apr/2024:14:20:37] ENGINE Bus STARTED 2024-04-14T14:20:38.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:20:38 smithi122 ceph-mon[67998]: from='client.25417 -' entity='client.iscsi.foo.smithi069.yuumtr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T14:20:38.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:20:38 smithi122 ceph-mon[67998]: pgmap v3: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail 2024-04-14T14:20:39.817 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:20:39 smithi069 ceph-mon[83680]: pgmap v4: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail 2024-04-14T14:20:39.817 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:20:39 smithi069 ceph-mon[83680]: mgrmap e46: y(active, since 2s), standbys: x 2024-04-14T14:20:39.817 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:20:39 smithi069 ceph-mon[29472]: pgmap v4: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail 2024-04-14T14:20:39.818 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:20:39 smithi069 ceph-mon[29472]: mgrmap e46: y(active, since 2s), standbys: x 2024-04-14T14:20:39.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:20:39 smithi122 ceph-mon[67998]: pgmap v4: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail 2024-04-14T14:20:39.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:20:39 smithi122 ceph-mon[67998]: mgrmap e46: y(active, since 2s), standbys: x 2024-04-14T14:20:41.818 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:20:41 smithi069 ceph-mon[83680]: pgmap v5: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail 2024-04-14T14:20:41.818 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:20:41 smithi069 ceph-mon[83680]: mgrmap e47: y(active, since 4s), standbys: x 2024-04-14T14:20:41.818 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:20:41 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:20:41.818 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:20:41 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:20:41.818 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:20:41 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:20:41.818 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:20:41 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:20:41.819 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:20:41 smithi069 ceph-mon[29472]: pgmap v5: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail 2024-04-14T14:20:41.819 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:20:41 smithi069 ceph-mon[29472]: mgrmap e47: y(active, since 4s), standbys: x 2024-04-14T14:20:41.819 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:20:41 smithi069 ceph-mon[29472]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:20:41.819 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:20:41 smithi069 ceph-mon[29472]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:20:41.819 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:20:41 smithi069 ceph-mon[29472]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:20:41.819 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:20:41 smithi069 ceph-mon[29472]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:20:41.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:20:41 smithi122 ceph-mon[67998]: pgmap v5: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail 2024-04-14T14:20:41.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:20:41 smithi122 ceph-mon[67998]: mgrmap e47: y(active, since 4s), standbys: x 2024-04-14T14:20:41.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:20:41 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:20:41.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:20:41 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:20:41.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:20:41 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:20:41.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:20:41 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:20:43.567 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:20:43 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:20:43.567 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:20:43 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:20:43.568 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:20:43 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:20:43.568 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:20:43 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:20:43.568 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:20:43 smithi069 ceph-mon[83680]: pgmap v6: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail 2024-04-14T14:20:43.568 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:20:43 smithi069 ceph-mon[29472]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:20:43.568 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:20:43 smithi069 ceph-mon[29472]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:20:43.568 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:20:43 smithi069 ceph-mon[29472]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:20:43.568 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:20:43 smithi069 ceph-mon[29472]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:20:43.568 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:20:43 smithi069 ceph-mon[29472]: pgmap v6: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail 2024-04-14T14:20:43.595 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:20:43 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:20:43.595 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:20:43 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:20:43.595 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:20:43 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:20:43.595 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:20:43 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:20:43.595 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:20:43 smithi122 ceph-mon[67998]: pgmap v6: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail 2024-04-14T14:20:45.719 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:20:45 smithi122 ceph-mon[67998]: pgmap v7: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 30 KiB/s rd, 0 B/s wr, 17 op/s 2024-04-14T14:20:45.817 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:20:45 smithi069 ceph-mon[29472]: pgmap v7: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 30 KiB/s rd, 0 B/s wr, 17 op/s 2024-04-14T14:20:45.817 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:20:45 smithi069 ceph-mon[83680]: pgmap v7: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 30 KiB/s rd, 0 B/s wr, 17 op/s 2024-04-14T14:20:46.777 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:20:46 smithi069 ceph-mon[83680]: Detected new or changed devices on smithi069 2024-04-14T14:20:46.777 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:20:46 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:20:46.777 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:20:46 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:20:46.777 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:20:46 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:20:46.777 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:20:46 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:20:46.777 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:20:46 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T14:20:46.777 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:20:46 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-14T14:20:46.777 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 14 14:20:46 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-y[63038]: ::ffff:172.21.15.122 - - [14/Apr/2024:14:20:46] "GET /metrics HTTP/1.1" 200 35206 "" "Prometheus/2.43.0" 2024-04-14T14:20:46.778 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:20:46 smithi069 ceph-mon[29472]: Detected new or changed devices on smithi069 2024-04-14T14:20:46.778 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:20:46 smithi069 ceph-mon[29472]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:20:46.778 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:20:46 smithi069 ceph-mon[29472]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:20:46.778 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:20:46 smithi069 ceph-mon[29472]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:20:46.778 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:20:46 smithi069 ceph-mon[29472]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:20:46.778 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:20:46 smithi069 ceph-mon[29472]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T14:20:46.778 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:20:46 smithi069 ceph-mon[29472]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-14T14:20:46.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:20:46 smithi122 ceph-mon[67998]: Detected new or changed devices on smithi069 2024-04-14T14:20:46.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:20:46 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:20:46.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:20:46 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:20:46.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:20:46 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:20:46.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:20:46 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:20:46.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:20:46 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T14:20:46.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:20:46 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-14T14:20:47.793 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:20:47 smithi069 ceph-mon[83680]: Updating smithi069:/etc/ceph/ceph.conf 2024-04-14T14:20:47.793 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:20:47 smithi069 ceph-mon[83680]: Updating smithi122:/etc/ceph/ceph.conf 2024-04-14T14:20:47.793 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:20:47 smithi069 ceph-mon[83680]: Updating smithi122:/var/lib/ceph/dc326fc2-fa65-11ee-bc8e-c7b262605968/config/ceph.conf 2024-04-14T14:20:47.794 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:20:47 smithi069 ceph-mon[83680]: Updating smithi069:/var/lib/ceph/dc326fc2-fa65-11ee-bc8e-c7b262605968/config/ceph.conf 2024-04-14T14:20:47.794 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:20:47 smithi069 ceph-mon[83680]: Updating smithi122:/etc/ceph/ceph.client.admin.keyring 2024-04-14T14:20:47.794 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:20:47 smithi069 ceph-mon[83680]: Updating smithi069:/etc/ceph/ceph.client.admin.keyring 2024-04-14T14:20:47.794 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:20:47 smithi069 ceph-mon[83680]: pgmap v8: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 23 KiB/s rd, 0 B/s wr, 13 op/s 2024-04-14T14:20:47.794 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:20:47 smithi069 ceph-mon[83680]: Updating smithi122:/var/lib/ceph/dc326fc2-fa65-11ee-bc8e-c7b262605968/config/ceph.client.admin.keyring 2024-04-14T14:20:47.794 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:20:47 smithi069 ceph-mon[83680]: Updating smithi069:/var/lib/ceph/dc326fc2-fa65-11ee-bc8e-c7b262605968/config/ceph.client.admin.keyring 2024-04-14T14:20:47.794 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:20:47 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:20:47.794 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:20:47 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:20:47.794 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:20:47 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:20:47.794 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:20:47 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:20:47.794 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:20:47 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:20:47.794 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:20:47 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-14T14:20:47.794 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:20:47 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T14:20:47.794 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:20:47 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "quorum_status"}]: dispatch 2024-04-14T14:20:47.795 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:20:47 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "mon ok-to-stop", "ids": ["c"]}]: dispatch 2024-04-14T14:20:47.795 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:20:47 smithi069 ceph-mon[29472]: Updating smithi069:/etc/ceph/ceph.conf 2024-04-14T14:20:47.795 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:20:47 smithi069 ceph-mon[29472]: Updating smithi122:/etc/ceph/ceph.conf 2024-04-14T14:20:47.795 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:20:47 smithi069 ceph-mon[29472]: Updating smithi122:/var/lib/ceph/dc326fc2-fa65-11ee-bc8e-c7b262605968/config/ceph.conf 2024-04-14T14:20:47.795 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:20:47 smithi069 ceph-mon[29472]: Updating smithi069:/var/lib/ceph/dc326fc2-fa65-11ee-bc8e-c7b262605968/config/ceph.conf 2024-04-14T14:20:47.795 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:20:47 smithi069 ceph-mon[29472]: Updating smithi122:/etc/ceph/ceph.client.admin.keyring 2024-04-14T14:20:47.795 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:20:47 smithi069 ceph-mon[29472]: Updating smithi069:/etc/ceph/ceph.client.admin.keyring 2024-04-14T14:20:47.795 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:20:47 smithi069 ceph-mon[29472]: pgmap v8: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 23 KiB/s rd, 0 B/s wr, 13 op/s 2024-04-14T14:20:47.795 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:20:47 smithi069 ceph-mon[29472]: Updating smithi122:/var/lib/ceph/dc326fc2-fa65-11ee-bc8e-c7b262605968/config/ceph.client.admin.keyring 2024-04-14T14:20:47.795 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:20:47 smithi069 ceph-mon[29472]: Updating smithi069:/var/lib/ceph/dc326fc2-fa65-11ee-bc8e-c7b262605968/config/ceph.client.admin.keyring 2024-04-14T14:20:47.795 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:20:47 smithi069 ceph-mon[29472]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:20:47.795 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:20:47 smithi069 ceph-mon[29472]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:20:47.795 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:20:47 smithi069 ceph-mon[29472]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:20:47.796 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:20:47 smithi069 ceph-mon[29472]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:20:47.796 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:20:47 smithi069 ceph-mon[29472]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:20:47.796 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:20:47 smithi069 ceph-mon[29472]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-14T14:20:47.796 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:20:47 smithi069 ceph-mon[29472]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T14:20:47.796 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:20:47 smithi069 ceph-mon[29472]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "quorum_status"}]: dispatch 2024-04-14T14:20:47.796 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:20:47 smithi069 ceph-mon[29472]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "mon ok-to-stop", "ids": ["c"]}]: dispatch 2024-04-14T14:20:47.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:20:47 smithi122 ceph-mon[67998]: Updating smithi069:/etc/ceph/ceph.conf 2024-04-14T14:20:47.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:20:47 smithi122 ceph-mon[67998]: Updating smithi122:/etc/ceph/ceph.conf 2024-04-14T14:20:47.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:20:47 smithi122 ceph-mon[67998]: Updating smithi122:/var/lib/ceph/dc326fc2-fa65-11ee-bc8e-c7b262605968/config/ceph.conf 2024-04-14T14:20:47.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:20:47 smithi122 ceph-mon[67998]: Updating smithi069:/var/lib/ceph/dc326fc2-fa65-11ee-bc8e-c7b262605968/config/ceph.conf 2024-04-14T14:20:47.846 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:20:47 smithi122 ceph-mon[67998]: Updating smithi122:/etc/ceph/ceph.client.admin.keyring 2024-04-14T14:20:47.846 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:20:47 smithi122 ceph-mon[67998]: Updating smithi069:/etc/ceph/ceph.client.admin.keyring 2024-04-14T14:20:47.846 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:20:47 smithi122 ceph-mon[67998]: pgmap v8: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 23 KiB/s rd, 0 B/s wr, 13 op/s 2024-04-14T14:20:47.846 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:20:47 smithi122 ceph-mon[67998]: Updating smithi122:/var/lib/ceph/dc326fc2-fa65-11ee-bc8e-c7b262605968/config/ceph.client.admin.keyring 2024-04-14T14:20:47.846 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:20:47 smithi122 ceph-mon[67998]: Updating smithi069:/var/lib/ceph/dc326fc2-fa65-11ee-bc8e-c7b262605968/config/ceph.client.admin.keyring 2024-04-14T14:20:47.846 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:20:47 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:20:47.846 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:20:47 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:20:47.846 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:20:47 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:20:47.846 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:20:47 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:20:47.846 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:20:47 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:20:47.846 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:20:47 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-14T14:20:47.846 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:20:47 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T14:20:47.846 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:20:47 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "quorum_status"}]: dispatch 2024-04-14T14:20:47.846 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:20:47 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "mon ok-to-stop", "ids": ["c"]}]: dispatch 2024-04-14T14:20:48.817 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:20:48 smithi069 ceph-mon[83680]: Upgrade: It appears safe to stop mon.c 2024-04-14T14:20:48.817 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:20:48 smithi069 ceph-mon[83680]: from='client.25417 -' entity='client.iscsi.foo.smithi069.yuumtr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T14:20:48.817 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:20:48 smithi069 ceph-mon[29472]: Upgrade: It appears safe to stop mon.c 2024-04-14T14:20:48.817 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:20:48 smithi069 ceph-mon[29472]: from='client.25417 -' entity='client.iscsi.foo.smithi069.yuumtr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T14:20:48.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:20:48 smithi122 ceph-mon[67998]: Upgrade: It appears safe to stop mon.c 2024-04-14T14:20:48.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:20:48 smithi122 ceph-mon[67998]: from='client.25417 -' entity='client.iscsi.foo.smithi069.yuumtr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T14:20:49.818 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:20:49 smithi069 ceph-mon[83680]: pgmap v9: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 20 KiB/s rd, 0 B/s wr, 11 op/s 2024-04-14T14:20:49.818 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:20:49 smithi069 ceph-mon[83680]: Upgrade: Updating mon.c 2024-04-14T14:20:49.818 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:20:49 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:20:49.818 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:20:49 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-14T14:20:49.818 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:20:49 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-14T14:20:49.818 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:20:49 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T14:20:49.818 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:20:49 smithi069 ceph-mon[83680]: Deploying daemon mon.c on smithi069 2024-04-14T14:20:49.819 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:20:49 smithi069 ceph-mon[29472]: pgmap v9: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 20 KiB/s rd, 0 B/s wr, 11 op/s 2024-04-14T14:20:49.819 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:20:49 smithi069 ceph-mon[29472]: Upgrade: Updating mon.c 2024-04-14T14:20:49.819 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:20:49 smithi069 ceph-mon[29472]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:20:49.819 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:20:49 smithi069 ceph-mon[29472]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-14T14:20:49.819 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:20:49 smithi069 ceph-mon[29472]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-14T14:20:49.819 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:20:49 smithi069 ceph-mon[29472]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T14:20:49.819 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:20:49 smithi069 ceph-mon[29472]: Deploying daemon mon.c on smithi069 2024-04-14T14:20:50.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:20:49 smithi122 ceph-mon[67998]: pgmap v9: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 20 KiB/s rd, 0 B/s wr, 11 op/s 2024-04-14T14:20:50.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:20:49 smithi122 ceph-mon[67998]: Upgrade: Updating mon.c 2024-04-14T14:20:50.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:20:49 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:20:50.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:20:49 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-14T14:20:50.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:20:49 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-14T14:20:50.096 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:20:49 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T14:20:50.096 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:20:49 smithi122 ceph-mon[67998]: Deploying daemon mon.c on smithi069 2024-04-14T14:20:50.684 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:20:50 smithi069 systemd[1]: Stopping Ceph mon.c for dc326fc2-fa65-11ee-bc8e-c7b262605968... 2024-04-14T14:20:51.017 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:20:50 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mon-c[29467]: 2024-04-14T14:20:50.681+0000 7f0a9ed54700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.c -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-04-14T14:20:51.017 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:20:50 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mon-c[29467]: 2024-04-14T14:20:50.681+0000 7f0a9ed54700 -1 mon.c@1(peon) e3 *** Got Signal Terminated *** 2024-04-14T14:20:51.017 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:20:50 smithi069 podman[88470]: 2024-04-14 14:20:50.723133484 +0000 UTC m=+0.162837429 container stop 23c078b9d1e57e77cfe408a41a18908e24a2a158dc23c923a094a9881cd26c25 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mon-c, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, com.redhat.component=centos-stream-container, name=centos-stream, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, io.openshift.tags=base centos centos-stream, release=754, maintainer=Guillaume Abrioux , url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, vendor=Red Hat, Inc., io.openshift.expose-services=, vcs-type=git, ceph=True, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, io.buildah.version=1.19.8, distribution-scope=public, GIT_CLEAN=True, GIT_REPO=https://github.com/ceph/ceph-container.git, RELEASE=HEAD, version=8, io.k8s.display-name=CentOS Stream 8, CEPH_POINT_RELEASE=-17.2.0, architecture=x86_64, GIT_BRANCH=HEAD, build-date=2022-05-03T08:36:31.336870, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly.) 2024-04-14T14:20:51.017 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:20:50 smithi069 podman[88470]: 2024-04-14 14:20:50.748826499 +0000 UTC m=+0.188530442 container died 23c078b9d1e57e77cfe408a41a18908e24a2a158dc23c923a094a9881cd26c25 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mon-c, maintainer=Guillaume Abrioux , RELEASE=HEAD, io.buildah.version=1.19.8, architecture=x86_64, io.openshift.expose-services=, build-date=2022-05-03T08:36:31.336870, vendor=Red Hat, Inc., distribution-scope=public, GIT_REPO=https://github.com/ceph/ceph-container.git, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, com.redhat.component=centos-stream-container, GIT_CLEAN=True, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, release=754, ceph=True, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, name=centos-stream, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, io.openshift.tags=base centos centos-stream, version=8, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., vcs-type=git, io.k8s.display-name=CentOS Stream 8, GIT_BRANCH=HEAD, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, CEPH_POINT_RELEASE=-17.2.0) 2024-04-14T14:20:51.276 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:20:51 smithi069 podman[88470]: 2024-04-14 14:20:51.115066991 +0000 UTC m=+0.554770933 container cleanup 23c078b9d1e57e77cfe408a41a18908e24a2a158dc23c923a094a9881cd26c25 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mon-c, io.openshift.expose-services=, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, CEPH_POINT_RELEASE=-17.2.0, GIT_REPO=https://github.com/ceph/ceph-container.git, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, vendor=Red Hat, Inc., vcs-type=git, build-date=2022-05-03T08:36:31.336870, architecture=x86_64, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, io.buildah.version=1.19.8, ceph=True, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, maintainer=Guillaume Abrioux , version=8, GIT_CLEAN=True, GIT_BRANCH=HEAD, name=centos-stream, RELEASE=HEAD, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, distribution-scope=public, io.openshift.tags=base centos centos-stream, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., io.k8s.display-name=CentOS Stream 8, release=754, com.redhat.component=centos-stream-container) 2024-04-14T14:20:51.276 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:20:51 smithi069 bash[88470]: ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mon-c 2024-04-14T14:20:51.567 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:20:51 smithi069 podman[88484]: 2024-04-14 14:20:51.274582691 +0000 UTC m=+0.546311867 container remove 23c078b9d1e57e77cfe408a41a18908e24a2a158dc23c923a094a9881cd26c25 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mon-c, GIT_REPO=https://github.com/ceph/ceph-container.git, distribution-scope=public, io.openshift.tags=base centos centos-stream, vendor=Red Hat, Inc., GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, maintainer=Guillaume Abrioux , version=8, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., architecture=x86_64, ceph=True, io.k8s.display-name=CentOS Stream 8, io.openshift.expose-services=, CEPH_POINT_RELEASE=-17.2.0, vcs-type=git, release=754, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, RELEASE=HEAD, GIT_CLEAN=True, com.redhat.component=centos-stream-container, io.buildah.version=1.19.8, name=centos-stream, GIT_BRANCH=HEAD, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, build-date=2022-05-03T08:36:31.336870) 2024-04-14T14:20:51.567 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:20:51 smithi069 systemd[1]: ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968@mon.c.service: Deactivated successfully. 2024-04-14T14:20:51.567 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:20:51 smithi069 systemd[1]: Stopped Ceph mon.c for dc326fc2-fa65-11ee-bc8e-c7b262605968. 2024-04-14T14:20:51.567 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:20:51 smithi069 systemd[1]: ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968@mon.c.service: Consumed 14.288s CPU time. 2024-04-14T14:20:51.973 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:20:51 smithi069 systemd[1]: Starting Ceph mon.c for dc326fc2-fa65-11ee-bc8e-c7b262605968... 2024-04-14T14:20:51.973 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:20:51 smithi069 podman[88617]: 2024-04-14 14:20:51.799993201 +0000 UTC m=+0.018108254 image pull b96c509eef841460b3271f5fdd331860da87b004524f4ed59414aa3680183b53 quay.ceph.io/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888 2024-04-14T14:20:51.973 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:20:51 smithi069 podman[88617]: 2024-04-14 14:20:51.900897922 +0000 UTC m=+0.119012949 container create 84aa95f11d0c392c7b8479408817b8e8804c7d09554d5f4083aadc86bb18d9b7 (image=quay.ceph.io/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888, name=ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mon-c, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=main-a971fd5, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_CLEAN=True, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.build-date=20240403, maintainer=Guillaume Abrioux , GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0) 2024-04-14T14:20:52.317 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:20:52 smithi069 ceph-mon[83680]: pgmap v10: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 10 op/s 2024-04-14T14:20:52.317 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:20:52 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T14:20:52.317 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:20:52 smithi069 podman[88617]: 2024-04-14 14:20:52.086644639 +0000 UTC m=+0.304759668 container init 84aa95f11d0c392c7b8479408817b8e8804c7d09554d5f4083aadc86bb18d9b7 (image=quay.ceph.io/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888, name=ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mon-c, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.build-date=20240403, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, RELEASE=main-a971fd5, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0) 2024-04-14T14:20:52.317 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:20:52 smithi069 podman[88617]: 2024-04-14 14:20:52.089627463 +0000 UTC m=+0.307742491 container start 84aa95f11d0c392c7b8479408817b8e8804c7d09554d5f4083aadc86bb18d9b7 (image=quay.ceph.io/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888, name=ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mon-c, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.build-date=20240403, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=main-a971fd5, maintainer=Guillaume Abrioux , ceph=True, io.buildah.version=1.33.5) 2024-04-14T14:20:52.317 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:20:52 smithi069 ceph-mon[88636]: set uid:gid to 167:167 (ceph:ceph) 2024-04-14T14:20:52.318 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:20:52 smithi069 ceph-mon[88636]: ceph version 19.0.0-2778-ga971fd5f (a971fd5f5719038c70b817bafacd6d5400b1d888) squid (dev), process ceph-mon, pid 2 2024-04-14T14:20:52.318 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:20:52 smithi069 ceph-mon[88636]: pidfile_write: ignore empty --pid-file 2024-04-14T14:20:52.318 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:20:52 smithi069 ceph-mon[88636]: load: jerasure load: lrc 2024-04-14T14:20:52.318 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:20:52 smithi069 ceph-mon[88636]: rocksdb: RocksDB version: 7.9.2 2024-04-14T14:20:52.318 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:20:52 smithi069 ceph-mon[88636]: rocksdb: Git sha 0 2024-04-14T14:20:52.318 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:20:52 smithi069 ceph-mon[88636]: rocksdb: Compile date 2024-04-06 20:28:24 2024-04-14T14:20:52.318 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:20:52 smithi069 ceph-mon[88636]: rocksdb: DB SUMMARY 2024-04-14T14:20:52.318 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:20:52 smithi069 ceph-mon[88636]: rocksdb: DB Session ID: DLW8FS0C47RWA17DF47T 2024-04-14T14:20:52.318 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:20:52 smithi069 ceph-mon[88636]: rocksdb: CURRENT file: CURRENT 2024-04-14T14:20:52.318 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:20:52 smithi069 ceph-mon[88636]: rocksdb: IDENTITY file: IDENTITY 2024-04-14T14:20:52.318 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:20:52 smithi069 ceph-mon[88636]: rocksdb: MANIFEST file: MANIFEST-000009 size: 3200 Bytes 2024-04-14T14:20:52.318 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:20:52 smithi069 ceph-mon[88636]: rocksdb: SST files in /var/lib/ceph/mon/ceph-c/store.db dir, Total Num: 1, files: 000060.sst 2024-04-14T14:20:52.318 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:20:52 smithi069 ceph-mon[88636]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-c/store.db: 000058.log size: 668047 ; 2024-04-14T14:20:52.318 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:20:52 smithi069 ceph-mon[88636]: rocksdb: Options.error_if_exists: 0 2024-04-14T14:20:52.319 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:20:52 smithi069 ceph-mon[88636]: rocksdb: Options.create_if_missing: 0 2024-04-14T14:20:52.319 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:20:52 smithi069 ceph-mon[88636]: rocksdb: Options.paranoid_checks: 1 2024-04-14T14:20:52.319 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:20:52 smithi069 ceph-mon[88636]: rocksdb: Options.flush_verify_memtable_count: 1 2024-04-14T14:20:52.319 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:20:52 smithi069 ceph-mon[88636]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-04-14T14:20:52.319 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:20:52 smithi069 ceph-mon[88636]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-04-14T14:20:52.319 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:20:52 smithi069 ceph-mon[88636]: rocksdb: Options.env: 0x56064775ac60 2024-04-14T14:20:52.319 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:20:52 smithi069 ceph-mon[88636]: rocksdb: Options.fs: PosixFileSystem 2024-04-14T14:20:52.319 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:20:52 smithi069 ceph-mon[88636]: rocksdb: Options.info_log: 0x56064976dbe0 2024-04-14T14:20:52.319 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:20:52 smithi069 ceph-mon[88636]: rocksdb: Options.max_file_opening_threads: 16 2024-04-14T14:20:52.319 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:20:52 smithi069 ceph-mon[88636]: rocksdb: Options.statistics: (nil) 2024-04-14T14:20:52.319 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:20:52 smithi069 ceph-mon[88636]: rocksdb: Options.use_fsync: 0 2024-04-14T14:20:52.319 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:20:52 smithi069 ceph-mon[88636]: rocksdb: Options.max_log_file_size: 0 2024-04-14T14:20:52.319 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:20:52 smithi069 ceph-mon[88636]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-04-14T14:20:52.319 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:20:52 smithi069 ceph-mon[88636]: rocksdb: Options.log_file_time_to_roll: 0 2024-04-14T14:20:52.319 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:20:52 smithi069 ceph-mon[88636]: rocksdb: Options.keep_log_file_num: 1000 2024-04-14T14:20:52.320 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:20:52 smithi069 ceph-mon[88636]: rocksdb: Options.recycle_log_file_num: 0 2024-04-14T14:20:52.320 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:20:52 smithi069 ceph-mon[88636]: rocksdb: Options.allow_fallocate: 1 2024-04-14T14:20:52.320 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:20:52 smithi069 ceph-mon[88636]: rocksdb: Options.allow_mmap_reads: 0 2024-04-14T14:20:52.320 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:20:52 smithi069 ceph-mon[88636]: rocksdb: Options.allow_mmap_writes: 0 2024-04-14T14:20:52.320 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:20:52 smithi069 ceph-mon[88636]: rocksdb: Options.use_direct_reads: 0 2024-04-14T14:20:52.320 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:20:52 smithi069 ceph-mon[88636]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-04-14T14:20:52.320 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:20:52 smithi069 ceph-mon[88636]: rocksdb: Options.create_missing_column_families: 0 2024-04-14T14:20:52.320 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:20:52 smithi069 ceph-mon[88636]: rocksdb: Options.db_log_dir: 2024-04-14T14:20:52.320 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:20:52 smithi069 ceph-mon[88636]: rocksdb: Options.wal_dir: 2024-04-14T14:20:52.320 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:20:52 smithi069 ceph-mon[88636]: rocksdb: Options.table_cache_numshardbits: 6 2024-04-14T14:20:52.320 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:20:52 smithi069 ceph-mon[88636]: rocksdb: Options.WAL_ttl_seconds: 0 2024-04-14T14:20:52.320 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:20:52 smithi069 ceph-mon[88636]: rocksdb: Options.WAL_size_limit_MB: 0 2024-04-14T14:20:52.320 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:20:52 smithi069 ceph-mon[88636]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-04-14T14:20:52.320 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:20:52 smithi069 ceph-mon[88636]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-04-14T14:20:52.320 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:20:52 smithi069 ceph-mon[88636]: rocksdb: Options.is_fd_close_on_exec: 1 2024-04-14T14:20:52.321 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:20:52 smithi069 ceph-mon[88636]: rocksdb: Options.advise_random_on_open: 1 2024-04-14T14:20:52.321 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:20:52 smithi069 ceph-mon[88636]: rocksdb: Options.db_write_buffer_size: 0 2024-04-14T14:20:52.321 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:20:52 smithi069 ceph-mon[88636]: rocksdb: Options.write_buffer_manager: 0x56064973e820 2024-04-14T14:20:52.321 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:20:52 smithi069 ceph-mon[88636]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-04-14T14:20:52.321 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:20:52 smithi069 ceph-mon[88636]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-04-14T14:20:52.321 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:20:52 smithi069 ceph-mon[88636]: rocksdb: Options.use_adaptive_mutex: 0 2024-04-14T14:20:52.321 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:20:52 smithi069 ceph-mon[88636]: rocksdb: Options.rate_limiter: (nil) 2024-04-14T14:20:52.321 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:20:52 smithi069 ceph-mon[88636]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-04-14T14:20:52.321 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:20:52 smithi069 ceph-mon[88636]: rocksdb: Options.wal_recovery_mode: 2 2024-04-14T14:20:52.321 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:20:52 smithi069 ceph-mon[88636]: rocksdb: Options.enable_thread_tracking: 0 2024-04-14T14:20:52.321 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:20:52 smithi069 ceph-mon[88636]: rocksdb: Options.enable_pipelined_write: 0 2024-04-14T14:20:52.321 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:20:52 smithi069 ceph-mon[88636]: rocksdb: Options.unordered_write: 0 2024-04-14T14:20:52.321 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:20:52 smithi069 ceph-mon[88636]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-04-14T14:20:52.321 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:20:52 smithi069 ceph-mon[88636]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-04-14T14:20:52.321 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:20:52 smithi069 ceph-mon[88636]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-04-14T14:20:52.322 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:20:52 smithi069 ceph-mon[88636]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-04-14T14:20:52.322 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:20:52 smithi069 ceph-mon[88636]: rocksdb: Options.row_cache: None 2024-04-14T14:20:52.322 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:20:52 smithi069 ceph-mon[88636]: rocksdb: Options.wal_filter: None 2024-04-14T14:20:52.322 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:20:52 smithi069 ceph-mon[88636]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-04-14T14:20:52.323 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:20:52 smithi069 ceph-mon[88636]: rocksdb: Options.allow_ingest_behind: 0 2024-04-14T14:20:52.323 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:20:52 smithi069 ceph-mon[88636]: rocksdb: Options.two_write_queues: 0 2024-04-14T14:20:52.323 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:20:52 smithi069 ceph-mon[88636]: rocksdb: Options.manual_wal_flush: 0 2024-04-14T14:20:52.323 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:20:52 smithi069 ceph-mon[88636]: rocksdb: Options.wal_compression: 0 2024-04-14T14:20:52.323 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:20:52 smithi069 ceph-mon[88636]: rocksdb: Options.atomic_flush: 0 2024-04-14T14:20:52.323 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:20:52 smithi069 ceph-mon[88636]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-04-14T14:20:52.323 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:20:52 smithi069 ceph-mon[88636]: rocksdb: Options.persist_stats_to_disk: 0 2024-04-14T14:20:52.323 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:20:52 smithi069 ceph-mon[88636]: rocksdb: Options.write_dbid_to_manifest: 0 2024-04-14T14:20:52.323 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:20:52 smithi069 ceph-mon[88636]: rocksdb: Options.log_readahead_size: 0 2024-04-14T14:20:52.324 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:20:52 smithi069 ceph-mon[88636]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-04-14T14:20:52.324 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:20:52 smithi069 ceph-mon[88636]: rocksdb: Options.best_efforts_recovery: 0 2024-04-14T14:20:52.324 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:20:52 smithi069 ceph-mon[88636]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-04-14T14:20:52.324 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:20:52 smithi069 ceph-mon[88636]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-04-14T14:20:52.324 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:20:52 smithi069 ceph-mon[88636]: rocksdb: Options.allow_data_in_errors: 0 2024-04-14T14:20:52.324 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:20:52 smithi069 ceph-mon[88636]: rocksdb: Options.db_host_id: __hostname__ 2024-04-14T14:20:52.324 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:20:52 smithi069 ceph-mon[88636]: rocksdb: Options.enforce_single_del_contracts: true 2024-04-14T14:20:52.324 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:20:52 smithi069 ceph-mon[88636]: rocksdb: Options.max_background_jobs: 2 2024-04-14T14:20:52.324 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:20:52 smithi069 ceph-mon[88636]: rocksdb: Options.max_background_compactions: -1 2024-04-14T14:20:52.324 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:20:52 smithi069 ceph-mon[88636]: rocksdb: Options.max_subcompactions: 1 2024-04-14T14:20:52.324 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:20:52 smithi069 ceph-mon[88636]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-04-14T14:20:52.324 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:20:52 smithi069 ceph-mon[88636]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-04-14T14:20:52.324 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:20:52 smithi069 ceph-mon[88636]: rocksdb: Options.delayed_write_rate : 16777216 2024-04-14T14:20:52.324 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:20:52 smithi069 ceph-mon[88636]: rocksdb: Options.max_total_wal_size: 0 2024-04-14T14:20:52.324 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:20:52 smithi069 ceph-mon[88636]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-04-14T14:20:52.325 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:20:52 smithi069 ceph-mon[88636]: rocksdb: Options.stats_dump_period_sec: 600 2024-04-14T14:20:52.325 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:20:52 smithi069 ceph-mon[88636]: rocksdb: Options.stats_persist_period_sec: 600 2024-04-14T14:20:52.325 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:20:52 smithi069 ceph-mon[88636]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-04-14T14:20:52.325 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:20:52 smithi069 ceph-mon[88636]: rocksdb: Options.max_open_files: -1 2024-04-14T14:20:52.325 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:20:52 smithi069 ceph-mon[88636]: rocksdb: Options.bytes_per_sync: 0 2024-04-14T14:20:52.325 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:20:52 smithi069 ceph-mon[88636]: rocksdb: Options.wal_bytes_per_sync: 0 2024-04-14T14:20:52.325 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:20:52 smithi069 ceph-mon[88636]: rocksdb: Options.strict_bytes_per_sync: 0 2024-04-14T14:20:52.325 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:20:52 smithi069 ceph-mon[88636]: rocksdb: Options.compaction_readahead_size: 0 2024-04-14T14:20:52.325 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:20:52 smithi069 ceph-mon[88636]: rocksdb: Options.max_background_flushes: -1 2024-04-14T14:20:52.325 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:20:52 smithi069 ceph-mon[88636]: rocksdb: Compression algorithms supported: 2024-04-14T14:20:52.325 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:20:52 smithi069 ceph-mon[88636]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-04-14T14:20:52.325 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:20:52 smithi069 ceph-mon[88636]: rocksdb: kZSTD supported: 0 2024-04-14T14:20:52.325 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:20:52 smithi069 ceph-mon[88636]: rocksdb: kXpressCompression supported: 0 2024-04-14T14:20:52.325 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:20:52 smithi069 ceph-mon[88636]: rocksdb: kLZ4HCCompression supported: 1 2024-04-14T14:20:52.325 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:20:52 smithi069 ceph-mon[88636]: rocksdb: kZlibCompression supported: 1 2024-04-14T14:20:52.326 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:20:52 smithi069 ceph-mon[88636]: rocksdb: kSnappyCompression supported: 1 2024-04-14T14:20:52.326 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:20:52 smithi069 ceph-mon[88636]: rocksdb: kLZ4Compression supported: 1 2024-04-14T14:20:52.326 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:20:52 smithi069 ceph-mon[88636]: rocksdb: kBZip2Compression supported: 0 2024-04-14T14:20:52.326 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:20:52 smithi069 ceph-mon[88636]: rocksdb: Fast CRC32 supported: Supported on x86 2024-04-14T14:20:52.326 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:20:52 smithi069 ceph-mon[88636]: rocksdb: DMutex implementation: pthread_mutex_t 2024-04-14T14:20:52.326 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:20:52 smithi069 ceph-mon[88636]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-c/store.db/MANIFEST-000009 2024-04-14T14:20:52.326 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:20:52 smithi069 ceph-mon[88636]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-04-14T14:20:52.326 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:20:52 smithi069 ceph-mon[88636]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-04-14T14:20:52.326 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:20:52 smithi069 ceph-mon[88636]: rocksdb: Options.merge_operator: 2024-04-14T14:20:52.326 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:20:52 smithi069 ceph-mon[88636]: rocksdb: Options.compaction_filter: None 2024-04-14T14:20:52.326 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:20:52 smithi069 ceph-mon[88636]: rocksdb: Options.compaction_filter_factory: None 2024-04-14T14:20:52.326 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:20:52 smithi069 ceph-mon[88636]: rocksdb: Options.sst_partitioner_factory: None 2024-04-14T14:20:52.326 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:20:52 smithi069 ceph-mon[88636]: rocksdb: Options.memtable_factory: SkipListFactory 2024-04-14T14:20:52.326 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:20:52 smithi069 ceph-mon[88636]: rocksdb: Options.table_factory: BlockBasedTable 2024-04-14T14:20:52.326 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:20:52 smithi069 ceph-mon[88636]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x56064976ddc0) 2024-04-14T14:20:52.327 INFO:journalctl@ceph.mon.c.smithi069.stdout: cache_index_and_filter_blocks: 1 2024-04-14T14:20:52.327 INFO:journalctl@ceph.mon.c.smithi069.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-04-14T14:20:52.327 INFO:journalctl@ceph.mon.c.smithi069.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-04-14T14:20:52.327 INFO:journalctl@ceph.mon.c.smithi069.stdout: pin_top_level_index_and_filter: 1 2024-04-14T14:20:52.327 INFO:journalctl@ceph.mon.c.smithi069.stdout: index_type: 0 2024-04-14T14:20:52.327 INFO:journalctl@ceph.mon.c.smithi069.stdout: data_block_index_type: 0 2024-04-14T14:20:52.327 INFO:journalctl@ceph.mon.c.smithi069.stdout: index_shortening: 1 2024-04-14T14:20:52.327 INFO:journalctl@ceph.mon.c.smithi069.stdout: data_block_hash_table_util_ratio: 0.750000 2024-04-14T14:20:52.327 INFO:journalctl@ceph.mon.c.smithi069.stdout: checksum: 4 2024-04-14T14:20:52.327 INFO:journalctl@ceph.mon.c.smithi069.stdout: no_block_cache: 0 2024-04-14T14:20:52.327 INFO:journalctl@ceph.mon.c.smithi069.stdout: block_cache: 0x560649771090 2024-04-14T14:20:52.327 INFO:journalctl@ceph.mon.c.smithi069.stdout: block_cache_name: BinnedLRUCache 2024-04-14T14:20:52.327 INFO:journalctl@ceph.mon.c.smithi069.stdout: block_cache_options: 2024-04-14T14:20:52.327 INFO:journalctl@ceph.mon.c.smithi069.stdout: capacity : 536870912 2024-04-14T14:20:52.327 INFO:journalctl@ceph.mon.c.smithi069.stdout: num_shard_bits : 4 2024-04-14T14:20:52.327 INFO:journalctl@ceph.mon.c.smithi069.stdout: strict_capacity_limit : 0 2024-04-14T14:20:52.328 INFO:journalctl@ceph.mon.c.smithi069.stdout: high_pri_pool_ratio: 0.000 2024-04-14T14:20:52.328 INFO:journalctl@ceph.mon.c.smithi069.stdout: block_cache_compressed: (nil) 2024-04-14T14:20:52.328 INFO:journalctl@ceph.mon.c.smithi069.stdout: persistent_cache: (nil) 2024-04-14T14:20:52.328 INFO:journalctl@ceph.mon.c.smithi069.stdout: block_size: 4096 2024-04-14T14:20:52.328 INFO:journalctl@ceph.mon.c.smithi069.stdout: block_size_deviation: 10 2024-04-14T14:20:52.328 INFO:journalctl@ceph.mon.c.smithi069.stdout: block_restart_interval: 16 2024-04-14T14:20:52.328 INFO:journalctl@ceph.mon.c.smithi069.stdout: index_block_restart_interval: 1 2024-04-14T14:20:52.328 INFO:journalctl@ceph.mon.c.smithi069.stdout: metadata_block_size: 4096 2024-04-14T14:20:52.328 INFO:journalctl@ceph.mon.c.smithi069.stdout: partition_filters: 0 2024-04-14T14:20:52.328 INFO:journalctl@ceph.mon.c.smithi069.stdout: use_delta_encoding: 1 2024-04-14T14:20:52.328 INFO:journalctl@ceph.mon.c.smithi069.stdout: filter_policy: bloomfilter 2024-04-14T14:20:52.328 INFO:journalctl@ceph.mon.c.smithi069.stdout: whole_key_filtering: 1 2024-04-14T14:20:52.328 INFO:journalctl@ceph.mon.c.smithi069.stdout: verify_compression: 0 2024-04-14T14:20:52.328 INFO:journalctl@ceph.mon.c.smithi069.stdout: read_amp_bytes_per_bit: 0 2024-04-14T14:20:52.328 INFO:journalctl@ceph.mon.c.smithi069.stdout: format_version: 5 2024-04-14T14:20:52.329 INFO:journalctl@ceph.mon.c.smithi069.stdout: enable_index_compression: 1 2024-04-14T14:20:52.329 INFO:journalctl@ceph.mon.c.smithi069.stdout: block_align: 0 2024-04-14T14:20:52.329 INFO:journalctl@ceph.mon.c.smithi069.stdout: max_auto_readahead_size: 262144 2024-04-14T14:20:52.329 INFO:journalctl@ceph.mon.c.smithi069.stdout: prepopulate_block_cache: 0 2024-04-14T14:20:52.329 INFO:journalctl@ceph.mon.c.smithi069.stdout: initial_auto_readahead_size: 8192 2024-04-14T14:20:52.329 INFO:journalctl@ceph.mon.c.smithi069.stdout: num_file_reads_for_auto_readahead: 2 2024-04-14T14:20:52.329 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:20:52 smithi069 ceph-mon[88636]: rocksdb: Options.write_buffer_size: 33554432 2024-04-14T14:20:52.329 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:20:52 smithi069 ceph-mon[88636]: rocksdb: Options.max_write_buffer_number: 2 2024-04-14T14:20:52.329 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:20:52 smithi069 ceph-mon[88636]: rocksdb: Options.compression: NoCompression 2024-04-14T14:20:52.329 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:20:52 smithi069 ceph-mon[88636]: rocksdb: Options.bottommost_compression: Disabled 2024-04-14T14:20:52.329 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:20:52 smithi069 ceph-mon[88636]: rocksdb: Options.prefix_extractor: nullptr 2024-04-14T14:20:52.329 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:20:52 smithi069 ceph-mon[88636]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-04-14T14:20:52.329 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:20:52 smithi069 ceph-mon[88636]: rocksdb: Options.num_levels: 7 2024-04-14T14:20:52.329 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:20:52 smithi069 ceph-mon[88636]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-04-14T14:20:52.330 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:20:52 smithi069 ceph-mon[88636]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-04-14T14:20:52.330 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:20:52 smithi069 ceph-mon[88636]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-04-14T14:20:52.330 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:20:52 smithi069 ceph-mon[88636]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-04-14T14:20:52.330 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:20:52 smithi069 ceph-mon[88636]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-04-14T14:20:52.330 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:20:52 smithi069 ceph-mon[88636]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-04-14T14:20:52.330 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:20:52 smithi069 ceph-mon[88636]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-04-14T14:20:52.330 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:20:52 smithi069 ceph-mon[88636]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-04-14T14:20:52.330 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:20:52 smithi069 ceph-mon[88636]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-04-14T14:20:52.330 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:20:52 smithi069 ceph-mon[88636]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-04-14T14:20:52.330 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:20:52 smithi069 ceph-mon[88636]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-04-14T14:20:52.330 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:20:52 smithi069 ceph-mon[88636]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-04-14T14:20:52.330 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:20:52 smithi069 ceph-mon[88636]: rocksdb: Options.compression_opts.window_bits: -14 2024-04-14T14:20:52.330 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:20:52 smithi069 ceph-mon[88636]: rocksdb: Options.compression_opts.level: 32767 2024-04-14T14:20:52.330 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:20:52 smithi069 ceph-mon[88636]: rocksdb: Options.compression_opts.strategy: 0 2024-04-14T14:20:52.330 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:20:52 smithi069 ceph-mon[88636]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-04-14T14:20:52.331 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:20:52 smithi069 ceph-mon[88636]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-04-14T14:20:52.331 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:20:52 smithi069 ceph-mon[88636]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-04-14T14:20:52.331 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:20:52 smithi069 ceph-mon[88636]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-04-14T14:20:52.331 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:20:52 smithi069 ceph-mon[88636]: rocksdb: Options.compression_opts.enabled: false 2024-04-14T14:20:52.331 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:20:52 smithi069 ceph-mon[88636]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-04-14T14:20:52.331 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:20:52 smithi069 ceph-mon[88636]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-04-14T14:20:52.331 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:20:52 smithi069 ceph-mon[88636]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-04-14T14:20:52.331 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:20:52 smithi069 ceph-mon[88636]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-04-14T14:20:52.331 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:20:52 smithi069 ceph-mon[88636]: rocksdb: Options.target_file_size_base: 67108864 2024-04-14T14:20:52.331 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:20:52 smithi069 ceph-mon[88636]: rocksdb: Options.target_file_size_multiplier: 1 2024-04-14T14:20:52.331 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:20:52 smithi069 ceph-mon[88636]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-04-14T14:20:52.331 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:20:52 smithi069 ceph-mon[88636]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-04-14T14:20:52.331 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:20:52 smithi069 ceph-mon[88636]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-04-14T14:20:52.331 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:20:52 smithi069 ceph-mon[88636]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-04-14T14:20:52.331 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:20:52 smithi069 ceph-mon[88636]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-04-14T14:20:52.332 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:20:52 smithi069 ceph-mon[88636]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-04-14T14:20:52.332 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:20:52 smithi069 ceph-mon[88636]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-04-14T14:20:52.332 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:20:52 smithi069 ceph-mon[88636]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-04-14T14:20:52.332 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:20:52 smithi069 ceph-mon[88636]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-04-14T14:20:52.332 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:20:52 smithi069 ceph-mon[88636]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-04-14T14:20:52.332 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:20:52 smithi069 ceph-mon[88636]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-04-14T14:20:52.332 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:20:52 smithi069 ceph-mon[88636]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-04-14T14:20:52.332 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:20:52 smithi069 ceph-mon[88636]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-04-14T14:20:52.332 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:20:52 smithi069 ceph-mon[88636]: rocksdb: Options.arena_block_size: 1048576 2024-04-14T14:20:52.332 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:20:52 smithi069 ceph-mon[88636]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-04-14T14:20:52.332 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:20:52 smithi069 ceph-mon[88636]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-04-14T14:20:52.332 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:20:52 smithi069 ceph-mon[88636]: rocksdb: Options.disable_auto_compactions: 0 2024-04-14T14:20:52.332 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:20:52 smithi069 ceph-mon[88636]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-04-14T14:20:52.332 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:20:52 smithi069 ceph-mon[88636]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-04-14T14:20:52.332 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:20:52 smithi069 ceph-mon[88636]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-04-14T14:20:52.333 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:20:52 smithi069 ceph-mon[88636]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-04-14T14:20:52.333 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:20:52 smithi069 ceph-mon[88636]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-04-14T14:20:52.333 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:20:52 smithi069 ceph-mon[88636]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-04-14T14:20:52.333 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:20:52 smithi069 ceph-mon[88636]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-04-14T14:20:52.333 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:20:52 smithi069 ceph-mon[88636]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-04-14T14:20:52.333 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:20:52 smithi069 ceph-mon[88636]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-04-14T14:20:52.333 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:20:52 smithi069 ceph-mon[88636]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-04-14T14:20:52.333 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:20:52 smithi069 ceph-mon[88636]: rocksdb: Options.table_properties_collectors: CompactOnDeletionCollector (Sliding window size = 32768 Deletion trigger = 16384 Deletion ratio = 0); 2024-04-14T14:20:52.333 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:20:52 smithi069 ceph-mon[88636]: rocksdb: Options.inplace_update_support: 0 2024-04-14T14:20:52.333 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:20:52 smithi069 ceph-mon[88636]: rocksdb: Options.inplace_update_num_locks: 10000 2024-04-14T14:20:52.333 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:20:52 smithi069 ceph-mon[88636]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-04-14T14:20:52.333 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:20:52 smithi069 ceph-mon[88636]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-04-14T14:20:52.333 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:20:52 smithi069 ceph-mon[88636]: rocksdb: Options.memtable_huge_page_size: 0 2024-04-14T14:20:52.333 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:20:52 smithi069 ceph-mon[88636]: rocksdb: Options.bloom_locality: 0 2024-04-14T14:20:52.333 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:20:52 smithi069 ceph-mon[88636]: rocksdb: Options.max_successive_merges: 0 2024-04-14T14:20:52.333 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:20:52 smithi069 ceph-mon[88636]: rocksdb: Options.optimize_filters_for_hits: 0 2024-04-14T14:20:52.334 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:20:52 smithi069 ceph-mon[88636]: rocksdb: Options.paranoid_file_checks: 0 2024-04-14T14:20:52.334 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:20:52 smithi069 ceph-mon[88636]: rocksdb: Options.force_consistency_checks: 1 2024-04-14T14:20:52.334 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:20:52 smithi069 ceph-mon[88636]: rocksdb: Options.report_bg_io_stats: 0 2024-04-14T14:20:52.334 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:20:52 smithi069 ceph-mon[88636]: rocksdb: Options.ttl: 2592000 2024-04-14T14:20:52.334 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:20:52 smithi069 ceph-mon[88636]: rocksdb: Options.periodic_compaction_seconds: 0 2024-04-14T14:20:52.334 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:20:52 smithi069 ceph-mon[88636]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-04-14T14:20:52.334 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:20:52 smithi069 ceph-mon[88636]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-04-14T14:20:52.334 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:20:52 smithi069 ceph-mon[88636]: rocksdb: Options.enable_blob_files: false 2024-04-14T14:20:52.334 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:20:52 smithi069 ceph-mon[88636]: rocksdb: Options.min_blob_size: 0 2024-04-14T14:20:52.334 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:20:52 smithi069 ceph-mon[88636]: rocksdb: Options.blob_file_size: 268435456 2024-04-14T14:20:52.334 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:20:52 smithi069 ceph-mon[88636]: rocksdb: Options.blob_compression_type: NoCompression 2024-04-14T14:20:52.334 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:20:52 smithi069 ceph-mon[88636]: rocksdb: Options.enable_blob_garbage_collection: false 2024-04-14T14:20:52.334 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:20:52 smithi069 ceph-mon[88636]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-04-14T14:20:52.335 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:20:52 smithi069 ceph-mon[88636]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-04-14T14:20:52.335 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:20:52 smithi069 ceph-mon[88636]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-04-14T14:20:52.335 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:20:52 smithi069 ceph-mon[88636]: rocksdb: Options.blob_file_starting_level: 0 2024-04-14T14:20:52.335 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:20:52 smithi069 ceph-mon[88636]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-04-14T14:20:52.335 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:20:52 smithi069 ceph-mon[88636]: rocksdb: [table/block_based/block_based_table_reader.cc:721] At least one SST file opened without unique ID to verify: 60.sst 2024-04-14T14:20:52.335 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:20:52 smithi069 ceph-mon[88636]: rocksdb: [db/version_set.cc:4390] More existing levels in DB than needed. max_bytes_for_level_multiplier may not be guaranteed. 2024-04-14T14:20:52.335 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:20:52 smithi069 ceph-mon[88636]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-c/store.db/MANIFEST-000009 succeeded,manifest_file_number is 9, next_file_number is 62, last_sequence is 32029, log_number is 58,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2024-04-14T14:20:52.335 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:20:52 smithi069 ceph-mon[88636]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 58 2024-04-14T14:20:52.335 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:20:52 smithi069 ceph-mon[88636]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: 35fb4195-ea7f-4315-9f8d-194453e662ba 2024-04-14T14:20:52.335 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:20:52 smithi069 ceph-mon[88636]: rocksdb: EVENT_LOG_v1 {"time_micros": 1713104452119521, "job": 1, "event": "recovery_started", "wal_files": [58]} 2024-04-14T14:20:52.335 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:20:52 smithi069 ceph-mon[88636]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #58 mode 2 2024-04-14T14:20:52.335 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:20:52 smithi069 ceph-mon[88636]: rocksdb: EVENT_LOG_v1 {"time_micros": 1713104452121719, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 63, "file_size": 393927, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 32026, "largest_seqno": 32252, "table_properties": {"data_size": 391747, "index_size": 822, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 325, "raw_key_size": 2588, "raw_average_key_size": 24, "raw_value_size": 389199, "raw_average_value_size": 3706, "num_data_blocks": 36, "num_entries": 105, "num_filter_entries": 105, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[CompactOnDeletionCollector]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1713104452, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "35fb4195-ea7f-4315-9f8d-194453e662ba", "db_session_id": "DLW8FS0C47RWA17DF47T", "orig_file_number": 63, "seqno_to_time_mapping": "N/A"}} 2024-04-14T14:20:52.335 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:20:52 smithi069 ceph-mon[88636]: rocksdb: EVENT_LOG_v1 {"time_micros": 1713104452121803, "job": 1, "event": "recovery_finished"} 2024-04-14T14:20:52.335 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:20:52 smithi069 ceph-mon[88636]: rocksdb: [db/version_set.cc:5047] Creating manifest 65 2024-04-14T14:20:52.335 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:20:52 smithi069 ceph-mon[88636]: rocksdb: [db/version_set.cc:4390] More existing levels in DB than needed. max_bytes_for_level_multiplier may not be guaranteed. 2024-04-14T14:20:52.336 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:20:52 smithi069 ceph-mon[88636]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-c/store.db/000058.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-04-14T14:20:52.336 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:20:52 smithi069 ceph-mon[88636]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x560649860000 2024-04-14T14:20:52.336 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:20:52 smithi069 ceph-mon[88636]: rocksdb: DB pointer 0x56064984a000 2024-04-14T14:20:52.336 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:20:52 smithi069 ceph-mon[88636]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-04-14T14:20:52.336 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:20:52 smithi069 ceph-mon[88636]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-04-14T14:20:52.336 INFO:journalctl@ceph.mon.c.smithi069.stdout: ** DB Stats ** 2024-04-14T14:20:52.336 INFO:journalctl@ceph.mon.c.smithi069.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-04-14T14:20:52.336 INFO:journalctl@ceph.mon.c.smithi069.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-04-14T14:20:52.336 INFO:journalctl@ceph.mon.c.smithi069.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-04-14T14:20:52.336 INFO:journalctl@ceph.mon.c.smithi069.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-04-14T14:20:52.336 INFO:journalctl@ceph.mon.c.smithi069.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-04-14T14:20:52.336 INFO:journalctl@ceph.mon.c.smithi069.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-04-14T14:20:52.336 INFO:journalctl@ceph.mon.c.smithi069.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-04-14T14:20:52.336 INFO:journalctl@ceph.mon.c.smithi069.stdout: 2024-04-14T14:20:52.337 INFO:journalctl@ceph.mon.c.smithi069.stdout: ** Compaction Stats [default] ** 2024-04-14T14:20:52.337 INFO:journalctl@ceph.mon.c.smithi069.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-04-14T14:20:52.337 INFO:journalctl@ceph.mon.c.smithi069.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-04-14T14:20:52.337 INFO:journalctl@ceph.mon.c.smithi069.stdout: L0 1/0 384.69 KB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 294.4 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-04-14T14:20:52.337 INFO:journalctl@ceph.mon.c.smithi069.stdout: L6 1/0 10.97 MB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 0.0 0.0 2024-04-14T14:20:52.337 INFO:journalctl@ceph.mon.c.smithi069.stdout: Sum 2/0 11.35 MB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 294.4 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-04-14T14:20:52.337 INFO:journalctl@ceph.mon.c.smithi069.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 294.4 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-04-14T14:20:52.337 INFO:journalctl@ceph.mon.c.smithi069.stdout: 2024-04-14T14:20:52.337 INFO:journalctl@ceph.mon.c.smithi069.stdout: ** Compaction Stats [default] ** 2024-04-14T14:20:52.337 INFO:journalctl@ceph.mon.c.smithi069.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-04-14T14:20:52.337 INFO:journalctl@ceph.mon.c.smithi069.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-04-14T14:20:52.337 INFO:journalctl@ceph.mon.c.smithi069.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 294.4 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-04-14T14:20:52.337 INFO:journalctl@ceph.mon.c.smithi069.stdout: 2024-04-14T14:20:52.337 INFO:journalctl@ceph.mon.c.smithi069.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-04-14T14:20:52.337 INFO:journalctl@ceph.mon.c.smithi069.stdout: 2024-04-14T14:20:52.338 INFO:journalctl@ceph.mon.c.smithi069.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-04-14T14:20:52.338 INFO:journalctl@ceph.mon.c.smithi069.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-04-14T14:20:52.338 INFO:journalctl@ceph.mon.c.smithi069.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-04-14T14:20:52.338 INFO:journalctl@ceph.mon.c.smithi069.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-04-14T14:20:52.338 INFO:journalctl@ceph.mon.c.smithi069.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-04-14T14:20:52.338 INFO:journalctl@ceph.mon.c.smithi069.stdout: AddFile(Keys): cumulative 0, interval 0 2024-04-14T14:20:52.338 INFO:journalctl@ceph.mon.c.smithi069.stdout: Cumulative compaction: 0.00 GB write, 87.63 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-04-14T14:20:52.338 INFO:journalctl@ceph.mon.c.smithi069.stdout: Interval compaction: 0.00 GB write, 87.63 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-04-14T14:20:52.338 INFO:journalctl@ceph.mon.c.smithi069.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-04-14T14:20:52.338 INFO:journalctl@ceph.mon.c.smithi069.stdout: Block cache BinnedLRUCache@0x560649771090#2 capacity: 512.00 MB usage: 1.33 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 9e-06 secs_since: 0 2024-04-14T14:20:52.338 INFO:journalctl@ceph.mon.c.smithi069.stdout: Block cache entry stats(count,size,portion): FilterBlock(1,0.38 KB,7.15256e-05%) IndexBlock(1,0.95 KB,0.000181794%) Misc(1,0.00 KB,0%) 2024-04-14T14:20:52.338 INFO:journalctl@ceph.mon.c.smithi069.stdout: 2024-04-14T14:20:52.338 INFO:journalctl@ceph.mon.c.smithi069.stdout: ** File Read Latency Histogram By Level [default] ** 2024-04-14T14:20:52.338 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:20:52 smithi069 ceph-mon[88636]: starting mon.c rank 1 at public addrs [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] at bind addrs [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon_data /var/lib/ceph/mon/ceph-c fsid dc326fc2-fa65-11ee-bc8e-c7b262605968 2024-04-14T14:20:52.339 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:20:52 smithi069 ceph-mon[88636]: mon.c@-1(???) e3 preinit fsid dc326fc2-fa65-11ee-bc8e-c7b262605968 2024-04-14T14:20:52.339 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:20:52 smithi069 ceph-mon[88636]: mon.c@-1(???).mds e1 new map 2024-04-14T14:20:52.339 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:20:52 smithi069 ceph-mon[88636]: mon.c@-1(???).mds e1 print_map 2024-04-14T14:20:52.339 INFO:journalctl@ceph.mon.c.smithi069.stdout: e1 2024-04-14T14:20:52.339 INFO:journalctl@ceph.mon.c.smithi069.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-04-14T14:20:52.339 INFO:journalctl@ceph.mon.c.smithi069.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-14T14:20:52.339 INFO:journalctl@ceph.mon.c.smithi069.stdout: legacy client fscid: -1 2024-04-14T14:20:52.339 INFO:journalctl@ceph.mon.c.smithi069.stdout: 2024-04-14T14:20:52.339 INFO:journalctl@ceph.mon.c.smithi069.stdout: No filesystems configured 2024-04-14T14:20:52.339 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:20:52 smithi069 ceph-mon[88636]: mon.c@-1(???).osd e111 crush map has features 3314933000854323200, adjusting msgr requires 2024-04-14T14:20:52.339 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:20:52 smithi069 ceph-mon[88636]: mon.c@-1(???).osd e111 crush map has features 432629239337189376, adjusting msgr requires 2024-04-14T14:20:52.339 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:20:52 smithi069 ceph-mon[88636]: mon.c@-1(???).osd e111 crush map has features 432629239337189376, adjusting msgr requires 2024-04-14T14:20:52.339 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:20:52 smithi069 ceph-mon[88636]: mon.c@-1(???).osd e111 crush map has features 432629239337189376, adjusting msgr requires 2024-04-14T14:20:52.340 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:20:52 smithi069 ceph-mon[88636]: mon.c@-1(???).paxosservice(auth 1..27) refresh upgraded, format 0 -> 3 2024-04-14T14:20:52.340 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:20:52 smithi069 bash[88617]: 84aa95f11d0c392c7b8479408817b8e8804c7d09554d5f4083aadc86bb18d9b7 2024-04-14T14:20:52.340 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:20:52 smithi069 systemd[1]: Started Ceph mon.c for dc326fc2-fa65-11ee-bc8e-c7b262605968. 2024-04-14T14:20:52.340 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:20:52 smithi069 ceph-mon[88636]: pgmap v10: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 10 op/s 2024-04-14T14:20:52.340 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:20:52 smithi069 ceph-mon[88636]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T14:20:52.596 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:20:52 smithi122 ceph-mon[67998]: pgmap v10: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 10 op/s 2024-04-14T14:20:52.596 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:20:52 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T14:20:53.450 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:20:53 smithi069 ceph-mon[88636]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-14T14:20:53.450 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:20:53 smithi069 ceph-mon[88636]: mon.a calling monitor election 2024-04-14T14:20:53.450 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:20:53 smithi069 ceph-mon[88636]: mon.b calling monitor election 2024-04-14T14:20:53.450 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:20:53 smithi069 ceph-mon[88636]: mon.c calling monitor election 2024-04-14T14:20:53.450 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:20:53 smithi069 ceph-mon[88636]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-14T14:20:53.450 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:20:53 smithi069 ceph-mon[88636]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-04-14T14:20:53.450 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:20:53 smithi069 ceph-mon[88636]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-14T14:20:53.450 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:20:53 smithi069 ceph-mon[88636]: monmap epoch 4 2024-04-14T14:20:53.451 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:20:53 smithi069 ceph-mon[88636]: fsid dc326fc2-fa65-11ee-bc8e-c7b262605968 2024-04-14T14:20:53.451 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:20:53 smithi069 ceph-mon[88636]: last_changed 2024-04-14T14:20:52.179013+0000 2024-04-14T14:20:53.451 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:20:53 smithi069 ceph-mon[88636]: created 2024-04-14T13:50:18.166086+0000 2024-04-14T14:20:53.451 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:20:53 smithi069 ceph-mon[88636]: min_mon_release 19 (squid) 2024-04-14T14:20:53.451 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:20:53 smithi069 ceph-mon[88636]: election_strategy: 1 2024-04-14T14:20:53.451 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:20:53 smithi069 ceph-mon[88636]: 0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2024-04-14T14:20:53.451 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:20:53 smithi069 ceph-mon[88636]: 1: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2024-04-14T14:20:53.451 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:20:53 smithi069 ceph-mon[88636]: 2: [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] mon.b 2024-04-14T14:20:53.451 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:20:53 smithi069 ceph-mon[88636]: fsmap 2024-04-14T14:20:53.451 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:20:53 smithi069 ceph-mon[88636]: osdmap e111: 8 total, 8 up, 8 in 2024-04-14T14:20:53.451 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:20:53 smithi069 ceph-mon[88636]: mgrmap e47: y(active, since 15s), standbys: x 2024-04-14T14:20:53.451 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:20:53 smithi069 ceph-mon[88636]: overall HEALTH_OK 2024-04-14T14:20:53.451 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:20:53 smithi069 ceph-mon[88636]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:20:53.451 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:20:53 smithi069 ceph-mon[88636]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:20:53.452 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:20:53 smithi069 ceph-mon[88636]: pgmap v11: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 10 op/s 2024-04-14T14:20:53.452 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:20:53 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-14T14:20:53.452 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:20:53 smithi069 ceph-mon[83680]: mon.a calling monitor election 2024-04-14T14:20:53.452 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:20:53 smithi069 ceph-mon[83680]: mon.b calling monitor election 2024-04-14T14:20:53.452 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:20:53 smithi069 ceph-mon[83680]: mon.c calling monitor election 2024-04-14T14:20:53.452 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:20:53 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-14T14:20:53.452 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:20:53 smithi069 ceph-mon[83680]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-04-14T14:20:53.452 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:20:53 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-14T14:20:53.452 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:20:53 smithi069 ceph-mon[83680]: monmap epoch 4 2024-04-14T14:20:53.452 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:20:53 smithi069 ceph-mon[83680]: fsid dc326fc2-fa65-11ee-bc8e-c7b262605968 2024-04-14T14:20:53.452 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:20:53 smithi069 ceph-mon[83680]: last_changed 2024-04-14T14:20:52.179013+0000 2024-04-14T14:20:53.452 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:20:53 smithi069 ceph-mon[83680]: created 2024-04-14T13:50:18.166086+0000 2024-04-14T14:20:53.453 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:20:53 smithi069 ceph-mon[83680]: min_mon_release 19 (squid) 2024-04-14T14:20:53.453 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:20:53 smithi069 ceph-mon[83680]: election_strategy: 1 2024-04-14T14:20:53.453 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:20:53 smithi069 ceph-mon[83680]: 0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2024-04-14T14:20:53.453 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:20:53 smithi069 ceph-mon[83680]: 1: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2024-04-14T14:20:53.453 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:20:53 smithi069 ceph-mon[83680]: 2: [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] mon.b 2024-04-14T14:20:53.453 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:20:53 smithi069 ceph-mon[83680]: fsmap 2024-04-14T14:20:53.453 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:20:53 smithi069 ceph-mon[83680]: osdmap e111: 8 total, 8 up, 8 in 2024-04-14T14:20:53.453 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:20:53 smithi069 ceph-mon[83680]: mgrmap e47: y(active, since 15s), standbys: x 2024-04-14T14:20:53.453 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:20:53 smithi069 ceph-mon[83680]: overall HEALTH_OK 2024-04-14T14:20:53.453 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:20:53 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:20:53.453 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:20:53 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:20:53.453 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:20:53 smithi069 ceph-mon[83680]: pgmap v11: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 10 op/s 2024-04-14T14:20:53.595 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:20:53 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-14T14:20:53.595 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:20:53 smithi122 ceph-mon[67998]: mon.a calling monitor election 2024-04-14T14:20:53.595 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:20:53 smithi122 ceph-mon[67998]: mon.b calling monitor election 2024-04-14T14:20:53.596 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:20:53 smithi122 ceph-mon[67998]: mon.c calling monitor election 2024-04-14T14:20:53.596 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:20:53 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-14T14:20:53.596 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:20:53 smithi122 ceph-mon[67998]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-04-14T14:20:53.596 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:20:53 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-14T14:20:53.596 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:20:53 smithi122 ceph-mon[67998]: monmap epoch 4 2024-04-14T14:20:53.596 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:20:53 smithi122 ceph-mon[67998]: fsid dc326fc2-fa65-11ee-bc8e-c7b262605968 2024-04-14T14:20:53.596 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:20:53 smithi122 ceph-mon[67998]: last_changed 2024-04-14T14:20:52.179013+0000 2024-04-14T14:20:53.596 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:20:53 smithi122 ceph-mon[67998]: created 2024-04-14T13:50:18.166086+0000 2024-04-14T14:20:53.596 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:20:53 smithi122 ceph-mon[67998]: min_mon_release 19 (squid) 2024-04-14T14:20:53.596 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:20:53 smithi122 ceph-mon[67998]: election_strategy: 1 2024-04-14T14:20:53.596 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:20:53 smithi122 ceph-mon[67998]: 0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2024-04-14T14:20:53.596 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:20:53 smithi122 ceph-mon[67998]: 1: [v2:172.21.15.69:3301/0,v1:172.21.15.69:6790/0] mon.c 2024-04-14T14:20:53.596 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:20:53 smithi122 ceph-mon[67998]: 2: [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] mon.b 2024-04-14T14:20:53.596 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:20:53 smithi122 ceph-mon[67998]: fsmap 2024-04-14T14:20:53.596 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:20:53 smithi122 ceph-mon[67998]: osdmap e111: 8 total, 8 up, 8 in 2024-04-14T14:20:53.597 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:20:53 smithi122 ceph-mon[67998]: mgrmap e47: y(active, since 15s), standbys: x 2024-04-14T14:20:53.597 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:20:53 smithi122 ceph-mon[67998]: overall HEALTH_OK 2024-04-14T14:20:53.597 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:20:53 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:20:53.597 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:20:53 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:20:53.597 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:20:53 smithi122 ceph-mon[67998]: pgmap v11: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 10 op/s 2024-04-14T14:20:55.417 INFO:teuthology.orchestra.run.smithi069.stdout:true 2024-04-14T14:20:55.726 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:20:55 smithi069 ceph-mon[83680]: pgmap v12: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 10 op/s 2024-04-14T14:20:55.726 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:20:55 smithi069 ceph-mon[88636]: pgmap v12: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 10 op/s 2024-04-14T14:20:55.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:20:55 smithi122 ceph-mon[67998]: pgmap v12: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 10 op/s 2024-04-14T14:20:56.091 INFO:teuthology.orchestra.run.smithi069.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-14T14:20:56.092 INFO:teuthology.orchestra.run.smithi069.stdout:alertmanager.a smithi069 *:9093,9094 running (2m) 14s ago 24m 19.0M - 0.25.0 c8568f914cd2 7524f106e7bd 2024-04-14T14:20:56.092 INFO:teuthology.orchestra.run.smithi069.stdout:grafana.a smithi122 *:3000 running (2m) 14s ago 24m 81.0M - 9.4.12 2bacad6d85d8 5685178e3eba 2024-04-14T14:20:56.092 INFO:teuthology.orchestra.run.smithi069.stdout:iscsi.foo.smithi069.yuumtr smithi069 running (3m) 14s ago 23m 50.1M - 3.5 e1d6a67b021e 38cc81ea8cfa 2024-04-14T14:20:56.092 INFO:teuthology.orchestra.run.smithi069.stdout:mgr.x smithi122 *:8443,9283,8765 running (3m) 14s ago 28m 450M - 19.0.0-2778-ga971fd5f b96c509eef84 e29977590b43 2024-04-14T14:20:56.092 INFO:teuthology.orchestra.run.smithi069.stdout:mgr.y smithi069 *:8443,9283,8765 running (13m) 14s ago 30m 475M - 19.0.0-2778-ga971fd5f b96c509eef84 7af3d29d0e5f 2024-04-14T14:20:56.092 INFO:teuthology.orchestra.run.smithi069.stdout:mon.a smithi069 running (28s) 14s ago 30m 33.2M 2048M 19.0.0-2778-ga971fd5f b96c509eef84 8bcd6b3d3846 2024-04-14T14:20:56.092 INFO:teuthology.orchestra.run.smithi069.stdout:mon.b smithi122 running (76s) 14s ago 29m 33.9M 2048M 19.0.0-2778-ga971fd5f b96c509eef84 05f437a1a9a1 2024-04-14T14:20:56.093 INFO:teuthology.orchestra.run.smithi069.stdout:mon.c smithi069 starting - - - 2048M 2024-04-14T14:20:56.093 INFO:teuthology.orchestra.run.smithi069.stdout:node-exporter.a smithi069 *:9100 running (3m) 14s ago 25m 8028k - 1.5.0 0da6a335fe13 0de0eb2171b9 2024-04-14T14:20:56.093 INFO:teuthology.orchestra.run.smithi069.stdout:node-exporter.b smithi122 *:9100 running (3m) 14s ago 25m 16.1M - 1.5.0 0da6a335fe13 67877bf02d8e 2024-04-14T14:20:56.093 INFO:teuthology.orchestra.run.smithi069.stdout:osd.0 smithi069 running (28m) 14s ago 28m 56.5M 2992M 17.2.0 e1d6a67b021e 46654654b6b6 2024-04-14T14:20:56.093 INFO:teuthology.orchestra.run.smithi069.stdout:osd.1 smithi069 running (28m) 14s ago 28m 55.1M 2992M 17.2.0 e1d6a67b021e cdba54a59609 2024-04-14T14:20:56.093 INFO:teuthology.orchestra.run.smithi069.stdout:osd.2 smithi069 running (27m) 14s ago 27m 51.0M 2992M 17.2.0 e1d6a67b021e bc37a1a9cba0 2024-04-14T14:20:56.093 INFO:teuthology.orchestra.run.smithi069.stdout:osd.3 smithi069 running (27m) 14s ago 27m 55.4M 2992M 17.2.0 e1d6a67b021e 0925ba3da801 2024-04-14T14:20:56.093 INFO:teuthology.orchestra.run.smithi069.stdout:osd.4 smithi122 running (27m) 14s ago 27m 56.1M 3248M 17.2.0 e1d6a67b021e c00975caf771 2024-04-14T14:20:56.093 INFO:teuthology.orchestra.run.smithi069.stdout:osd.5 smithi122 running (26m) 14s ago 26m 55.2M 3248M 17.2.0 e1d6a67b021e 5f52d217ceb8 2024-04-14T14:20:56.093 INFO:teuthology.orchestra.run.smithi069.stdout:osd.6 smithi122 running (26m) 14s ago 26m 52.4M 3248M 17.2.0 e1d6a67b021e a2acfec61475 2024-04-14T14:20:56.093 INFO:teuthology.orchestra.run.smithi069.stdout:osd.7 smithi122 running (25m) 14s ago 25m 57.1M 3248M 17.2.0 e1d6a67b021e 62411cd06801 2024-04-14T14:20:56.093 INFO:teuthology.orchestra.run.smithi069.stdout:prometheus.a smithi122 *:9095 running (2m) 14s ago 25m 66.4M - 2.43.0 a07b618ecd1d 5c1f41d3bbe6 2024-04-14T14:20:56.093 INFO:teuthology.orchestra.run.smithi069.stdout:rgw.foo.smithi069.wesedz smithi069 *:8000 running (23m) 14s ago 23m 78.5M - 17.2.0 e1d6a67b021e 16de491440f3 2024-04-14T14:20:56.094 INFO:teuthology.orchestra.run.smithi069.stdout:rgw.foo.smithi122.jiobyv smithi122 *:8000 running (23m) 14s ago 23m 75.8M - 17.2.0 e1d6a67b021e 6d16de445dba 2024-04-14T14:20:56.503 INFO:teuthology.orchestra.run.smithi069.stdout:{ 2024-04-14T14:20:56.503 INFO:teuthology.orchestra.run.smithi069.stdout: "mon": { 2024-04-14T14:20:56.503 INFO:teuthology.orchestra.run.smithi069.stdout: "ceph version 19.0.0-2778-ga971fd5f (a971fd5f5719038c70b817bafacd6d5400b1d888) squid (dev)": 3 2024-04-14T14:20:56.503 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-14T14:20:56.503 INFO:teuthology.orchestra.run.smithi069.stdout: "mgr": { 2024-04-14T14:20:56.503 INFO:teuthology.orchestra.run.smithi069.stdout: "ceph version 19.0.0-2778-ga971fd5f (a971fd5f5719038c70b817bafacd6d5400b1d888) squid (dev)": 2 2024-04-14T14:20:56.503 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-14T14:20:56.503 INFO:teuthology.orchestra.run.smithi069.stdout: "osd": { 2024-04-14T14:20:56.503 INFO:teuthology.orchestra.run.smithi069.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 8 2024-04-14T14:20:56.503 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-14T14:20:56.503 INFO:teuthology.orchestra.run.smithi069.stdout: "rgw": { 2024-04-14T14:20:56.504 INFO:teuthology.orchestra.run.smithi069.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-04-14T14:20:56.504 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-14T14:20:56.504 INFO:teuthology.orchestra.run.smithi069.stdout: "overall": { 2024-04-14T14:20:56.504 INFO:teuthology.orchestra.run.smithi069.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 10, 2024-04-14T14:20:56.504 INFO:teuthology.orchestra.run.smithi069.stdout: "ceph version 19.0.0-2778-ga971fd5f (a971fd5f5719038c70b817bafacd6d5400b1d888) squid (dev)": 5 2024-04-14T14:20:56.504 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-14T14:20:56.504 INFO:teuthology.orchestra.run.smithi069.stdout:} 2024-04-14T14:20:56.817 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 14 14:20:56 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-y[63038]: ::ffff:172.21.15.122 - - [14/Apr/2024:14:20:56] "GET /metrics HTTP/1.1" 200 38010 "" "Prometheus/2.43.0" 2024-04-14T14:20:56.817 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:20:56 smithi069 ceph-mon[83680]: from='client.44136 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T14:20:56.817 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:20:56 smithi069 ceph-mon[83680]: from='client.? 172.21.15.69:0/664101114' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T14:20:56.817 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:20:56 smithi069 ceph-mon[88636]: from='client.44136 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T14:20:56.818 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:20:56 smithi069 ceph-mon[88636]: from='client.? 172.21.15.69:0/664101114' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T14:20:56.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:20:56 smithi122 ceph-mon[67998]: from='client.44136 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T14:20:56.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:20:56 smithi122 ceph-mon[67998]: from='client.? 172.21.15.69:0/664101114' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T14:20:56.873 INFO:teuthology.orchestra.run.smithi069.stdout:{ 2024-04-14T14:20:56.873 INFO:teuthology.orchestra.run.smithi069.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888", 2024-04-14T14:20:56.873 INFO:teuthology.orchestra.run.smithi069.stdout: "in_progress": true, 2024-04-14T14:20:56.873 INFO:teuthology.orchestra.run.smithi069.stdout: "which": "Upgrading daemons of type(s) mon on host(s) smithi069", 2024-04-14T14:20:56.874 INFO:teuthology.orchestra.run.smithi069.stdout: "services_complete": [ 2024-04-14T14:20:56.874 INFO:teuthology.orchestra.run.smithi069.stdout: "mon" 2024-04-14T14:20:56.874 INFO:teuthology.orchestra.run.smithi069.stdout: ], 2024-04-14T14:20:56.874 INFO:teuthology.orchestra.run.smithi069.stdout: "progress": "2/2 daemons upgraded", 2024-04-14T14:20:56.874 INFO:teuthology.orchestra.run.smithi069.stdout: "message": "Currently upgrading mon daemons", 2024-04-14T14:20:56.874 INFO:teuthology.orchestra.run.smithi069.stdout: "is_paused": false 2024-04-14T14:20:56.874 INFO:teuthology.orchestra.run.smithi069.stdout:} 2024-04-14T14:20:57.526 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 14 14:20:57 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-y[63038]: 2024-04-14T14:20:57.129+0000 7fdd38467700 -1 mgr.server handle_report got status from non-daemon mon.c 2024-04-14T14:20:57.817 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:20:57 smithi069 ceph-mon[83680]: from='client.44142 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T14:20:57.817 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:20:57 smithi069 ceph-mon[83680]: from='client.44148 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T14:20:57.817 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:20:57 smithi069 ceph-mon[83680]: pgmap v13: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:20:57.817 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:20:57 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:20:57.817 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:20:57 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:20:57.817 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:20:57 smithi069 ceph-mon[88636]: from='client.44142 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T14:20:57.818 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:20:57 smithi069 ceph-mon[88636]: from='client.44148 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T14:20:57.818 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:20:57 smithi069 ceph-mon[88636]: pgmap v13: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:20:57.818 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:20:57 smithi069 ceph-mon[88636]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:20:57.818 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:20:57 smithi069 ceph-mon[88636]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:20:57.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:20:57 smithi122 ceph-mon[67998]: from='client.44142 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T14:20:57.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:20:57 smithi122 ceph-mon[67998]: from='client.44148 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T14:20:57.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:20:57 smithi122 ceph-mon[67998]: pgmap v13: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:20:57.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:20:57 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:20:57.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:20:57 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:20:58.760 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:20:58 smithi069 ceph-mon[83680]: from='client.44160 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T14:20:58.760 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:20:58 smithi069 ceph-mon[83680]: from='client.25417 -' entity='client.iscsi.foo.smithi069.yuumtr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T14:20:58.760 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:20:58 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:20:58.760 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:20:58 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:20:58.761 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:20:58 smithi069 ceph-mon[88636]: from='client.44160 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T14:20:58.761 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:20:58 smithi069 ceph-mon[88636]: from='client.25417 -' entity='client.iscsi.foo.smithi069.yuumtr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T14:20:58.761 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:20:58 smithi069 ceph-mon[88636]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:20:58.761 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:20:58 smithi069 ceph-mon[88636]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:20:58.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:20:58 smithi122 ceph-mon[67998]: from='client.44160 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T14:20:58.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:20:58 smithi122 ceph-mon[67998]: from='client.25417 -' entity='client.iscsi.foo.smithi069.yuumtr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T14:20:58.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:20:58 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:20:58.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:20:58 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:20:59.693 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:20:59 smithi069 ceph-mon[88636]: pgmap v14: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:20:59.694 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:20:59 smithi069 ceph-mon[83680]: pgmap v14: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:20:59.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:20:59 smithi122 ceph-mon[67998]: pgmap v14: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:21:01.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:21:01 smithi122 ceph-mon[67998]: pgmap v15: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:21:01.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:21:01 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:21:01.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:21:01 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:21:01.846 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:21:01 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T14:21:01.846 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:21:01 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-14T14:21:01.846 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:21:01 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:21:01.846 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:21:01 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-14T14:21:01.846 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:21:01 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-14T14:21:01.846 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:21:01 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T14:21:01.936 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:21:01 smithi069 ceph-mon[83680]: pgmap v15: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:21:01.937 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:21:01 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:21:01.937 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:21:01 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:21:01.937 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:21:01 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T14:21:01.937 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:21:01 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-14T14:21:01.937 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:21:01 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:21:01.937 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:21:01 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-14T14:21:01.937 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:21:01 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-14T14:21:01.937 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:21:01 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T14:21:01.937 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:21:01 smithi069 ceph-mon[88636]: pgmap v15: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:21:01.937 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:21:01 smithi069 ceph-mon[88636]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:21:01.938 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:21:01 smithi069 ceph-mon[88636]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:21:01.938 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:21:01 smithi069 ceph-mon[88636]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T14:21:01.938 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:21:01 smithi069 ceph-mon[88636]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-14T14:21:01.938 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:21:01 smithi069 ceph-mon[88636]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:21:01.938 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:21:01 smithi069 ceph-mon[88636]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-14T14:21:01.938 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:21:01 smithi069 ceph-mon[88636]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-14T14:21:01.938 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:21:01 smithi069 ceph-mon[88636]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T14:21:02.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:21:02 smithi122 ceph-mon[67998]: Reconfiguring mon.a (monmap changed)... 2024-04-14T14:21:02.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:21:02 smithi122 ceph-mon[67998]: Reconfiguring daemon mon.a on smithi069 2024-04-14T14:21:03.067 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:21:02 smithi069 ceph-mon[83680]: Reconfiguring mon.a (monmap changed)... 2024-04-14T14:21:03.067 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:21:02 smithi069 ceph-mon[83680]: Reconfiguring daemon mon.a on smithi069 2024-04-14T14:21:03.067 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:21:02 smithi069 ceph-mon[88636]: Reconfiguring mon.a (monmap changed)... 2024-04-14T14:21:03.067 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:21:02 smithi069 ceph-mon[88636]: Reconfiguring daemon mon.a on smithi069 2024-04-14T14:21:03.689 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:21:03 smithi069 ceph-mon[83680]: pgmap v16: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:21:03.689 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:21:03 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:21:03.689 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:21:03 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:21:03.689 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:21:03 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-14T14:21:03.689 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:21:03 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-14T14:21:03.689 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:21:03 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T14:21:03.689 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:21:03 smithi069 ceph-mon[88636]: pgmap v16: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:21:03.689 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:21:03 smithi069 ceph-mon[88636]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:21:03.690 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:21:03 smithi069 ceph-mon[88636]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:21:03.690 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:21:03 smithi069 ceph-mon[88636]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-14T14:21:03.690 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:21:03 smithi069 ceph-mon[88636]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-14T14:21:03.690 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:21:03 smithi069 ceph-mon[88636]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T14:21:03.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:21:03 smithi122 ceph-mon[67998]: pgmap v16: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:21:03.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:21:03 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:21:03.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:21:03 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:21:03.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:21:03 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-14T14:21:03.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:21:03 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-14T14:21:03.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:21:03 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T14:21:04.817 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:21:04 smithi069 ceph-mon[88636]: Reconfiguring mgr.y (monmap changed)... 2024-04-14T14:21:04.817 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:21:04 smithi069 ceph-mon[88636]: Reconfiguring daemon mgr.y on smithi069 2024-04-14T14:21:04.817 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:21:04 smithi069 ceph-mon[88636]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:21:04.817 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:21:04 smithi069 ceph-mon[88636]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:21:04.818 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:21:04 smithi069 ceph-mon[83680]: Reconfiguring mgr.y (monmap changed)... 2024-04-14T14:21:04.818 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:21:04 smithi069 ceph-mon[83680]: Reconfiguring daemon mgr.y on smithi069 2024-04-14T14:21:04.818 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:21:04 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:21:04.818 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:21:04 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:21:04.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:21:04 smithi122 ceph-mon[67998]: Reconfiguring mgr.y (monmap changed)... 2024-04-14T14:21:04.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:21:04 smithi122 ceph-mon[67998]: Reconfiguring daemon mgr.y on smithi069 2024-04-14T14:21:04.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:21:04 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:21:04.846 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:21:04 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:21:05.817 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:21:05 smithi069 ceph-mon[83680]: pgmap v17: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:21:05.817 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:21:05 smithi069 ceph-mon[83680]: Reconfiguring mon.c (monmap changed)... 2024-04-14T14:21:05.817 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:21:05 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-14T14:21:05.817 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:21:05 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-14T14:21:05.817 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:21:05 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T14:21:05.817 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:21:05 smithi069 ceph-mon[83680]: Reconfiguring daemon mon.c on smithi069 2024-04-14T14:21:05.818 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:21:05 smithi069 ceph-mon[88636]: pgmap v17: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:21:05.818 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:21:05 smithi069 ceph-mon[88636]: Reconfiguring mon.c (monmap changed)... 2024-04-14T14:21:05.818 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:21:05 smithi069 ceph-mon[88636]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-14T14:21:05.818 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:21:05 smithi069 ceph-mon[88636]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-14T14:21:05.818 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:21:05 smithi069 ceph-mon[88636]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T14:21:05.818 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:21:05 smithi069 ceph-mon[88636]: Reconfiguring daemon mon.c on smithi069 2024-04-14T14:21:05.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:21:05 smithi122 ceph-mon[67998]: pgmap v17: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:21:05.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:21:05 smithi122 ceph-mon[67998]: Reconfiguring mon.c (monmap changed)... 2024-04-14T14:21:05.846 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:21:05 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-14T14:21:05.846 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:21:05 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-14T14:21:05.846 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:21:05 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T14:21:05.846 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:21:05 smithi122 ceph-mon[67998]: Reconfiguring daemon mon.c on smithi069 2024-04-14T14:21:06.800 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 14 14:21:06 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-y[63038]: ::ffff:172.21.15.122 - - [14/Apr/2024:14:21:06] "GET /metrics HTTP/1.1" 200 38010 "" "Prometheus/2.43.0" 2024-04-14T14:21:07.317 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:21:07 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:21:07.317 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:21:07 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:21:07.317 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:21:07 smithi069 ceph-mon[83680]: Reconfiguring osd.0 (monmap changed)... 2024-04-14T14:21:07.317 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:21:07 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-04-14T14:21:07.317 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:21:07 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T14:21:07.317 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:21:07 smithi069 ceph-mon[83680]: Reconfiguring daemon osd.0 on smithi069 2024-04-14T14:21:07.318 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:21:07 smithi069 ceph-mon[83680]: pgmap v18: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:21:07.318 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:21:07 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T14:21:07.318 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:21:07 smithi069 ceph-mon[88636]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:21:07.318 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:21:07 smithi069 ceph-mon[88636]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:21:07.318 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:21:07 smithi069 ceph-mon[88636]: Reconfiguring osd.0 (monmap changed)... 2024-04-14T14:21:07.318 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:21:07 smithi069 ceph-mon[88636]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-04-14T14:21:07.319 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:21:07 smithi069 ceph-mon[88636]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T14:21:07.319 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:21:07 smithi069 ceph-mon[88636]: Reconfiguring daemon osd.0 on smithi069 2024-04-14T14:21:07.319 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:21:07 smithi069 ceph-mon[88636]: pgmap v18: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:21:07.319 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:21:07 smithi069 ceph-mon[88636]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T14:21:07.345 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:21:07 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:21:07.346 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:21:07 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:21:07.346 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:21:07 smithi122 ceph-mon[67998]: Reconfiguring osd.0 (monmap changed)... 2024-04-14T14:21:07.346 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:21:07 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-04-14T14:21:07.346 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:21:07 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T14:21:07.346 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:21:07 smithi122 ceph-mon[67998]: Reconfiguring daemon osd.0 on smithi069 2024-04-14T14:21:07.347 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:21:07 smithi122 ceph-mon[67998]: pgmap v18: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:21:07.347 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:21:07 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T14:21:08.317 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:21:08 smithi069 ceph-mon[83680]: from='client.25417 -' entity='client.iscsi.foo.smithi069.yuumtr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T14:21:08.317 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:21:08 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:21:08.317 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:21:08 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:21:08.317 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:21:08 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-04-14T14:21:08.317 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:21:08 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T14:21:08.318 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:21:08 smithi069 ceph-mon[88636]: from='client.25417 -' entity='client.iscsi.foo.smithi069.yuumtr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T14:21:08.318 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:21:08 smithi069 ceph-mon[88636]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:21:08.318 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:21:08 smithi069 ceph-mon[88636]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:21:08.318 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:21:08 smithi069 ceph-mon[88636]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-04-14T14:21:08.318 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:21:08 smithi069 ceph-mon[88636]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T14:21:08.345 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:21:08 smithi122 ceph-mon[67998]: from='client.25417 -' entity='client.iscsi.foo.smithi069.yuumtr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T14:21:08.345 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:21:08 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:21:08.345 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:21:08 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:21:08.345 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:21:08 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-04-14T14:21:08.345 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:21:08 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T14:21:09.317 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:21:09 smithi069 ceph-mon[83680]: Reconfiguring osd.1 (monmap changed)... 2024-04-14T14:21:09.317 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:21:09 smithi069 ceph-mon[83680]: Reconfiguring daemon osd.1 on smithi069 2024-04-14T14:21:09.317 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:21:09 smithi069 ceph-mon[83680]: pgmap v19: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:21:09.317 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:21:09 smithi069 ceph-mon[88636]: Reconfiguring osd.1 (monmap changed)... 2024-04-14T14:21:09.317 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:21:09 smithi069 ceph-mon[88636]: Reconfiguring daemon osd.1 on smithi069 2024-04-14T14:21:09.317 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:21:09 smithi069 ceph-mon[88636]: pgmap v19: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:21:09.345 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:21:09 smithi122 ceph-mon[67998]: Reconfiguring osd.1 (monmap changed)... 2024-04-14T14:21:09.345 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:21:09 smithi122 ceph-mon[67998]: Reconfiguring daemon osd.1 on smithi069 2024-04-14T14:21:09.345 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:21:09 smithi122 ceph-mon[67998]: pgmap v19: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:21:10.567 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:21:10 smithi069 ceph-mon[88636]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:21:10.567 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:21:10 smithi069 ceph-mon[88636]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:21:10.567 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:21:10 smithi069 ceph-mon[88636]: Reconfiguring osd.2 (monmap changed)... 2024-04-14T14:21:10.567 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:21:10 smithi069 ceph-mon[88636]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-04-14T14:21:10.567 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:21:10 smithi069 ceph-mon[88636]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T14:21:10.567 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:21:10 smithi069 ceph-mon[88636]: Reconfiguring daemon osd.2 on smithi069 2024-04-14T14:21:10.568 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:21:10 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:21:10.568 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:21:10 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:21:10.568 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:21:10 smithi069 ceph-mon[83680]: Reconfiguring osd.2 (monmap changed)... 2024-04-14T14:21:10.568 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:21:10 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-04-14T14:21:10.568 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:21:10 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T14:21:10.568 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:21:10 smithi069 ceph-mon[83680]: Reconfiguring daemon osd.2 on smithi069 2024-04-14T14:21:10.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:21:10 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:21:10.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:21:10 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:21:10.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:21:10 smithi122 ceph-mon[67998]: Reconfiguring osd.2 (monmap changed)... 2024-04-14T14:21:10.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:21:10 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-04-14T14:21:10.846 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:21:10 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T14:21:10.846 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:21:10 smithi122 ceph-mon[67998]: Reconfiguring daemon osd.2 on smithi069 2024-04-14T14:21:11.698 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:21:11 smithi069 ceph-mon[83680]: pgmap v20: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:21:11.698 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:21:11 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:21:11.699 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:21:11 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:21:11.699 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:21:11 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-04-14T14:21:11.699 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:21:11 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T14:21:11.699 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:21:11 smithi069 ceph-mon[88636]: pgmap v20: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:21:11.699 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:21:11 smithi069 ceph-mon[88636]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:21:11.699 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:21:11 smithi069 ceph-mon[88636]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:21:11.699 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:21:11 smithi069 ceph-mon[88636]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-04-14T14:21:11.699 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:21:11 smithi069 ceph-mon[88636]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T14:21:11.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:21:11 smithi122 ceph-mon[67998]: pgmap v20: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:21:11.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:21:11 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:21:11.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:21:11 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:21:11.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:21:11 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-04-14T14:21:11.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:21:11 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T14:21:12.817 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:21:12 smithi069 ceph-mon[83680]: Reconfiguring osd.3 (monmap changed)... 2024-04-14T14:21:12.817 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:21:12 smithi069 ceph-mon[83680]: Reconfiguring daemon osd.3 on smithi069 2024-04-14T14:21:12.817 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:21:12 smithi069 ceph-mon[88636]: Reconfiguring osd.3 (monmap changed)... 2024-04-14T14:21:12.817 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:21:12 smithi069 ceph-mon[88636]: Reconfiguring daemon osd.3 on smithi069 2024-04-14T14:21:12.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:21:12 smithi122 ceph-mon[67998]: Reconfiguring osd.3 (monmap changed)... 2024-04-14T14:21:12.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:21:12 smithi122 ceph-mon[67998]: Reconfiguring daemon osd.3 on smithi069 2024-04-14T14:21:13.510 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:21:13 smithi069 ceph-mon[83680]: pgmap v21: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:21:13.511 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:21:13 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:21:13.511 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:21:13 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:21:13.511 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:21:13 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi069.wesedz", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-04-14T14:21:13.511 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:21:13 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T14:21:13.511 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:21:13 smithi069 ceph-mon[88636]: pgmap v21: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:21:13.511 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:21:13 smithi069 ceph-mon[88636]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:21:13.511 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:21:13 smithi069 ceph-mon[88636]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:21:13.511 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:21:13 smithi069 ceph-mon[88636]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi069.wesedz", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-04-14T14:21:13.511 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:21:13 smithi069 ceph-mon[88636]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T14:21:13.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:21:13 smithi122 ceph-mon[67998]: pgmap v21: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:21:13.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:21:13 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:21:13.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:21:13 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:21:13.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:21:13 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi069.wesedz", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-04-14T14:21:13.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:21:13 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T14:21:14.817 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:21:14 smithi069 ceph-mon[83680]: Reconfiguring rgw.foo.smithi069.wesedz (monmap changed)... 2024-04-14T14:21:14.817 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:21:14 smithi069 ceph-mon[83680]: Reconfiguring daemon rgw.foo.smithi069.wesedz on smithi069 2024-04-14T14:21:14.817 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:21:14 smithi069 ceph-mon[88636]: Reconfiguring rgw.foo.smithi069.wesedz (monmap changed)... 2024-04-14T14:21:14.817 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:21:14 smithi069 ceph-mon[88636]: Reconfiguring daemon rgw.foo.smithi069.wesedz on smithi069 2024-04-14T14:21:14.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:21:14 smithi122 ceph-mon[67998]: Reconfiguring rgw.foo.smithi069.wesedz (monmap changed)... 2024-04-14T14:21:14.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:21:14 smithi122 ceph-mon[67998]: Reconfiguring daemon rgw.foo.smithi069.wesedz on smithi069 2024-04-14T14:21:15.570 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:21:15 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:21:15.571 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:21:15 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:21:15.571 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:21:15 smithi122 ceph-mon[67998]: Reconfiguring mon.b (monmap changed)... 2024-04-14T14:21:15.571 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:21:15 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-14T14:21:15.571 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:21:15 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-14T14:21:15.571 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:21:15 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T14:21:15.571 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:21:15 smithi122 ceph-mon[67998]: Reconfiguring daemon mon.b on smithi122 2024-04-14T14:21:15.571 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:21:15 smithi122 ceph-mon[67998]: pgmap v22: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:21:15.817 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:21:15 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:21:15.818 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:21:15 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:21:15.818 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:21:15 smithi069 ceph-mon[83680]: Reconfiguring mon.b (monmap changed)... 2024-04-14T14:21:15.818 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:21:15 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-14T14:21:15.818 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:21:15 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-14T14:21:15.818 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:21:15 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T14:21:15.818 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:21:15 smithi069 ceph-mon[83680]: Reconfiguring daemon mon.b on smithi122 2024-04-14T14:21:15.818 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:21:15 smithi069 ceph-mon[83680]: pgmap v22: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:21:15.818 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:21:15 smithi069 ceph-mon[88636]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:21:15.818 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:21:15 smithi069 ceph-mon[88636]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:21:15.819 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:21:15 smithi069 ceph-mon[88636]: Reconfiguring mon.b (monmap changed)... 2024-04-14T14:21:15.819 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:21:15 smithi069 ceph-mon[88636]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-14T14:21:15.819 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:21:15 smithi069 ceph-mon[88636]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-14T14:21:15.819 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:21:15 smithi069 ceph-mon[88636]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T14:21:15.819 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:21:15 smithi069 ceph-mon[88636]: Reconfiguring daemon mon.b on smithi122 2024-04-14T14:21:15.819 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:21:15 smithi069 ceph-mon[88636]: pgmap v22: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:21:16.817 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 14 14:21:16 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-y[63038]: ::ffff:172.21.15.122 - - [14/Apr/2024:14:21:16] "GET /metrics HTTP/1.1" 200 38025 "" "Prometheus/2.43.0" 2024-04-14T14:21:17.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:21:16 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:21:17.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:21:16 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:21:17.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:21:16 smithi122 ceph-mon[67998]: Reconfiguring mgr.x (monmap changed)... 2024-04-14T14:21:17.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:21:16 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-14T14:21:17.096 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:21:16 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-14T14:21:17.096 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:21:16 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T14:21:17.096 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:21:16 smithi122 ceph-mon[67998]: Reconfiguring daemon mgr.x on smithi122 2024-04-14T14:21:17.096 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:21:16 smithi122 ceph-mon[67998]: pgmap v23: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:21:17.317 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:21:16 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:21:17.317 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:21:16 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:21:17.317 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:21:16 smithi069 ceph-mon[83680]: Reconfiguring mgr.x (monmap changed)... 2024-04-14T14:21:17.317 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:21:16 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-14T14:21:17.318 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:21:16 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-14T14:21:17.318 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:21:16 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T14:21:17.318 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:21:16 smithi069 ceph-mon[83680]: Reconfiguring daemon mgr.x on smithi122 2024-04-14T14:21:17.318 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:21:16 smithi069 ceph-mon[83680]: pgmap v23: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:21:17.318 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:21:16 smithi069 ceph-mon[88636]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:21:17.318 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:21:16 smithi069 ceph-mon[88636]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:21:17.318 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:21:16 smithi069 ceph-mon[88636]: Reconfiguring mgr.x (monmap changed)... 2024-04-14T14:21:17.318 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:21:16 smithi069 ceph-mon[88636]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-14T14:21:17.318 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:21:16 smithi069 ceph-mon[88636]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-14T14:21:17.318 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:21:16 smithi069 ceph-mon[88636]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T14:21:17.318 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:21:16 smithi069 ceph-mon[88636]: Reconfiguring daemon mgr.x on smithi122 2024-04-14T14:21:17.319 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:21:16 smithi069 ceph-mon[88636]: pgmap v23: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:21:18.595 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:21:18 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:21:18.595 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:21:18 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:21:18.595 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:21:18 smithi122 ceph-mon[67998]: Reconfiguring osd.4 (monmap changed)... 2024-04-14T14:21:18.595 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:21:18 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-04-14T14:21:18.595 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:21:18 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T14:21:18.596 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:21:18 smithi122 ceph-mon[67998]: Reconfiguring daemon osd.4 on smithi122 2024-04-14T14:21:18.596 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:21:18 smithi122 ceph-mon[67998]: from='client.25417 -' entity='client.iscsi.foo.smithi069.yuumtr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T14:21:18.719 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:21:18 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:21:18.719 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:21:18 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:21:18.719 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:21:18 smithi069 ceph-mon[83680]: Reconfiguring osd.4 (monmap changed)... 2024-04-14T14:21:18.719 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:21:18 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-04-14T14:21:18.719 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:21:18 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T14:21:18.719 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:21:18 smithi069 ceph-mon[83680]: Reconfiguring daemon osd.4 on smithi122 2024-04-14T14:21:18.719 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:21:18 smithi069 ceph-mon[83680]: from='client.25417 -' entity='client.iscsi.foo.smithi069.yuumtr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T14:21:18.719 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:21:18 smithi069 ceph-mon[88636]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:21:18.719 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:21:18 smithi069 ceph-mon[88636]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:21:18.719 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:21:18 smithi069 ceph-mon[88636]: Reconfiguring osd.4 (monmap changed)... 2024-04-14T14:21:18.720 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:21:18 smithi069 ceph-mon[88636]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-04-14T14:21:18.720 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:21:18 smithi069 ceph-mon[88636]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T14:21:18.720 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:21:18 smithi069 ceph-mon[88636]: Reconfiguring daemon osd.4 on smithi122 2024-04-14T14:21:18.720 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:21:18 smithi069 ceph-mon[88636]: from='client.25417 -' entity='client.iscsi.foo.smithi069.yuumtr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T14:21:19.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:21:19 smithi122 ceph-mon[67998]: pgmap v24: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:21:19.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:21:19 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:21:19.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:21:19 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:21:19.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:21:19 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-04-14T14:21:19.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:21:19 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T14:21:19.730 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:21:19 smithi069 ceph-mon[83680]: pgmap v24: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:21:19.730 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:21:19 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:21:19.731 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:21:19 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:21:19.731 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:21:19 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-04-14T14:21:19.731 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:21:19 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T14:21:19.731 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:21:19 smithi069 ceph-mon[88636]: pgmap v24: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:21:19.731 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:21:19 smithi069 ceph-mon[88636]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:21:19.731 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:21:19 smithi069 ceph-mon[88636]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:21:19.731 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:21:19 smithi069 ceph-mon[88636]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-04-14T14:21:19.731 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:21:19 smithi069 ceph-mon[88636]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T14:21:20.817 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:21:20 smithi069 ceph-mon[83680]: Reconfiguring osd.5 (monmap changed)... 2024-04-14T14:21:20.817 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:21:20 smithi069 ceph-mon[83680]: Reconfiguring daemon osd.5 on smithi122 2024-04-14T14:21:20.818 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:21:20 smithi069 ceph-mon[88636]: Reconfiguring osd.5 (monmap changed)... 2024-04-14T14:21:20.818 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:21:20 smithi069 ceph-mon[88636]: Reconfiguring daemon osd.5 on smithi122 2024-04-14T14:21:20.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:21:20 smithi122 ceph-mon[67998]: Reconfiguring osd.5 (monmap changed)... 2024-04-14T14:21:20.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:21:20 smithi122 ceph-mon[67998]: Reconfiguring daemon osd.5 on smithi122 2024-04-14T14:21:21.540 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:21:21 smithi122 ceph-mon[67998]: pgmap v25: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:21:21.540 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:21:21 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:21:21.540 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:21:21 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:21:21.540 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:21:21 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-04-14T14:21:21.540 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:21:21 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T14:21:21.817 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:21:21 smithi069 ceph-mon[83680]: pgmap v25: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:21:21.817 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:21:21 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:21:21.817 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:21:21 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:21:21.817 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:21:21 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-04-14T14:21:21.817 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:21:21 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T14:21:21.817 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:21:21 smithi069 ceph-mon[88636]: pgmap v25: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:21:21.818 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:21:21 smithi069 ceph-mon[88636]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:21:21.818 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:21:21 smithi069 ceph-mon[88636]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:21:21.818 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:21:21 smithi069 ceph-mon[88636]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-04-14T14:21:21.818 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:21:21 smithi069 ceph-mon[88636]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T14:21:22.817 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:21:22 smithi069 ceph-mon[83680]: Reconfiguring osd.6 (monmap changed)... 2024-04-14T14:21:22.817 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:21:22 smithi069 ceph-mon[83680]: Reconfiguring daemon osd.6 on smithi122 2024-04-14T14:21:22.817 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:21:22 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T14:21:22.817 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:21:22 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:21:22.817 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:21:22 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:21:22.817 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:21:22 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-04-14T14:21:22.817 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:21:22 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T14:21:22.818 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:21:22 smithi069 ceph-mon[88636]: Reconfiguring osd.6 (monmap changed)... 2024-04-14T14:21:22.818 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:21:22 smithi069 ceph-mon[88636]: Reconfiguring daemon osd.6 on smithi122 2024-04-14T14:21:22.818 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:21:22 smithi069 ceph-mon[88636]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T14:21:22.818 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:21:22 smithi069 ceph-mon[88636]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:21:22.818 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:21:22 smithi069 ceph-mon[88636]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:21:22.818 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:21:22 smithi069 ceph-mon[88636]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-04-14T14:21:22.818 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:21:22 smithi069 ceph-mon[88636]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T14:21:22.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:21:22 smithi122 ceph-mon[67998]: Reconfiguring osd.6 (monmap changed)... 2024-04-14T14:21:22.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:21:22 smithi122 ceph-mon[67998]: Reconfiguring daemon osd.6 on smithi122 2024-04-14T14:21:22.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:21:22 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T14:21:22.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:21:22 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:21:22.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:21:22 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:21:22.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:21:22 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-04-14T14:21:22.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:21:22 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T14:21:23.817 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:21:23 smithi069 ceph-mon[83680]: Reconfiguring osd.7 (monmap changed)... 2024-04-14T14:21:23.817 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:21:23 smithi069 ceph-mon[83680]: Reconfiguring daemon osd.7 on smithi122 2024-04-14T14:21:23.817 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:21:23 smithi069 ceph-mon[83680]: pgmap v26: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:21:23.818 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:21:23 smithi069 ceph-mon[88636]: Reconfiguring osd.7 (monmap changed)... 2024-04-14T14:21:23.818 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:21:23 smithi069 ceph-mon[88636]: Reconfiguring daemon osd.7 on smithi122 2024-04-14T14:21:23.818 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:21:23 smithi069 ceph-mon[88636]: pgmap v26: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:21:23.844 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:21:23 smithi122 ceph-mon[67998]: Reconfiguring osd.7 (monmap changed)... 2024-04-14T14:21:23.844 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:21:23 smithi122 ceph-mon[67998]: Reconfiguring daemon osd.7 on smithi122 2024-04-14T14:21:23.844 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:21:23 smithi122 ceph-mon[67998]: pgmap v26: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:21:25.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:21:24 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:21:25.096 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:21:24 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:21:25.096 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:21:24 smithi122 ceph-mon[67998]: Reconfiguring rgw.foo.smithi122.jiobyv (monmap changed)... 2024-04-14T14:21:25.096 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:21:24 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi122.jiobyv", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-04-14T14:21:25.096 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:21:24 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T14:21:25.096 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:21:24 smithi122 ceph-mon[67998]: Reconfiguring daemon rgw.foo.smithi122.jiobyv on smithi122 2024-04-14T14:21:25.096 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:21:24 smithi122 ceph-mon[67998]: pgmap v27: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:21:25.317 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:21:24 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:21:25.317 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:21:24 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:21:25.317 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:21:24 smithi069 ceph-mon[83680]: Reconfiguring rgw.foo.smithi122.jiobyv (monmap changed)... 2024-04-14T14:21:25.317 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:21:24 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi122.jiobyv", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-04-14T14:21:25.317 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:21:24 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T14:21:25.317 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:21:24 smithi069 ceph-mon[83680]: Reconfiguring daemon rgw.foo.smithi122.jiobyv on smithi122 2024-04-14T14:21:25.318 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:21:24 smithi069 ceph-mon[83680]: pgmap v27: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:21:25.318 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:21:24 smithi069 ceph-mon[88636]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:21:25.318 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:21:24 smithi069 ceph-mon[88636]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:21:25.318 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:21:24 smithi069 ceph-mon[88636]: Reconfiguring rgw.foo.smithi122.jiobyv (monmap changed)... 2024-04-14T14:21:25.318 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:21:24 smithi069 ceph-mon[88636]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi122.jiobyv", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-04-14T14:21:25.318 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:21:24 smithi069 ceph-mon[88636]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T14:21:25.318 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:21:24 smithi069 ceph-mon[88636]: Reconfiguring daemon rgw.foo.smithi122.jiobyv on smithi122 2024-04-14T14:21:25.318 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:21:24 smithi069 ceph-mon[88636]: pgmap v27: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:21:26.817 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 14 14:21:26 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-y[63038]: ::ffff:172.21.15.122 - - [14/Apr/2024:14:21:26] "GET /metrics HTTP/1.1" 200 38022 "" "Prometheus/2.43.0" 2024-04-14T14:21:26.818 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:21:26 smithi069 ceph-mon[88636]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:21:26.818 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:21:26 smithi069 ceph-mon[88636]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:21:26.818 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:21:26 smithi069 ceph-mon[88636]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-14T14:21:26.818 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:21:26 smithi069 ceph-mon[88636]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T14:21:26.818 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:21:26 smithi069 ceph-mon[88636]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T14:21:26.818 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:21:26 smithi069 ceph-mon[88636]: Upgrade: Setting container_image for all mon 2024-04-14T14:21:26.818 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:21:26 smithi069 ceph-mon[88636]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:21:26.818 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:21:26 smithi069 ceph-mon[88636]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.a"}]: dispatch 2024-04-14T14:21:26.819 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:21:26 smithi069 ceph-mon[88636]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon.a"}]': finished 2024-04-14T14:21:26.819 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:21:26 smithi069 ceph-mon[88636]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.c"}]: dispatch 2024-04-14T14:21:26.819 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:21:26 smithi069 ceph-mon[88636]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon.c"}]': finished 2024-04-14T14:21:26.819 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:21:26 smithi069 ceph-mon[88636]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T14:21:26.819 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:21:26 smithi069 ceph-mon[88636]: Upgrade: Setting container_image for all crash 2024-04-14T14:21:26.819 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:21:26 smithi069 ceph-mon[88636]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:21:26.819 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:21:26 smithi069 ceph-mon[88636]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T14:21:26.819 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:21:26 smithi069 ceph-mon[88636]: Upgrade: Setting container_image for all mds 2024-04-14T14:21:26.819 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:21:26 smithi069 ceph-mon[88636]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:21:26.819 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:21:26 smithi069 ceph-mon[88636]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T14:21:26.819 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:21:26 smithi069 ceph-mon[88636]: Upgrade: Setting container_image for all rbd-mirror 2024-04-14T14:21:26.819 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:21:26 smithi069 ceph-mon[88636]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:21:26.819 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:21:26 smithi069 ceph-mon[88636]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T14:21:26.819 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:21:26 smithi069 ceph-mon[88636]: Upgrade: Setting container_image for all cephfs-mirror 2024-04-14T14:21:26.820 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:21:26 smithi069 ceph-mon[88636]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.a"}]: dispatch 2024-04-14T14:21:26.820 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:21:26 smithi069 ceph-mon[88636]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.c"}]: dispatch 2024-04-14T14:21:26.820 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:21:26 smithi069 ceph-mon[88636]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T14:21:26.820 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:21:26 smithi069 ceph-mon[88636]: Upgrade: Setting container_image for all ceph-exporter 2024-04-14T14:21:26.820 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:21:26 smithi069 ceph-mon[88636]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:21:26.820 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:21:26 smithi069 ceph-mon[88636]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T14:21:26.820 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:21:26 smithi069 ceph-mon[88636]: Upgrade: Setting container_image for all nfs 2024-04-14T14:21:26.820 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:21:26 smithi069 ceph-mon[88636]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:21:26.820 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:21:26 smithi069 ceph-mon[88636]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T14:21:26.820 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:21:26 smithi069 ceph-mon[88636]: Upgrade: Setting container_image for all nvmeof 2024-04-14T14:21:26.820 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:21:26 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:21:26.820 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:21:26 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:21:26.821 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:21:26 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-14T14:21:26.821 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:21:26 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T14:21:26.821 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:21:26 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T14:21:26.821 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:21:26 smithi069 ceph-mon[83680]: Upgrade: Setting container_image for all mon 2024-04-14T14:21:26.821 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:21:26 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:21:26.821 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:21:26 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.a"}]: dispatch 2024-04-14T14:21:26.821 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:21:26 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon.a"}]': finished 2024-04-14T14:21:26.821 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:21:26 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.c"}]: dispatch 2024-04-14T14:21:26.821 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:21:26 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon.c"}]': finished 2024-04-14T14:21:26.821 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:21:26 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T14:21:26.821 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:21:26 smithi069 ceph-mon[83680]: Upgrade: Setting container_image for all crash 2024-04-14T14:21:26.821 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:21:26 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:21:26.821 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:21:26 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T14:21:26.821 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:21:26 smithi069 ceph-mon[83680]: Upgrade: Setting container_image for all mds 2024-04-14T14:21:26.822 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:21:26 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:21:26.822 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:21:26 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T14:21:26.822 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:21:26 smithi069 ceph-mon[83680]: Upgrade: Setting container_image for all rbd-mirror 2024-04-14T14:21:26.822 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:21:26 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:21:26.822 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:21:26 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T14:21:26.822 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:21:26 smithi069 ceph-mon[83680]: Upgrade: Setting container_image for all cephfs-mirror 2024-04-14T14:21:26.822 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:21:26 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.a"}]: dispatch 2024-04-14T14:21:26.822 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:21:26 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.c"}]: dispatch 2024-04-14T14:21:26.822 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:21:26 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T14:21:26.822 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:21:26 smithi069 ceph-mon[83680]: Upgrade: Setting container_image for all ceph-exporter 2024-04-14T14:21:26.822 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:21:26 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:21:26.822 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:21:26 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T14:21:26.822 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:21:26 smithi069 ceph-mon[83680]: Upgrade: Setting container_image for all nfs 2024-04-14T14:21:26.822 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:21:26 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:21:26.822 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:21:26 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T14:21:26.823 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:21:26 smithi069 ceph-mon[83680]: Upgrade: Setting container_image for all nvmeof 2024-04-14T14:21:26.823 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:21:26 smithi069 ceph-mon[88636]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:21:26.823 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:21:26 smithi069 ceph-mon[88636]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T14:21:26.823 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:21:26 smithi069 ceph-mon[88636]: Upgrade: Setting container_image for all node-exporter 2024-04-14T14:21:26.823 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:21:26 smithi069 ceph-mon[88636]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.a"}]: dispatch 2024-04-14T14:21:26.823 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:21:26 smithi069 ceph-mon[88636]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.c"}]: dispatch 2024-04-14T14:21:26.823 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:21:26 smithi069 ceph-mon[88636]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T14:21:26.823 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:21:26 smithi069 ceph-mon[88636]: Upgrade: Setting container_image for all prometheus 2024-04-14T14:21:26.823 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:21:26 smithi069 ceph-mon[88636]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.a"}]: dispatch 2024-04-14T14:21:26.823 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:21:26 smithi069 ceph-mon[88636]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.c"}]: dispatch 2024-04-14T14:21:26.823 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:21:26 smithi069 ceph-mon[88636]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T14:21:26.823 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:21:26 smithi069 ceph-mon[88636]: Upgrade: Setting container_image for all alertmanager 2024-04-14T14:21:26.824 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:21:26 smithi069 ceph-mon[88636]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.a"}]: dispatch 2024-04-14T14:21:26.824 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:21:26 smithi069 ceph-mon[88636]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.c"}]: dispatch 2024-04-14T14:21:26.824 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:21:26 smithi069 ceph-mon[88636]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T14:21:26.824 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:21:26 smithi069 ceph-mon[88636]: Upgrade: Setting container_image for all grafana 2024-04-14T14:21:26.824 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:21:26 smithi069 ceph-mon[88636]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.a"}]: dispatch 2024-04-14T14:21:26.824 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:21:26 smithi069 ceph-mon[88636]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.c"}]: dispatch 2024-04-14T14:21:26.824 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:21:26 smithi069 ceph-mon[88636]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T14:21:26.824 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:21:26 smithi069 ceph-mon[88636]: Upgrade: Setting container_image for all loki 2024-04-14T14:21:26.824 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:21:26 smithi069 ceph-mon[88636]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.a"}]: dispatch 2024-04-14T14:21:26.824 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:21:26 smithi069 ceph-mon[88636]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.c"}]: dispatch 2024-04-14T14:21:26.824 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:21:26 smithi069 ceph-mon[88636]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T14:21:26.824 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:21:26 smithi069 ceph-mon[88636]: Upgrade: Setting container_image for all promtail 2024-04-14T14:21:26.824 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:21:26 smithi069 ceph-mon[88636]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.a"}]: dispatch 2024-04-14T14:21:26.824 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:21:26 smithi069 ceph-mon[88636]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.c"}]: dispatch 2024-04-14T14:21:26.824 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:21:26 smithi069 ceph-mon[88636]: Upgrade: Finalizing container_image settings 2024-04-14T14:21:26.825 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:21:26 smithi069 ceph-mon[88636]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]: dispatch 2024-04-14T14:21:26.825 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:21:26 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:21:26.825 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:21:26 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T14:21:26.825 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:21:26 smithi069 ceph-mon[83680]: Upgrade: Setting container_image for all node-exporter 2024-04-14T14:21:26.825 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:21:26 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.a"}]: dispatch 2024-04-14T14:21:26.825 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:21:26 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.c"}]: dispatch 2024-04-14T14:21:26.825 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:21:26 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T14:21:26.825 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:21:26 smithi069 ceph-mon[83680]: Upgrade: Setting container_image for all prometheus 2024-04-14T14:21:26.825 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:21:26 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.a"}]: dispatch 2024-04-14T14:21:26.825 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:21:26 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.c"}]: dispatch 2024-04-14T14:21:26.826 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:21:26 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T14:21:26.826 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:21:26 smithi069 ceph-mon[83680]: Upgrade: Setting container_image for all alertmanager 2024-04-14T14:21:26.826 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:21:26 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.a"}]: dispatch 2024-04-14T14:21:26.826 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:21:26 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.c"}]: dispatch 2024-04-14T14:21:26.826 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:21:26 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T14:21:26.826 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:21:26 smithi069 ceph-mon[83680]: Upgrade: Setting container_image for all grafana 2024-04-14T14:21:26.826 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:21:26 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.a"}]: dispatch 2024-04-14T14:21:26.826 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:21:26 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.c"}]: dispatch 2024-04-14T14:21:26.826 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:21:26 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T14:21:26.826 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:21:26 smithi069 ceph-mon[83680]: Upgrade: Setting container_image for all loki 2024-04-14T14:21:26.826 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:21:26 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.a"}]: dispatch 2024-04-14T14:21:26.826 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:21:26 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.c"}]: dispatch 2024-04-14T14:21:26.826 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:21:26 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T14:21:26.826 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:21:26 smithi069 ceph-mon[83680]: Upgrade: Setting container_image for all promtail 2024-04-14T14:21:26.826 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:21:26 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.a"}]: dispatch 2024-04-14T14:21:26.827 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:21:26 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.c"}]: dispatch 2024-04-14T14:21:26.827 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:21:26 smithi069 ceph-mon[83680]: Upgrade: Finalizing container_image settings 2024-04-14T14:21:26.827 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:21:26 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]: dispatch 2024-04-14T14:21:26.827 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:21:26 smithi069 ceph-mon[88636]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]': finished 2024-04-14T14:21:26.827 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:21:26 smithi069 ceph-mon[88636]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-14T14:21:26.827 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:21:26 smithi069 ceph-mon[88636]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon"}]': finished 2024-04-14T14:21:26.827 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:21:26 smithi069 ceph-mon[88636]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]: dispatch 2024-04-14T14:21:26.827 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:21:26 smithi069 ceph-mon[88636]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]': finished 2024-04-14T14:21:26.827 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:21:26 smithi069 ceph-mon[88636]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd"}]: dispatch 2024-04-14T14:21:26.828 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:21:26 smithi069 ceph-mon[88636]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mds"}]: dispatch 2024-04-14T14:21:26.828 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:21:26 smithi069 ceph-mon[88636]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mds"}]': finished 2024-04-14T14:21:26.828 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:21:26 smithi069 ceph-mon[88636]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw"}]: dispatch 2024-04-14T14:21:26.828 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:21:26 smithi069 ceph-mon[88636]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"}]: dispatch 2024-04-14T14:21:26.828 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:21:26 smithi069 ceph-mon[88636]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"}]': finished 2024-04-14T14:21:26.828 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:21:26 smithi069 ceph-mon[88636]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-14T14:21:26.828 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:21:26 smithi069 ceph-mon[88636]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter"}]: dispatch 2024-04-14T14:21:26.828 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:21:26 smithi069 ceph-mon[88636]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter"}]': finished 2024-04-14T14:21:26.828 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:21:26 smithi069 ceph-mon[88636]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.iscsi"}]: dispatch 2024-04-14T14:21:26.828 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:21:26 smithi069 ceph-mon[88636]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nfs"}]: dispatch 2024-04-14T14:21:26.828 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:21:26 smithi069 ceph-mon[88636]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.nfs"}]': finished 2024-04-14T14:21:26.828 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:21:26 smithi069 ceph-mon[88636]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]: dispatch 2024-04-14T14:21:26.828 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:21:26 smithi069 ceph-mon[88636]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]': finished 2024-04-14T14:21:26.828 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:21:26 smithi069 ceph-mon[88636]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-14T14:21:26.828 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:21:26 smithi069 ceph-mon[88636]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-14T14:21:26.829 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:21:26 smithi069 ceph-mon[88636]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-14T14:21:26.829 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:21:26 smithi069 ceph-mon[88636]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-14T14:21:26.829 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:21:26 smithi069 ceph-mon[88636]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-14T14:21:26.829 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:21:26 smithi069 ceph-mon[88636]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-14T14:21:26.829 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:21:26 smithi069 ceph-mon[88636]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]: dispatch 2024-04-14T14:21:26.829 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:21:26 smithi069 ceph-mon[88636]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd='[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]': finished 2024-04-14T14:21:26.829 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:21:26 smithi069 ceph-mon[88636]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T14:21:26.829 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:21:26 smithi069 ceph-mon[88636]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-14T14:21:26.829 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:21:26 smithi069 ceph-mon[88636]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:21:26.829 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:21:26 smithi069 ceph-mon[88636]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T14:21:26.829 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:21:26 smithi069 ceph-mon[88636]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-14T14:21:26.829 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:21:26 smithi069 ceph-mon[88636]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:21:26.830 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:21:26 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]': finished 2024-04-14T14:21:26.830 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:21:26 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-14T14:21:26.830 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:21:26 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon"}]': finished 2024-04-14T14:21:26.830 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:21:26 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]: dispatch 2024-04-14T14:21:26.830 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:21:26 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]': finished 2024-04-14T14:21:26.830 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:21:26 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd"}]: dispatch 2024-04-14T14:21:26.830 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:21:26 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mds"}]: dispatch 2024-04-14T14:21:26.830 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:21:26 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mds"}]': finished 2024-04-14T14:21:26.830 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:21:26 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw"}]: dispatch 2024-04-14T14:21:26.830 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:21:26 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"}]: dispatch 2024-04-14T14:21:26.830 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:21:26 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"}]': finished 2024-04-14T14:21:26.830 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:21:26 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-14T14:21:26.831 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:21:26 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter"}]: dispatch 2024-04-14T14:21:26.831 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:21:26 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter"}]': finished 2024-04-14T14:21:26.831 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:21:26 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.iscsi"}]: dispatch 2024-04-14T14:21:26.831 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:21:26 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nfs"}]: dispatch 2024-04-14T14:21:26.831 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:21:26 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.nfs"}]': finished 2024-04-14T14:21:26.831 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:21:26 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]: dispatch 2024-04-14T14:21:26.831 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:21:26 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]': finished 2024-04-14T14:21:26.831 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:21:26 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-14T14:21:26.831 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:21:26 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-14T14:21:26.831 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:21:26 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-14T14:21:26.831 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:21:26 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-14T14:21:26.831 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:21:26 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-14T14:21:26.831 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:21:26 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-14T14:21:26.831 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:21:26 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]: dispatch 2024-04-14T14:21:26.831 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:21:26 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd='[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]': finished 2024-04-14T14:21:26.832 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:21:26 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T14:21:26.832 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:21:26 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-14T14:21:26.832 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:21:26 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:21:26.832 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:21:26 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T14:21:26.832 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:21:26 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-14T14:21:26.832 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:21:26 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:21:26.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:21:26 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:21:26.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:21:26 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:21:26.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:21:26 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-14T14:21:26.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:21:26 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T14:21:26.846 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:21:26 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T14:21:26.846 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:21:26 smithi122 ceph-mon[67998]: Upgrade: Setting container_image for all mon 2024-04-14T14:21:26.846 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:21:26 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:21:26.846 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:21:26 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.a"}]: dispatch 2024-04-14T14:21:26.846 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:21:26 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon.a"}]': finished 2024-04-14T14:21:26.846 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:21:26 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.c"}]: dispatch 2024-04-14T14:21:26.846 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:21:26 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon.c"}]': finished 2024-04-14T14:21:26.846 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:21:26 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T14:21:26.846 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:21:26 smithi122 ceph-mon[67998]: Upgrade: Setting container_image for all crash 2024-04-14T14:21:26.846 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:21:26 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:21:26.846 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:21:26 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T14:21:26.846 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:21:26 smithi122 ceph-mon[67998]: Upgrade: Setting container_image for all mds 2024-04-14T14:21:26.847 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:21:26 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:21:26.847 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:21:26 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T14:21:26.847 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:21:26 smithi122 ceph-mon[67998]: Upgrade: Setting container_image for all rbd-mirror 2024-04-14T14:21:26.847 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:21:26 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:21:26.847 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:21:26 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T14:21:26.847 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:21:26 smithi122 ceph-mon[67998]: Upgrade: Setting container_image for all cephfs-mirror 2024-04-14T14:21:26.847 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:21:26 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.a"}]: dispatch 2024-04-14T14:21:26.847 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:21:26 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.c"}]: dispatch 2024-04-14T14:21:26.847 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:21:26 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T14:21:26.847 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:21:26 smithi122 ceph-mon[67998]: Upgrade: Setting container_image for all ceph-exporter 2024-04-14T14:21:26.847 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:21:26 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:21:26.847 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:21:26 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T14:21:26.847 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:21:26 smithi122 ceph-mon[67998]: Upgrade: Setting container_image for all nfs 2024-04-14T14:21:26.847 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:21:26 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:21:26.848 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:21:26 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T14:21:26.848 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:21:26 smithi122 ceph-mon[67998]: Upgrade: Setting container_image for all nvmeof 2024-04-14T14:21:26.848 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:21:26 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:21:26.848 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:21:26 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T14:21:26.848 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:21:26 smithi122 ceph-mon[67998]: Upgrade: Setting container_image for all node-exporter 2024-04-14T14:21:26.848 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:21:26 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.a"}]: dispatch 2024-04-14T14:21:26.848 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:21:26 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.c"}]: dispatch 2024-04-14T14:21:26.848 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:21:26 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T14:21:26.848 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:21:26 smithi122 ceph-mon[67998]: Upgrade: Setting container_image for all prometheus 2024-04-14T14:21:26.848 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:21:26 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.a"}]: dispatch 2024-04-14T14:21:26.848 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:21:26 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.c"}]: dispatch 2024-04-14T14:21:26.848 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:21:26 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T14:21:26.848 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:21:26 smithi122 ceph-mon[67998]: Upgrade: Setting container_image for all alertmanager 2024-04-14T14:21:26.848 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:21:26 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.a"}]: dispatch 2024-04-14T14:21:26.848 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:21:26 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.c"}]: dispatch 2024-04-14T14:21:26.849 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:21:26 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T14:21:26.849 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:21:26 smithi122 ceph-mon[67998]: Upgrade: Setting container_image for all grafana 2024-04-14T14:21:26.849 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:21:26 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.a"}]: dispatch 2024-04-14T14:21:26.849 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:21:26 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.c"}]: dispatch 2024-04-14T14:21:26.849 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:21:26 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T14:21:26.849 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:21:26 smithi122 ceph-mon[67998]: Upgrade: Setting container_image for all loki 2024-04-14T14:21:26.849 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:21:26 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.a"}]: dispatch 2024-04-14T14:21:26.849 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:21:26 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.c"}]: dispatch 2024-04-14T14:21:26.849 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:21:26 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T14:21:26.849 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:21:26 smithi122 ceph-mon[67998]: Upgrade: Setting container_image for all promtail 2024-04-14T14:21:26.849 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:21:26 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.a"}]: dispatch 2024-04-14T14:21:26.849 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:21:26 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.c"}]: dispatch 2024-04-14T14:21:26.849 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:21:26 smithi122 ceph-mon[67998]: Upgrade: Finalizing container_image settings 2024-04-14T14:21:26.849 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:21:26 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]: dispatch 2024-04-14T14:21:26.850 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:21:26 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]': finished 2024-04-14T14:21:26.850 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:21:26 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-14T14:21:26.850 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:21:26 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon"}]': finished 2024-04-14T14:21:26.850 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:21:26 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]: dispatch 2024-04-14T14:21:26.850 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:21:26 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]': finished 2024-04-14T14:21:26.850 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:21:26 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd"}]: dispatch 2024-04-14T14:21:26.850 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:21:26 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mds"}]: dispatch 2024-04-14T14:21:26.850 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:21:26 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mds"}]': finished 2024-04-14T14:21:26.851 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:21:26 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw"}]: dispatch 2024-04-14T14:21:26.851 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:21:26 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"}]: dispatch 2024-04-14T14:21:26.851 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:21:26 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"}]': finished 2024-04-14T14:21:26.851 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:21:26 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-14T14:21:26.851 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:21:26 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter"}]: dispatch 2024-04-14T14:21:26.851 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:21:26 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter"}]': finished 2024-04-14T14:21:26.851 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:21:26 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.iscsi"}]: dispatch 2024-04-14T14:21:26.851 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:21:26 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nfs"}]: dispatch 2024-04-14T14:21:26.851 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:21:26 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.nfs"}]': finished 2024-04-14T14:21:26.851 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:21:26 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]: dispatch 2024-04-14T14:21:26.851 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:21:26 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]': finished 2024-04-14T14:21:26.851 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:21:26 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-14T14:21:26.851 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:21:26 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-14T14:21:26.851 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:21:26 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-14T14:21:26.852 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:21:26 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-14T14:21:26.852 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:21:26 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-14T14:21:26.852 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:21:26 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-14T14:21:26.852 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:21:26 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]: dispatch 2024-04-14T14:21:26.852 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:21:26 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd='[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]': finished 2024-04-14T14:21:26.852 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:21:26 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T14:21:26.852 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:21:26 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-14T14:21:26.852 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:21:26 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:21:26.852 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:21:26 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T14:21:26.852 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:21:26 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-14T14:21:26.852 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:21:26 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:21:27.817 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:21:27 smithi069 ceph-mon[83680]: Upgrade: Complete! 2024-04-14T14:21:27.817 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:21:27 smithi069 ceph-mon[83680]: pgmap v28: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:21:27.817 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:21:27 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:21:27.817 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:21:27 smithi069 ceph-mon[88636]: Upgrade: Complete! 2024-04-14T14:21:27.817 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:21:27 smithi069 ceph-mon[88636]: pgmap v28: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:21:27.817 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:21:27 smithi069 ceph-mon[88636]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:21:27.844 DEBUG:teuthology.orchestra.run.smithi069:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid dc326fc2-fa65-11ee-bc8e-c7b262605968 -e sha1=a971fd5f5719038c70b817bafacd6d5400b1d888 -- bash -c 'ceph orch ps' 2024-04-14T14:21:27.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:21:27 smithi122 ceph-mon[67998]: Upgrade: Complete! 2024-04-14T14:21:27.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:21:27 smithi122 ceph-mon[67998]: pgmap v28: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:21:27.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:21:27 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:21:28.802 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:21:28 smithi069 ceph-mon[88636]: from='client.44166 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T14:21:28.802 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:21:28 smithi069 ceph-mon[88636]: from='client.25417 -' entity='client.iscsi.foo.smithi069.yuumtr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T14:21:28.802 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:21:28 smithi069 ceph-mon[83680]: from='client.44166 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T14:21:28.802 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:21:28 smithi069 ceph-mon[83680]: from='client.25417 -' entity='client.iscsi.foo.smithi069.yuumtr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T14:21:28.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:21:28 smithi122 ceph-mon[67998]: from='client.44166 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T14:21:28.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:21:28 smithi122 ceph-mon[67998]: from='client.25417 -' entity='client.iscsi.foo.smithi069.yuumtr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T14:21:29.726 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:21:29 smithi069 ceph-mon[83680]: pgmap v29: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:21:29.727 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:21:29 smithi069 ceph-mon[88636]: pgmap v29: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:21:29.727 INFO:teuthology.orchestra.run.smithi069.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-14T14:21:29.727 INFO:teuthology.orchestra.run.smithi069.stdout:alertmanager.a smithi069 *:9093,9094 running (3m) 32s ago 24m 19.0M - 0.25.0 c8568f914cd2 7524f106e7bd 2024-04-14T14:21:29.727 INFO:teuthology.orchestra.run.smithi069.stdout:grafana.a smithi122 *:3000 running (3m) 48s ago 24m 81.0M - 9.4.12 2bacad6d85d8 5685178e3eba 2024-04-14T14:21:29.727 INFO:teuthology.orchestra.run.smithi069.stdout:iscsi.foo.smithi069.yuumtr smithi069 running (4m) 32s ago 24m 55.4M - 3.5 e1d6a67b021e 38cc81ea8cfa 2024-04-14T14:21:29.727 INFO:teuthology.orchestra.run.smithi069.stdout:mgr.x smithi122 *:8443,9283,8765 running (4m) 48s ago 29m 450M - 19.0.0-2778-ga971fd5f b96c509eef84 e29977590b43 2024-04-14T14:21:29.727 INFO:teuthology.orchestra.run.smithi069.stdout:mgr.y smithi069 *:8443,9283,8765 running (13m) 32s ago 31m 492M - 19.0.0-2778-ga971fd5f b96c509eef84 7af3d29d0e5f 2024-04-14T14:21:29.727 INFO:teuthology.orchestra.run.smithi069.stdout:mon.a smithi069 running (62s) 32s ago 31m 43.1M 2048M 19.0.0-2778-ga971fd5f b96c509eef84 8bcd6b3d3846 2024-04-14T14:21:29.728 INFO:teuthology.orchestra.run.smithi069.stdout:mon.b smithi122 running (110s) 48s ago 29m 33.9M 2048M 19.0.0-2778-ga971fd5f b96c509eef84 05f437a1a9a1 2024-04-14T14:21:29.728 INFO:teuthology.orchestra.run.smithi069.stdout:mon.c smithi069 running (37s) 32s ago 29m 17.9M 2048M 19.0.0-2778-ga971fd5f b96c509eef84 84aa95f11d0c 2024-04-14T14:21:29.728 INFO:teuthology.orchestra.run.smithi069.stdout:node-exporter.a smithi069 *:9100 running (3m) 32s ago 25m 8334k - 1.5.0 0da6a335fe13 0de0eb2171b9 2024-04-14T14:21:29.728 INFO:teuthology.orchestra.run.smithi069.stdout:node-exporter.b smithi122 *:9100 running (3m) 48s ago 25m 16.1M - 1.5.0 0da6a335fe13 67877bf02d8e 2024-04-14T14:21:29.728 INFO:teuthology.orchestra.run.smithi069.stdout:osd.0 smithi069 running (29m) 32s ago 29m 56.6M 2992M 17.2.0 e1d6a67b021e 46654654b6b6 2024-04-14T14:21:29.728 INFO:teuthology.orchestra.run.smithi069.stdout:osd.1 smithi069 running (28m) 32s ago 28m 55.1M 2992M 17.2.0 e1d6a67b021e cdba54a59609 2024-04-14T14:21:29.728 INFO:teuthology.orchestra.run.smithi069.stdout:osd.2 smithi069 running (28m) 32s ago 28m 51.1M 2992M 17.2.0 e1d6a67b021e bc37a1a9cba0 2024-04-14T14:21:29.728 INFO:teuthology.orchestra.run.smithi069.stdout:osd.3 smithi069 running (27m) 32s ago 27m 55.7M 2992M 17.2.0 e1d6a67b021e 0925ba3da801 2024-04-14T14:21:29.728 INFO:teuthology.orchestra.run.smithi069.stdout:osd.4 smithi122 running (27m) 48s ago 27m 56.1M 3248M 17.2.0 e1d6a67b021e c00975caf771 2024-04-14T14:21:29.728 INFO:teuthology.orchestra.run.smithi069.stdout:osd.5 smithi122 running (27m) 48s ago 27m 55.2M 3248M 17.2.0 e1d6a67b021e 5f52d217ceb8 2024-04-14T14:21:29.728 INFO:teuthology.orchestra.run.smithi069.stdout:osd.6 smithi122 running (26m) 48s ago 26m 52.4M 3248M 17.2.0 e1d6a67b021e a2acfec61475 2024-04-14T14:21:29.728 INFO:teuthology.orchestra.run.smithi069.stdout:osd.7 smithi122 running (26m) 48s ago 26m 57.1M 3248M 17.2.0 e1d6a67b021e 62411cd06801 2024-04-14T14:21:29.728 INFO:teuthology.orchestra.run.smithi069.stdout:prometheus.a smithi122 *:9095 running (3m) 48s ago 25m 66.4M - 2.43.0 a07b618ecd1d 5c1f41d3bbe6 2024-04-14T14:21:29.728 INFO:teuthology.orchestra.run.smithi069.stdout:rgw.foo.smithi069.wesedz smithi069 *:8000 running (24m) 32s ago 24m 78.8M - 17.2.0 e1d6a67b021e 16de491440f3 2024-04-14T14:21:29.729 INFO:teuthology.orchestra.run.smithi069.stdout:rgw.foo.smithi122.jiobyv smithi122 *:8000 running (24m) 48s ago 24m 75.8M - 17.2.0 e1d6a67b021e 6d16de445dba 2024-04-14T14:21:29.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:21:29 smithi122 ceph-mon[67998]: pgmap v29: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:21:30.334 DEBUG:teuthology.orchestra.run.smithi069:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid dc326fc2-fa65-11ee-bc8e-c7b262605968 -e sha1=a971fd5f5719038c70b817bafacd6d5400b1d888 -- bash -c 'ceph versions | jq -e '"'"'.mon | length == 1'"'"'' 2024-04-14T14:21:31.840 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:21:31 smithi069 ceph-mon[83680]: from='client.44172 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T14:21:31.840 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:21:31 smithi069 ceph-mon[83680]: pgmap v30: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:21:31.840 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:21:31 smithi069 ceph-mon[88636]: from='client.44172 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T14:21:31.840 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:21:31 smithi069 ceph-mon[88636]: pgmap v30: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:21:31.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:21:31 smithi122 ceph-mon[67998]: from='client.44172 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T14:21:31.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:21:31 smithi122 ceph-mon[67998]: pgmap v30: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:21:32.372 INFO:teuthology.orchestra.run.smithi069.stdout:true 2024-04-14T14:21:32.673 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:21:32 smithi069 ceph-mon[83680]: from='client.? 172.21.15.69:0/1064049873' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T14:21:32.673 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:21:32 smithi069 ceph-mon[88636]: from='client.? 172.21.15.69:0/1064049873' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T14:21:32.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:21:32 smithi122 ceph-mon[67998]: from='client.? 172.21.15.69:0/1064049873' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T14:21:33.008 DEBUG:teuthology.orchestra.run.smithi069:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid dc326fc2-fa65-11ee-bc8e-c7b262605968 -e sha1=a971fd5f5719038c70b817bafacd6d5400b1d888 -- bash -c 'ceph versions | jq -e '"'"'.mon | keys'"'"' | grep $sha1' 2024-04-14T14:21:33.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:21:33 smithi122 ceph-mon[67998]: pgmap v31: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:21:34.063 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:21:33 smithi069 ceph-mon[83680]: pgmap v31: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:21:34.063 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:21:33 smithi069 ceph-mon[88636]: pgmap v31: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:21:35.073 INFO:teuthology.orchestra.run.smithi069.stdout: "ceph version 19.0.0-2778-ga971fd5f (a971fd5f5719038c70b817bafacd6d5400b1d888) squid (dev)" 2024-04-14T14:21:35.678 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:21:35 smithi069 ceph-mon[83680]: pgmap v32: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:21:35.679 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:21:35 smithi069 ceph-mon[83680]: from='client.? 172.21.15.69:0/337668617' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T14:21:35.679 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:21:35 smithi069 ceph-mon[88636]: pgmap v32: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:21:35.679 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:21:35 smithi069 ceph-mon[88636]: from='client.? 172.21.15.69:0/337668617' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T14:21:35.702 DEBUG:teuthology.orchestra.run.smithi069:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid dc326fc2-fa65-11ee-bc8e-c7b262605968 -e sha1=a971fd5f5719038c70b817bafacd6d5400b1d888 -- bash -c 'ceph orch upgrade check quay.ceph.io/ceph-ci/ceph:$sha1 | jq -e '"'"'.up_to_date | length == 5'"'"'' 2024-04-14T14:21:35.846 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:21:35 smithi122 ceph-mon[67998]: pgmap v32: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:21:35.846 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:21:35 smithi122 ceph-mon[67998]: from='client.? 172.21.15.69:0/337668617' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T14:21:36.817 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:21:36 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T14:21:36.817 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:21:36 smithi069 ceph-mon[88636]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T14:21:36.817 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 14 14:21:36 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-y[63038]: ::ffff:172.21.15.122 - - [14/Apr/2024:14:21:36] "GET /metrics HTTP/1.1" 200 38022 "" "Prometheus/2.43.0" 2024-04-14T14:21:36.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:21:36 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T14:21:37.817 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:21:37 smithi069 ceph-mon[83680]: pgmap v33: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:21:37.817 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:21:37 smithi069 ceph-mon[88636]: pgmap v33: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:21:37.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:21:37 smithi122 ceph-mon[67998]: pgmap v33: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:21:39.023 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:21:38 smithi069 ceph-mon[83680]: from='client.25417 -' entity='client.iscsi.foo.smithi069.yuumtr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T14:21:39.023 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:21:38 smithi069 ceph-mon[88636]: from='client.25417 -' entity='client.iscsi.foo.smithi069.yuumtr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T14:21:39.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:21:38 smithi122 ceph-mon[67998]: from='client.25417 -' entity='client.iscsi.foo.smithi069.yuumtr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T14:21:39.251 INFO:teuthology.orchestra.run.smithi069.stdout:true 2024-04-14T14:21:39.817 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:21:39 smithi069 ceph-mon[83680]: from='client.44190 -' entity='client.admin' cmd=[{"prefix": "orch upgrade check", "image": "quay.ceph.io/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T14:21:39.817 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:21:39 smithi069 ceph-mon[83680]: pgmap v34: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:21:39.817 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:21:39 smithi069 ceph-mon[88636]: from='client.44190 -' entity='client.admin' cmd=[{"prefix": "orch upgrade check", "image": "quay.ceph.io/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T14:21:39.817 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:21:39 smithi069 ceph-mon[88636]: pgmap v34: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:21:39.866 DEBUG:teuthology.orchestra.run.smithi069:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid dc326fc2-fa65-11ee-bc8e-c7b262605968 -e sha1=a971fd5f5719038c70b817bafacd6d5400b1d888 -- bash -c 'ceph orch upgrade status' 2024-04-14T14:21:40.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:21:39 smithi122 ceph-mon[67998]: from='client.44190 -' entity='client.admin' cmd=[{"prefix": "orch upgrade check", "image": "quay.ceph.io/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T14:21:40.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:21:39 smithi122 ceph-mon[67998]: pgmap v34: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:21:41.817 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:21:41 smithi069 ceph-mon[83680]: pgmap v35: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:21:41.817 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:21:41 smithi069 ceph-mon[88636]: pgmap v35: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:21:41.828 INFO:teuthology.orchestra.run.smithi069.stdout:{ 2024-04-14T14:21:41.829 INFO:teuthology.orchestra.run.smithi069.stdout: "target_image": null, 2024-04-14T14:21:41.829 INFO:teuthology.orchestra.run.smithi069.stdout: "in_progress": false, 2024-04-14T14:21:41.829 INFO:teuthology.orchestra.run.smithi069.stdout: "which": "", 2024-04-14T14:21:41.829 INFO:teuthology.orchestra.run.smithi069.stdout: "services_complete": [], 2024-04-14T14:21:41.829 INFO:teuthology.orchestra.run.smithi069.stdout: "progress": null, 2024-04-14T14:21:41.829 INFO:teuthology.orchestra.run.smithi069.stdout: "message": "", 2024-04-14T14:21:41.829 INFO:teuthology.orchestra.run.smithi069.stdout: "is_paused": false 2024-04-14T14:21:41.829 INFO:teuthology.orchestra.run.smithi069.stdout:} 2024-04-14T14:21:41.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:21:41 smithi122 ceph-mon[67998]: pgmap v35: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:21:42.475 DEBUG:teuthology.orchestra.run.smithi069:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid dc326fc2-fa65-11ee-bc8e-c7b262605968 -e sha1=a971fd5f5719038c70b817bafacd6d5400b1d888 -- bash -c 'ceph health detail' 2024-04-14T14:21:43.673 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:21:43 smithi069 ceph-mon[83680]: from='client.44196 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T14:21:43.673 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:21:43 smithi069 ceph-mon[83680]: pgmap v36: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:21:43.673 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:21:43 smithi069 ceph-mon[88636]: from='client.44196 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T14:21:43.673 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:21:43 smithi069 ceph-mon[88636]: pgmap v36: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:21:44.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:21:43 smithi122 ceph-mon[67998]: from='client.44196 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T14:21:44.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:21:43 smithi122 ceph-mon[67998]: pgmap v36: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:21:44.437 INFO:teuthology.orchestra.run.smithi069.stdout:HEALTH_OK 2024-04-14T14:21:44.796 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:21:44 smithi069 ceph-mon[83680]: from='client.? 172.21.15.69:0/3364893963' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-14T14:21:44.797 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:21:44 smithi069 ceph-mon[88636]: from='client.? 172.21.15.69:0/3364893963' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-14T14:21:45.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:21:44 smithi122 ceph-mon[67998]: from='client.? 172.21.15.69:0/3364893963' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-14T14:21:45.154 DEBUG:teuthology.orchestra.run.smithi069:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid dc326fc2-fa65-11ee-bc8e-c7b262605968 -e sha1=a971fd5f5719038c70b817bafacd6d5400b1d888 -- bash -c 'ceph orch upgrade start --image quay.ceph.io/ceph-ci/ceph:$sha1 --daemon-types osd --limit 2' 2024-04-14T14:21:45.809 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:21:45 smithi069 ceph-mon[83680]: pgmap v37: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:21:45.809 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:21:45 smithi069 ceph-mon[88636]: pgmap v37: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:21:46.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:21:45 smithi122 ceph-mon[67998]: pgmap v37: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:21:46.782 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 14 14:21:46 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-y[63038]: ::ffff:172.21.15.122 - - [14/Apr/2024:14:21:46] "GET /metrics HTTP/1.1" 200 38020 "" "Prometheus/2.43.0" 2024-04-14T14:21:47.907 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:21:47 smithi069 ceph-mon[83680]: pgmap v38: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:21:47.908 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:21:47 smithi069 ceph-mon[88636]: pgmap v38: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:21:48.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:21:47 smithi122 ceph-mon[67998]: pgmap v38: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:21:48.891 INFO:teuthology.orchestra.run.smithi069.stdout:Initiating upgrade to quay.ceph.io/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888 2024-04-14T14:21:48.891 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:21:48 smithi069 ceph-mon[83680]: from='client.54145 -' entity='client.admin' cmd=[{"prefix": "orch upgrade start", "image": "quay.ceph.io/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888", "daemon_types": "osd", "limit": 2, "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T14:21:48.891 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:21:48 smithi069 ceph-mon[83680]: from='client.25417 -' entity='client.iscsi.foo.smithi069.yuumtr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T14:21:48.892 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:21:48 smithi069 ceph-mon[88636]: from='client.54145 -' entity='client.admin' cmd=[{"prefix": "orch upgrade start", "image": "quay.ceph.io/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888", "daemon_types": "osd", "limit": 2, "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T14:21:48.892 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:21:48 smithi069 ceph-mon[88636]: from='client.25417 -' entity='client.iscsi.foo.smithi069.yuumtr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T14:21:49.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:21:48 smithi122 ceph-mon[67998]: from='client.54145 -' entity='client.admin' cmd=[{"prefix": "orch upgrade start", "image": "quay.ceph.io/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888", "daemon_types": "osd", "limit": 2, "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T14:21:49.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:21:48 smithi122 ceph-mon[67998]: from='client.25417 -' entity='client.iscsi.foo.smithi069.yuumtr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T14:21:49.584 DEBUG:teuthology.orchestra.run.smithi069:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid dc326fc2-fa65-11ee-bc8e-c7b262605968 -e sha1=a971fd5f5719038c70b817bafacd6d5400b1d888 -- bash -c 'while ceph orch upgrade status | jq '"'"'.in_progress'"'"' | grep true && ! ceph orch upgrade status | jq '"'"'.message'"'"' | grep Error ; do ceph orch ps ; ceph versions ; ceph orch upgrade status ; sleep 30 ; done' 2024-04-14T14:21:49.817 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:21:49 smithi069 ceph-mon[83680]: pgmap v39: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:21:49.817 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:21:49 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:21:49.817 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:21:49 smithi069 ceph-mon[88636]: pgmap v39: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:21:49.817 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:21:49 smithi069 ceph-mon[88636]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:21:50.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:21:49 smithi122 ceph-mon[67998]: pgmap v39: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:21:50.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:21:49 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:21:50.923 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:21:50 smithi069 ceph-mon[83680]: Upgrade: Started with target quay.ceph.io/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888 2024-04-14T14:21:50.923 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:21:50 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:21:50.923 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:21:50 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:21:50.923 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:21:50 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T14:21:50.923 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:21:50 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-14T14:21:50.923 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:21:50 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:21:50.924 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:21:50 smithi069 ceph-mon[88636]: Upgrade: Started with target quay.ceph.io/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888 2024-04-14T14:21:50.924 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:21:50 smithi069 ceph-mon[88636]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:21:50.924 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:21:50 smithi069 ceph-mon[88636]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:21:50.924 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:21:50 smithi069 ceph-mon[88636]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T14:21:50.924 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:21:50 smithi069 ceph-mon[88636]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-14T14:21:50.924 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:21:50 smithi069 ceph-mon[88636]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:21:51.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:21:50 smithi122 ceph-mon[67998]: Upgrade: Started with target quay.ceph.io/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888 2024-04-14T14:21:51.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:21:50 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:21:51.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:21:50 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:21:51.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:21:50 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T14:21:51.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:21:50 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-14T14:21:51.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:21:50 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:21:51.642 INFO:teuthology.orchestra.run.smithi069.stdout:true 2024-04-14T14:21:51.817 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:21:51 smithi069 ceph-mon[83680]: Upgrade: First pull of quay.ceph.io/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888 2024-04-14T14:21:51.817 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:21:51 smithi069 ceph-mon[83680]: pgmap v40: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:21:51.817 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:21:51 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T14:21:51.817 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:21:51 smithi069 ceph-mon[88636]: Upgrade: First pull of quay.ceph.io/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888 2024-04-14T14:21:51.817 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:21:51 smithi069 ceph-mon[88636]: pgmap v40: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:21:51.818 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:21:51 smithi069 ceph-mon[88636]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T14:21:52.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:21:51 smithi122 ceph-mon[67998]: Upgrade: First pull of quay.ceph.io/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888 2024-04-14T14:21:52.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:21:51 smithi122 ceph-mon[67998]: pgmap v40: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:21:52.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:21:51 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T14:21:52.309 INFO:teuthology.orchestra.run.smithi069.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-14T14:21:52.309 INFO:teuthology.orchestra.run.smithi069.stdout:alertmanager.a smithi069 *:9093,9094 running (3m) 55s ago 25m 19.0M - 0.25.0 c8568f914cd2 7524f106e7bd 2024-04-14T14:21:52.310 INFO:teuthology.orchestra.run.smithi069.stdout:grafana.a smithi122 *:3000 running (3m) 71s ago 25m 81.0M - 9.4.12 2bacad6d85d8 5685178e3eba 2024-04-14T14:21:52.310 INFO:teuthology.orchestra.run.smithi069.stdout:iscsi.foo.smithi069.yuumtr smithi069 running (4m) 55s ago 24m 55.4M - 3.5 e1d6a67b021e 38cc81ea8cfa 2024-04-14T14:21:52.310 INFO:teuthology.orchestra.run.smithi069.stdout:mgr.x smithi122 *:8443,9283,8765 running (4m) 71s ago 29m 450M - 19.0.0-2778-ga971fd5f b96c509eef84 e29977590b43 2024-04-14T14:21:52.310 INFO:teuthology.orchestra.run.smithi069.stdout:mgr.y smithi069 *:8443,9283,8765 running (14m) 55s ago 31m 492M - 19.0.0-2778-ga971fd5f b96c509eef84 7af3d29d0e5f 2024-04-14T14:21:52.310 INFO:teuthology.orchestra.run.smithi069.stdout:mon.a smithi069 running (84s) 55s ago 31m 43.1M 2048M 19.0.0-2778-ga971fd5f b96c509eef84 8bcd6b3d3846 2024-04-14T14:21:52.310 INFO:teuthology.orchestra.run.smithi069.stdout:mon.b smithi122 running (2m) 71s ago 30m 33.9M 2048M 19.0.0-2778-ga971fd5f b96c509eef84 05f437a1a9a1 2024-04-14T14:21:52.311 INFO:teuthology.orchestra.run.smithi069.stdout:mon.c smithi069 running (60s) 55s ago 30m 17.9M 2048M 19.0.0-2778-ga971fd5f b96c509eef84 84aa95f11d0c 2024-04-14T14:21:52.311 INFO:teuthology.orchestra.run.smithi069.stdout:node-exporter.a smithi069 *:9100 running (4m) 55s ago 26m 8334k - 1.5.0 0da6a335fe13 0de0eb2171b9 2024-04-14T14:21:52.311 INFO:teuthology.orchestra.run.smithi069.stdout:node-exporter.b smithi122 *:9100 running (4m) 71s ago 25m 16.1M - 1.5.0 0da6a335fe13 67877bf02d8e 2024-04-14T14:21:52.311 INFO:teuthology.orchestra.run.smithi069.stdout:osd.0 smithi069 running (29m) 55s ago 29m 56.6M 2992M 17.2.0 e1d6a67b021e 46654654b6b6 2024-04-14T14:21:52.311 INFO:teuthology.orchestra.run.smithi069.stdout:osd.1 smithi069 running (29m) 55s ago 29m 55.1M 2992M 17.2.0 e1d6a67b021e cdba54a59609 2024-04-14T14:21:52.311 INFO:teuthology.orchestra.run.smithi069.stdout:osd.2 smithi069 running (28m) 55s ago 28m 51.1M 2992M 17.2.0 e1d6a67b021e bc37a1a9cba0 2024-04-14T14:21:52.312 INFO:teuthology.orchestra.run.smithi069.stdout:osd.3 smithi069 running (28m) 55s ago 28m 55.7M 2992M 17.2.0 e1d6a67b021e 0925ba3da801 2024-04-14T14:21:52.312 INFO:teuthology.orchestra.run.smithi069.stdout:osd.4 smithi122 running (27m) 71s ago 27m 56.1M 3248M 17.2.0 e1d6a67b021e c00975caf771 2024-04-14T14:21:52.312 INFO:teuthology.orchestra.run.smithi069.stdout:osd.5 smithi122 running (27m) 71s ago 27m 55.2M 3248M 17.2.0 e1d6a67b021e 5f52d217ceb8 2024-04-14T14:21:52.312 INFO:teuthology.orchestra.run.smithi069.stdout:osd.6 smithi122 running (27m) 71s ago 27m 52.4M 3248M 17.2.0 e1d6a67b021e a2acfec61475 2024-04-14T14:21:52.312 INFO:teuthology.orchestra.run.smithi069.stdout:osd.7 smithi122 running (26m) 71s ago 26m 57.1M 3248M 17.2.0 e1d6a67b021e 62411cd06801 2024-04-14T14:21:52.312 INFO:teuthology.orchestra.run.smithi069.stdout:prometheus.a smithi122 *:9095 running (3m) 71s ago 26m 66.4M - 2.43.0 a07b618ecd1d 5c1f41d3bbe6 2024-04-14T14:21:52.313 INFO:teuthology.orchestra.run.smithi069.stdout:rgw.foo.smithi069.wesedz smithi069 *:8000 running (24m) 55s ago 24m 78.8M - 17.2.0 e1d6a67b021e 16de491440f3 2024-04-14T14:21:52.313 INFO:teuthology.orchestra.run.smithi069.stdout:rgw.foo.smithi122.jiobyv smithi122 *:8000 running (24m) 71s ago 24m 75.8M - 17.2.0 e1d6a67b021e 6d16de445dba 2024-04-14T14:21:52.723 INFO:teuthology.orchestra.run.smithi069.stdout:{ 2024-04-14T14:21:52.723 INFO:teuthology.orchestra.run.smithi069.stdout: "mon": { 2024-04-14T14:21:52.724 INFO:teuthology.orchestra.run.smithi069.stdout: "ceph version 19.0.0-2778-ga971fd5f (a971fd5f5719038c70b817bafacd6d5400b1d888) squid (dev)": 3 2024-04-14T14:21:52.724 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-14T14:21:52.724 INFO:teuthology.orchestra.run.smithi069.stdout: "mgr": { 2024-04-14T14:21:52.724 INFO:teuthology.orchestra.run.smithi069.stdout: "ceph version 19.0.0-2778-ga971fd5f (a971fd5f5719038c70b817bafacd6d5400b1d888) squid (dev)": 2 2024-04-14T14:21:52.724 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-14T14:21:52.724 INFO:teuthology.orchestra.run.smithi069.stdout: "osd": { 2024-04-14T14:21:52.724 INFO:teuthology.orchestra.run.smithi069.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 8 2024-04-14T14:21:52.724 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-14T14:21:52.724 INFO:teuthology.orchestra.run.smithi069.stdout: "rgw": { 2024-04-14T14:21:52.724 INFO:teuthology.orchestra.run.smithi069.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-04-14T14:21:52.724 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-14T14:21:52.724 INFO:teuthology.orchestra.run.smithi069.stdout: "overall": { 2024-04-14T14:21:52.724 INFO:teuthology.orchestra.run.smithi069.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 10, 2024-04-14T14:21:52.724 INFO:teuthology.orchestra.run.smithi069.stdout: "ceph version 19.0.0-2778-ga971fd5f (a971fd5f5719038c70b817bafacd6d5400b1d888) squid (dev)": 5 2024-04-14T14:21:52.725 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-14T14:21:52.725 INFO:teuthology.orchestra.run.smithi069.stdout:} 2024-04-14T14:21:52.817 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:21:52 smithi069 ceph-mon[88636]: from='client.44214 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T14:21:52.817 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:21:52 smithi069 ceph-mon[88636]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:21:52.817 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:21:52 smithi069 ceph-mon[88636]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-14T14:21:52.817 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:21:52 smithi069 ceph-mon[88636]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T14:21:52.817 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:21:52 smithi069 ceph-mon[88636]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:21:52.817 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:21:52 smithi069 ceph-mon[88636]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T14:21:52.817 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:21:52 smithi069 ceph-mon[88636]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:21:52.817 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:21:52 smithi069 ceph-mon[88636]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T14:21:52.817 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:21:52 smithi069 ceph-mon[88636]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:21:52.818 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:21:52 smithi069 ceph-mon[88636]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["0"], "max": 16}]: dispatch 2024-04-14T14:21:52.818 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:21:52 smithi069 ceph-mon[83680]: from='client.44214 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T14:21:52.818 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:21:52 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:21:52.818 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:21:52 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-14T14:21:52.818 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:21:52 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T14:21:52.818 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:21:52 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:21:52.818 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:21:52 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T14:21:52.818 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:21:52 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:21:52.818 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:21:52 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T14:21:52.818 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:21:52 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:21:52.819 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:21:52 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["0"], "max": 16}]: dispatch 2024-04-14T14:21:53.075 INFO:teuthology.orchestra.run.smithi069.stdout:{ 2024-04-14T14:21:53.075 INFO:teuthology.orchestra.run.smithi069.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888", 2024-04-14T14:21:53.075 INFO:teuthology.orchestra.run.smithi069.stdout: "in_progress": true, 2024-04-14T14:21:53.075 INFO:teuthology.orchestra.run.smithi069.stdout: "which": "Upgrading daemons of type(s) osd. Upgrade limited to 2 daemons (2 remaining).", 2024-04-14T14:21:53.075 INFO:teuthology.orchestra.run.smithi069.stdout: "services_complete": [], 2024-04-14T14:21:53.076 INFO:teuthology.orchestra.run.smithi069.stdout: "progress": "0/8 daemons upgraded", 2024-04-14T14:21:53.076 INFO:teuthology.orchestra.run.smithi069.stdout: "message": "Currently upgrading osd daemons", 2024-04-14T14:21:53.076 INFO:teuthology.orchestra.run.smithi069.stdout: "is_paused": false 2024-04-14T14:21:53.076 INFO:teuthology.orchestra.run.smithi069.stdout:} 2024-04-14T14:21:53.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:21:52 smithi122 ceph-mon[67998]: from='client.44214 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T14:21:53.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:21:52 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:21:53.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:21:52 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-14T14:21:53.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:21:52 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T14:21:53.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:21:52 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:21:53.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:21:52 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T14:21:53.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:21:52 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:21:53.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:21:52 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T14:21:53.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:21:52 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:21:53.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:21:52 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["0"], "max": 16}]: dispatch 2024-04-14T14:21:54.067 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:21:53 smithi069 ceph-mon[83680]: from='client.54157 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T14:21:54.067 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:21:53 smithi069 ceph-mon[83680]: from='client.44226 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T14:21:54.067 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:21:53 smithi069 ceph-mon[83680]: pgmap v41: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:21:54.067 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:21:53 smithi069 ceph-mon[83680]: Upgrade: Target is version 19.0.0-2778-ga971fd5f (squid) 2024-04-14T14:21:54.067 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:21:53 smithi069 ceph-mon[83680]: Upgrade: Target container is quay.ceph.io/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888, digests ['quay.ceph.io/ceph-ci/ceph@sha256:02f0a191d4d693952654c64f93cc9c0152602c0915251b3c94c2c4e32855c608'] 2024-04-14T14:21:54.067 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:21:53 smithi069 ceph-mon[83680]: Upgrade: Setting container_image for all mgr 2024-04-14T14:21:54.067 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:21:53 smithi069 ceph-mon[83680]: Upgrade: Setting container_image for all mon 2024-04-14T14:21:54.068 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:21:53 smithi069 ceph-mon[83680]: Upgrade: Setting container_image for all crash 2024-04-14T14:21:54.068 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:21:53 smithi069 ceph-mon[83680]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["0"], "max": 16}]: dispatch 2024-04-14T14:21:54.068 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:21:53 smithi069 ceph-mon[83680]: Upgrade: osd.0 is safe to restart 2024-04-14T14:21:54.068 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:21:53 smithi069 ceph-mon[83680]: from='client.? 172.21.15.69:0/4279883351' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T14:21:54.068 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:21:53 smithi069 ceph-mon[88636]: from='client.54157 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T14:21:54.068 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:21:53 smithi069 ceph-mon[88636]: from='client.44226 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T14:21:54.068 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:21:53 smithi069 ceph-mon[88636]: pgmap v41: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:21:54.068 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:21:53 smithi069 ceph-mon[88636]: Upgrade: Target is version 19.0.0-2778-ga971fd5f (squid) 2024-04-14T14:21:54.068 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:21:53 smithi069 ceph-mon[88636]: Upgrade: Target container is quay.ceph.io/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888, digests ['quay.ceph.io/ceph-ci/ceph@sha256:02f0a191d4d693952654c64f93cc9c0152602c0915251b3c94c2c4e32855c608'] 2024-04-14T14:21:54.068 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:21:53 smithi069 ceph-mon[88636]: Upgrade: Setting container_image for all mgr 2024-04-14T14:21:54.068 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:21:53 smithi069 ceph-mon[88636]: Upgrade: Setting container_image for all mon 2024-04-14T14:21:54.069 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:21:53 smithi069 ceph-mon[88636]: Upgrade: Setting container_image for all crash 2024-04-14T14:21:54.069 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:21:53 smithi069 ceph-mon[88636]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["0"], "max": 16}]: dispatch 2024-04-14T14:21:54.069 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:21:53 smithi069 ceph-mon[88636]: Upgrade: osd.0 is safe to restart 2024-04-14T14:21:54.069 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:21:53 smithi069 ceph-mon[88636]: from='client.? 172.21.15.69:0/4279883351' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T14:21:54.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:21:53 smithi122 ceph-mon[67998]: from='client.54157 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T14:21:54.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:21:53 smithi122 ceph-mon[67998]: from='client.44226 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T14:21:54.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:21:53 smithi122 ceph-mon[67998]: pgmap v41: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:21:54.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:21:53 smithi122 ceph-mon[67998]: Upgrade: Target is version 19.0.0-2778-ga971fd5f (squid) 2024-04-14T14:21:54.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:21:53 smithi122 ceph-mon[67998]: Upgrade: Target container is quay.ceph.io/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888, digests ['quay.ceph.io/ceph-ci/ceph@sha256:02f0a191d4d693952654c64f93cc9c0152602c0915251b3c94c2c4e32855c608'] 2024-04-14T14:21:54.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:21:53 smithi122 ceph-mon[67998]: Upgrade: Setting container_image for all mgr 2024-04-14T14:21:54.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:21:53 smithi122 ceph-mon[67998]: Upgrade: Setting container_image for all mon 2024-04-14T14:21:54.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:21:53 smithi122 ceph-mon[67998]: Upgrade: Setting container_image for all crash 2024-04-14T14:21:54.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:21:53 smithi122 ceph-mon[67998]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["0"], "max": 16}]: dispatch 2024-04-14T14:21:54.096 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:21:53 smithi122 ceph-mon[67998]: Upgrade: osd.0 is safe to restart 2024-04-14T14:21:54.096 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:21:53 smithi122 ceph-mon[67998]: from='client.? 172.21.15.69:0/4279883351' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T14:21:54.888 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:21:54 smithi069 ceph-mon[83680]: from='client.44238 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T14:21:54.888 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:21:54 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:21:54.888 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:21:54 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-04-14T14:21:54.888 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:21:54 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T14:21:54.889 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:21:54 smithi069 ceph-mon[88636]: from='client.44238 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T14:21:54.889 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:21:54 smithi069 ceph-mon[88636]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:21:54.889 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:21:54 smithi069 ceph-mon[88636]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-04-14T14:21:54.889 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:21:54 smithi069 ceph-mon[88636]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T14:21:55.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:21:54 smithi122 ceph-mon[67998]: from='client.44238 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T14:21:55.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:21:54 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:21:55.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:21:54 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-04-14T14:21:55.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:21:54 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T14:21:55.801 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:21:55 smithi069 ceph-mon[83680]: Upgrade: Updating osd.0 2024-04-14T14:21:55.801 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:21:55 smithi069 ceph-mon[83680]: Deploying daemon osd.0 on smithi069 2024-04-14T14:21:55.801 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:21:55 smithi069 ceph-mon[83680]: pgmap v42: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:21:55.801 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:21:55 smithi069 ceph-mon[88636]: Upgrade: Updating osd.0 2024-04-14T14:21:55.801 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:21:55 smithi069 ceph-mon[88636]: Deploying daemon osd.0 on smithi069 2024-04-14T14:21:55.802 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:21:55 smithi069 ceph-mon[88636]: pgmap v42: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:21:56.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:21:55 smithi122 ceph-mon[67998]: Upgrade: Updating osd.0 2024-04-14T14:21:56.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:21:55 smithi122 ceph-mon[67998]: Deploying daemon osd.0 on smithi069 2024-04-14T14:21:56.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:21:55 smithi122 ceph-mon[67998]: pgmap v42: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:21:56.792 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 14 14:21:56 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-y[63038]: ::ffff:172.21.15.122 - - [14/Apr/2024:14:21:56] "GET /metrics HTTP/1.1" 200 37954 "" "Prometheus/2.43.0" 2024-04-14T14:21:57.747 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:21:57 smithi069 ceph-mon[83680]: pgmap v43: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:21:57.747 INFO:journalctl@ceph.osd.0.smithi069.stdout:Apr 14 14:21:57 smithi069 systemd[1]: Stopping Ceph osd.0 for dc326fc2-fa65-11ee-bc8e-c7b262605968... 2024-04-14T14:21:57.748 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:21:57 smithi069 ceph-mon[88636]: pgmap v43: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:21:58.067 INFO:journalctl@ceph.osd.0.smithi069.stdout:Apr 14 14:21:57 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-osd-0[32331]: 2024-04-14T14:21:57.744+0000 7ffa4a783700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.0 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-14T14:21:58.067 INFO:journalctl@ceph.osd.0.smithi069.stdout:Apr 14 14:21:57 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-osd-0[32331]: 2024-04-14T14:21:57.744+0000 7ffa4a783700 -1 osd.0 111 *** Got signal Terminated *** 2024-04-14T14:21:58.067 INFO:journalctl@ceph.osd.0.smithi069.stdout:Apr 14 14:21:57 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-osd-0[32331]: 2024-04-14T14:21:57.744+0000 7ffa4a783700 -1 osd.0 111 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-14T14:21:58.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:21:57 smithi122 ceph-mon[67998]: pgmap v43: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:21:59.055 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:21:58 smithi069 ceph-mon[83680]: from='client.25417 -' entity='client.iscsi.foo.smithi069.yuumtr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T14:21:59.055 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:21:58 smithi069 ceph-mon[83680]: osd.0 marked itself down and dead 2024-04-14T14:21:59.055 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:21:58 smithi069 ceph-mon[88636]: from='client.25417 -' entity='client.iscsi.foo.smithi069.yuumtr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T14:21:59.055 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:21:58 smithi069 ceph-mon[88636]: osd.0 marked itself down and dead 2024-04-14T14:21:59.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:21:58 smithi122 ceph-mon[67998]: from='client.25417 -' entity='client.iscsi.foo.smithi069.yuumtr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T14:21:59.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:21:58 smithi122 ceph-mon[67998]: osd.0 marked itself down and dead 2024-04-14T14:21:59.317 INFO:journalctl@ceph.osd.0.smithi069.stdout:Apr 14 14:21:59 smithi069 podman[95270]: 2024-04-14 14:21:59.053081773 +0000 UTC m=+1.436475488 container died 46654654b6b6f679fdc41bd0474782f729dc20c9b40f257994fbc65c28529e70 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-osd-0, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, name=centos-stream, vendor=Red Hat, Inc., vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., build-date=2022-05-03T08:36:31.336870, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, io.buildah.version=1.19.8, architecture=x86_64, com.redhat.component=centos-stream-container, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, io.k8s.display-name=CentOS Stream 8, GIT_CLEAN=True, RELEASE=HEAD, version=8, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, maintainer=Guillaume Abrioux , ceph=True, CEPH_POINT_RELEASE=-17.2.0, vcs-type=git, distribution-scope=public, release=754, GIT_BRANCH=HEAD, io.openshift.expose-services=, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, GIT_REPO=https://github.com/ceph/ceph-container.git, io.openshift.tags=base centos centos-stream) 2024-04-14T14:21:59.654 INFO:journalctl@ceph.osd.0.smithi069.stdout:Apr 14 14:21:59 smithi069 podman[95270]: 2024-04-14 14:21:59.520104813 +0000 UTC m=+1.903498522 container cleanup 46654654b6b6f679fdc41bd0474782f729dc20c9b40f257994fbc65c28529e70 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-osd-0, CEPH_POINT_RELEASE=-17.2.0, io.openshift.tags=base centos centos-stream, vendor=Red Hat, Inc., GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, distribution-scope=public, version=8, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, ceph=True, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, io.openshift.expose-services=, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, architecture=x86_64, com.redhat.component=centos-stream-container, release=754, vcs-type=git, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., build-date=2022-05-03T08:36:31.336870, name=centos-stream, GIT_BRANCH=HEAD, RELEASE=HEAD, io.buildah.version=1.19.8, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, io.k8s.display-name=CentOS Stream 8, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_REPO=https://github.com/ceph/ceph-container.git) 2024-04-14T14:21:59.654 INFO:journalctl@ceph.osd.0.smithi069.stdout:Apr 14 14:21:59 smithi069 bash[95270]: ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-osd-0 2024-04-14T14:21:59.996 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:21:59 smithi122 ceph-mon[67998]: pgmap v44: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:21:59.996 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:21:59 smithi122 ceph-mon[67998]: Health check failed: 1 osds down (OSD_DOWN) 2024-04-14T14:21:59.996 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:21:59 smithi122 ceph-mon[67998]: osdmap e112: 8 total, 7 up, 8 in 2024-04-14T14:22:00.067 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:21:59 smithi069 ceph-mon[83680]: pgmap v44: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:22:00.067 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:21:59 smithi069 ceph-mon[83680]: Health check failed: 1 osds down (OSD_DOWN) 2024-04-14T14:22:00.067 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:21:59 smithi069 ceph-mon[83680]: osdmap e112: 8 total, 7 up, 8 in 2024-04-14T14:22:00.067 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:21:59 smithi069 ceph-mon[88636]: pgmap v44: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:22:00.067 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:21:59 smithi069 ceph-mon[88636]: Health check failed: 1 osds down (OSD_DOWN) 2024-04-14T14:22:00.067 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:21:59 smithi069 ceph-mon[88636]: osdmap e112: 8 total, 7 up, 8 in 2024-04-14T14:22:00.068 INFO:journalctl@ceph.osd.0.smithi069.stdout:Apr 14 14:21:59 smithi069 podman[95287]: 2024-04-14 14:21:59.788220732 +0000 UTC m=+0.731481242 container remove 46654654b6b6f679fdc41bd0474782f729dc20c9b40f257994fbc65c28529e70 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-osd-0, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=-17.2.0, architecture=x86_64, com.redhat.component=centos-stream-container, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, vendor=Red Hat, Inc., GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, maintainer=Guillaume Abrioux , com.redhat.license_terms=https://centos.org/legal/licensing-policy/, distribution-scope=public, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, io.openshift.expose-services=, version=8, build-date=2022-05-03T08:36:31.336870, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, name=centos-stream, io.openshift.tags=base centos centos-stream, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., GIT_REPO=https://github.com/ceph/ceph-container.git, io.k8s.display-name=CentOS Stream 8, release=754, io.buildah.version=1.19.8, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, GIT_CLEAN=True, RELEASE=HEAD, vcs-type=git, ceph=True) 2024-04-14T14:22:00.401 INFO:journalctl@ceph.osd.0.smithi069.stdout:Apr 14 14:22:00 smithi069 podman[95376]: 2024-04-14 14:22:00.041968838 +0000 UTC m=+0.018381237 image pull b96c509eef841460b3271f5fdd331860da87b004524f4ed59414aa3680183b53 quay.ceph.io/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888 2024-04-14T14:22:00.402 INFO:journalctl@ceph.osd.0.smithi069.stdout:Apr 14 14:22:00 smithi069 podman[95376]: 2024-04-14 14:22:00.180980249 +0000 UTC m=+0.157392687 container create 9fcf85433ac27f241368c42c13d4e382fcfe3745cd697075512eb64570aa8feb (image=quay.ceph.io/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888, name=ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-osd-0-deactivate, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, RELEASE=main-a971fd5, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, GIT_CLEAN=True, org.label-schema.build-date=20240403, ceph=True, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-14T14:22:00.659 INFO:journalctl@ceph.osd.0.smithi069.stdout:Apr 14 14:22:00 smithi069 podman[95376]: 2024-04-14 14:22:00.400277891 +0000 UTC m=+0.376690287 container init 9fcf85433ac27f241368c42c13d4e382fcfe3745cd697075512eb64570aa8feb (image=quay.ceph.io/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888, name=ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-osd-0-deactivate, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.build-date=20240403, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, RELEASE=main-a971fd5) 2024-04-14T14:22:00.659 INFO:journalctl@ceph.osd.0.smithi069.stdout:Apr 14 14:22:00 smithi069 podman[95376]: 2024-04-14 14:22:00.403808434 +0000 UTC m=+0.380220834 container start 9fcf85433ac27f241368c42c13d4e382fcfe3745cd697075512eb64570aa8feb (image=quay.ceph.io/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888, name=ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-osd-0-deactivate, org.label-schema.build-date=20240403, RELEASE=main-a971fd5, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, GIT_BRANCH=HEAD) 2024-04-14T14:22:00.659 INFO:journalctl@ceph.osd.0.smithi069.stdout:Apr 14 14:22:00 smithi069 podman[95376]: 2024-04-14 14:22:00.483645382 +0000 UTC m=+0.460057782 container attach 9fcf85433ac27f241368c42c13d4e382fcfe3745cd697075512eb64570aa8feb (image=quay.ceph.io/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888, name=ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-osd-0-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240403, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, GIT_CLEAN=True, RELEASE=main-a971fd5, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=) 2024-04-14T14:22:01.010 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:22:00 smithi122 ceph-mon[67998]: osdmap e113: 8 total, 7 up, 8 in 2024-04-14T14:22:01.010 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:22:00 smithi122 ceph-mon[67998]: pgmap v47: 161 pgs: 21 stale+active+clean, 140 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-04-14T14:22:01.038 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:22:00 smithi069 ceph-mon[83680]: osdmap e113: 8 total, 7 up, 8 in 2024-04-14T14:22:01.039 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:22:00 smithi069 ceph-mon[83680]: pgmap v47: 161 pgs: 21 stale+active+clean, 140 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-04-14T14:22:01.039 INFO:journalctl@ceph.osd.0.smithi069.stdout:Apr 14 14:22:00 smithi069 podman[95376]: 2024-04-14 14:22:00.697118835 +0000 UTC m=+0.673531232 container died 9fcf85433ac27f241368c42c13d4e382fcfe3745cd697075512eb64570aa8feb (image=quay.ceph.io/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888, name=ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-osd-0-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.schema-version=1.0, RELEASE=main-a971fd5, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.build-date=20240403) 2024-04-14T14:22:01.039 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:22:00 smithi069 ceph-mon[88636]: osdmap e113: 8 total, 7 up, 8 in 2024-04-14T14:22:01.039 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:22:00 smithi069 ceph-mon[88636]: pgmap v47: 161 pgs: 21 stale+active+clean, 140 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-04-14T14:22:01.303 INFO:journalctl@ceph.osd.0.smithi069.stdout:Apr 14 14:22:01 smithi069 podman[95402]: 2024-04-14 14:22:01.278642167 +0000 UTC m=+0.576386927 container remove 9fcf85433ac27f241368c42c13d4e382fcfe3745cd697075512eb64570aa8feb (image=quay.ceph.io/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888, name=ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-osd-0-deactivate, maintainer=Guillaume Abrioux , GIT_CLEAN=True, RELEASE=main-a971fd5, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240403, ceph=True) 2024-04-14T14:22:01.303 INFO:journalctl@ceph.osd.0.smithi069.stdout:Apr 14 14:22:01 smithi069 systemd[1]: ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968@osd.0.service: Deactivated successfully. 2024-04-14T14:22:01.303 INFO:journalctl@ceph.osd.0.smithi069.stdout:Apr 14 14:22:01 smithi069 systemd[1]: Stopped Ceph osd.0 for dc326fc2-fa65-11ee-bc8e-c7b262605968. 2024-04-14T14:22:01.304 INFO:journalctl@ceph.osd.0.smithi069.stdout:Apr 14 14:22:01 smithi069 systemd[1]: ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968@osd.0.service: Consumed 13.835s CPU time. 2024-04-14T14:22:01.567 INFO:journalctl@ceph.osd.0.smithi069.stdout:Apr 14 14:22:01 smithi069 systemd[1]: Starting Ceph osd.0 for dc326fc2-fa65-11ee-bc8e-c7b262605968... 2024-04-14T14:22:02.017 INFO:journalctl@ceph.osd.0.smithi069.stdout:Apr 14 14:22:01 smithi069 podman[95506]: 2024-04-14 14:22:01.713134652 +0000 UTC m=+0.109700775 container create 50ef0a3123318fcffd73c3a5205c6a20020802346d7ccd5bcb2219c672b6387e (image=quay.ceph.io/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888, name=ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-osd-0-activate, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, maintainer=Guillaume Abrioux , RELEASE=main-a971fd5, org.label-schema.build-date=20240403, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-14T14:22:02.017 INFO:journalctl@ceph.osd.0.smithi069.stdout:Apr 14 14:22:01 smithi069 podman[95506]: 2024-04-14 14:22:01.622593738 +0000 UTC m=+0.019159867 image pull b96c509eef841460b3271f5fdd331860da87b004524f4ed59414aa3680183b53 quay.ceph.io/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888 2024-04-14T14:22:02.017 INFO:journalctl@ceph.osd.0.smithi069.stdout:Apr 14 14:22:01 smithi069 podman[95506]: 2024-04-14 14:22:01.924178418 +0000 UTC m=+0.320744544 container init 50ef0a3123318fcffd73c3a5205c6a20020802346d7ccd5bcb2219c672b6387e (image=quay.ceph.io/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888, name=ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-osd-0-activate, GIT_CLEAN=True, RELEASE=main-a971fd5, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.build-date=20240403, io.buildah.version=1.33.5, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, ceph=True, maintainer=Guillaume Abrioux ) 2024-04-14T14:22:02.017 INFO:journalctl@ceph.osd.0.smithi069.stdout:Apr 14 14:22:01 smithi069 podman[95506]: 2024-04-14 14:22:01.928000111 +0000 UTC m=+0.324566243 container start 50ef0a3123318fcffd73c3a5205c6a20020802346d7ccd5bcb2219c672b6387e (image=quay.ceph.io/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888, name=ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-osd-0-activate, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=main-a971fd5, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_CLEAN=True, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240403, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0) 2024-04-14T14:22:02.018 INFO:journalctl@ceph.osd.0.smithi069.stdout:Apr 14 14:22:02 smithi069 podman[95506]: 2024-04-14 14:22:02.015873445 +0000 UTC m=+0.412439578 container attach 50ef0a3123318fcffd73c3a5205c6a20020802346d7ccd5bcb2219c672b6387e (image=quay.ceph.io/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888, name=ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-osd-0-activate, ceph=True, GIT_BRANCH=HEAD, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240403, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, RELEASE=main-a971fd5) 2024-04-14T14:22:02.629 INFO:journalctl@ceph.osd.0.smithi069.stdout:Apr 14 14:22:02 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-osd-0-activate[95528]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-04-14T14:22:02.629 INFO:journalctl@ceph.osd.0.smithi069.stdout:Apr 14 14:22:02 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-osd-0-activate[95528]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-0 --no-mon-config --dev /dev/mapper/vg_nvme-lv_4 2024-04-14T14:22:02.629 INFO:journalctl@ceph.osd.0.smithi069.stdout:Apr 14 14:22:02 smithi069 bash[95506]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-04-14T14:22:02.629 INFO:journalctl@ceph.osd.0.smithi069.stdout:Apr 14 14:22:02 smithi069 bash[95506]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-0 --no-mon-config --dev /dev/mapper/vg_nvme-lv_4 2024-04-14T14:22:02.630 INFO:journalctl@ceph.osd.0.smithi069.stdout:Apr 14 14:22:02 smithi069 bash[95506]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_4 2024-04-14T14:22:02.630 INFO:journalctl@ceph.osd.0.smithi069.stdout:Apr 14 14:22:02 smithi069 bash[95506]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-3 2024-04-14T14:22:02.630 INFO:journalctl@ceph.osd.0.smithi069.stdout:Apr 14 14:22:02 smithi069 bash[95506]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-0/block 2024-04-14T14:22:02.630 INFO:journalctl@ceph.osd.0.smithi069.stdout:Apr 14 14:22:02 smithi069 bash[95506]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-04-14T14:22:02.630 INFO:journalctl@ceph.osd.0.smithi069.stdout:Apr 14 14:22:02 smithi069 bash[95506]: --> ceph-volume raw activate successful for osd ID: 0 2024-04-14T14:22:02.630 INFO:journalctl@ceph.osd.0.smithi069.stdout:Apr 14 14:22:02 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-osd-0-activate[95528]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_4 2024-04-14T14:22:02.630 INFO:journalctl@ceph.osd.0.smithi069.stdout:Apr 14 14:22:02 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-osd-0-activate[95528]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-3 2024-04-14T14:22:02.630 INFO:journalctl@ceph.osd.0.smithi069.stdout:Apr 14 14:22:02 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-osd-0-activate[95528]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-0/block 2024-04-14T14:22:02.630 INFO:journalctl@ceph.osd.0.smithi069.stdout:Apr 14 14:22:02 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-osd-0-activate[95528]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-04-14T14:22:02.630 INFO:journalctl@ceph.osd.0.smithi069.stdout:Apr 14 14:22:02 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-osd-0-activate[95528]: --> ceph-volume raw activate successful for osd ID: 0 2024-04-14T14:22:02.630 INFO:journalctl@ceph.osd.0.smithi069.stdout:Apr 14 14:22:02 smithi069 podman[95506]: 2024-04-14 14:22:02.399119273 +0000 UTC m=+0.795685400 container died 50ef0a3123318fcffd73c3a5205c6a20020802346d7ccd5bcb2219c672b6387e (image=quay.ceph.io/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888, name=ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-osd-0-activate, org.label-schema.build-date=20240403, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_CLEAN=True, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, RELEASE=main-a971fd5, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=) 2024-04-14T14:22:02.987 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:22:02 smithi069 ceph-mon[83680]: Health check failed: Degraded data redundancy: 71/627 objects degraded (11.324%), 22 pgs degraded (PG_DEGRADED) 2024-04-14T14:22:02.987 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:22:02 smithi069 ceph-mon[88636]: Health check failed: Degraded data redundancy: 71/627 objects degraded (11.324%), 22 pgs degraded (PG_DEGRADED) 2024-04-14T14:22:03.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:22:02 smithi122 ceph-mon[67998]: Health check failed: Degraded data redundancy: 71/627 objects degraded (11.324%), 22 pgs degraded (PG_DEGRADED) 2024-04-14T14:22:03.317 INFO:journalctl@ceph.osd.0.smithi069.stdout:Apr 14 14:22:02 smithi069 podman[95651]: 2024-04-14 14:22:02.986109393 +0000 UTC m=+0.581300233 container remove 50ef0a3123318fcffd73c3a5205c6a20020802346d7ccd5bcb2219c672b6387e (image=quay.ceph.io/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888, name=ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-osd-0-activate, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, RELEASE=main-a971fd5, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240403, GIT_CLEAN=True, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-14T14:22:03.317 INFO:journalctl@ceph.osd.0.smithi069.stdout:Apr 14 14:22:03 smithi069 podman[95718]: 2024-04-14 14:22:03.101594353 +0000 UTC m=+0.018140874 image pull b96c509eef841460b3271f5fdd331860da87b004524f4ed59414aa3680183b53 quay.ceph.io/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888 2024-04-14T14:22:03.317 INFO:journalctl@ceph.osd.0.smithi069.stdout:Apr 14 14:22:03 smithi069 podman[95718]: 2024-04-14 14:22:03.203650789 +0000 UTC m=+0.120197300 container create a0e57e3fb494794d3a44191703521befbaa302aea1f47acc3e8bf914481c9af9 (image=quay.ceph.io/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888, name=ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-osd-0, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_CLEAN=True, RELEASE=main-a971fd5, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.build-date=20240403, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2) 2024-04-14T14:22:03.636 INFO:journalctl@ceph.osd.0.smithi069.stdout:Apr 14 14:22:03 smithi069 podman[95718]: 2024-04-14 14:22:03.44807366 +0000 UTC m=+0.364620181 container init a0e57e3fb494794d3a44191703521befbaa302aea1f47acc3e8bf914481c9af9 (image=quay.ceph.io/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888, name=ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-osd-0, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, RELEASE=main-a971fd5, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , ceph=True, CEPH_POINT_RELEASE=, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.build-date=20240403, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-14T14:22:03.636 INFO:journalctl@ceph.osd.0.smithi069.stdout:Apr 14 14:22:03 smithi069 podman[95718]: 2024-04-14 14:22:03.451186302 +0000 UTC m=+0.367732812 container start a0e57e3fb494794d3a44191703521befbaa302aea1f47acc3e8bf914481c9af9 (image=quay.ceph.io/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888, name=ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-osd-0, RELEASE=main-a971fd5, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.build-date=20240403, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, ceph=True, CEPH_POINT_RELEASE=) 2024-04-14T14:22:03.636 INFO:journalctl@ceph.osd.0.smithi069.stdout:Apr 14 14:22:03 smithi069 ceph-osd[95737]: -- 172.21.15.69:0/2621597458 <== mon.0 v2:172.21.15.69:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x5606fdd37680 con 0x5606fdd00c00 2024-04-14T14:22:03.636 INFO:journalctl@ceph.osd.0.smithi069.stdout:Apr 14 14:22:03 smithi069 bash[95718]: a0e57e3fb494794d3a44191703521befbaa302aea1f47acc3e8bf914481c9af9 2024-04-14T14:22:03.636 INFO:journalctl@ceph.osd.0.smithi069.stdout:Apr 14 14:22:03 smithi069 systemd[1]: Started Ceph osd.0 for dc326fc2-fa65-11ee-bc8e-c7b262605968. 2024-04-14T14:22:04.002 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:22:03 smithi069 ceph-mon[83680]: pgmap v48: 161 pgs: 22 active+undersized+degraded, 36 active+undersized, 103 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 71/627 objects degraded (11.324%) 2024-04-14T14:22:04.003 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:22:03 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:22:04.003 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:22:03 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:22:04.003 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:22:03 smithi069 ceph-mon[88636]: pgmap v48: 161 pgs: 22 active+undersized+degraded, 36 active+undersized, 103 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 71/627 objects degraded (11.324%) 2024-04-14T14:22:04.003 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:22:03 smithi069 ceph-mon[88636]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:22:04.003 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:22:03 smithi069 ceph-mon[88636]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:22:04.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:22:03 smithi122 ceph-mon[67998]: pgmap v48: 161 pgs: 22 active+undersized+degraded, 36 active+undersized, 103 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 71/627 objects degraded (11.324%) 2024-04-14T14:22:04.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:22:03 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:22:04.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:22:03 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:22:04.317 INFO:journalctl@ceph.osd.0.smithi069.stdout:Apr 14 14:22:04 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-osd-0[95733]: 2024-04-14T14:22:04.000+0000 7fef04fa47c0 -1 Falling back to public interface 2024-04-14T14:22:04.817 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:22:04 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:22:04.817 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:22:04 smithi069 ceph-mon[88636]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:22:05.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:22:04 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:22:06.007 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:22:05 smithi069 ceph-mon[83680]: pgmap v49: 161 pgs: 22 active+undersized+degraded, 36 active+undersized, 103 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 71/627 objects degraded (11.324%) 2024-04-14T14:22:06.008 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:22:05 smithi069 ceph-mon[88636]: pgmap v49: 161 pgs: 22 active+undersized+degraded, 36 active+undersized, 103 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 71/627 objects degraded (11.324%) 2024-04-14T14:22:06.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:22:05 smithi122 ceph-mon[67998]: pgmap v49: 161 pgs: 22 active+undersized+degraded, 36 active+undersized, 103 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 71/627 objects degraded (11.324%) 2024-04-14T14:22:06.760 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 14 14:22:06 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-y[63038]: ::ffff:172.21.15.122 - - [14/Apr/2024:14:22:06] "GET /metrics HTTP/1.1" 200 37954 "" "Prometheus/2.43.0" 2024-04-14T14:22:06.760 INFO:journalctl@ceph.osd.0.smithi069.stdout:Apr 14 14:22:06 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-osd-0[95733]: 2024-04-14T14:22:06.726+0000 7fef04fa47c0 -1 osd.0 0 read_superblock omap replica is missing. 2024-04-14T14:22:07.066 INFO:journalctl@ceph.osd.0.smithi069.stdout:Apr 14 14:22:06 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-osd-0[95733]: 2024-04-14T14:22:06.757+0000 7fef04fa47c0 -1 osd.0 111 log_to_monitors true 2024-04-14T14:22:07.817 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:22:07 smithi069 ceph-mon[83680]: pgmap v50: 161 pgs: 22 active+undersized+degraded, 36 active+undersized, 103 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 71/627 objects degraded (11.324%) 2024-04-14T14:22:07.817 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:22:07 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:22:07.817 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:22:07 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T14:22:07.817 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:22:07 smithi069 ceph-mon[83680]: from='osd.0 [v2:172.21.15.69:6802/3545396991,v1:172.21.15.69:6803/3545396991]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-04-14T14:22:07.817 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:22:07 smithi069 ceph-mon[83680]: from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-04-14T14:22:07.818 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:22:07 smithi069 ceph-mon[88636]: pgmap v50: 161 pgs: 22 active+undersized+degraded, 36 active+undersized, 103 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 71/627 objects degraded (11.324%) 2024-04-14T14:22:07.818 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:22:07 smithi069 ceph-mon[88636]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:22:07.818 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:22:07 smithi069 ceph-mon[88636]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T14:22:07.818 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:22:07 smithi069 ceph-mon[88636]: from='osd.0 [v2:172.21.15.69:6802/3545396991,v1:172.21.15.69:6803/3545396991]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-04-14T14:22:07.818 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:22:07 smithi069 ceph-mon[88636]: from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-04-14T14:22:07.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:22:07 smithi122 ceph-mon[67998]: pgmap v50: 161 pgs: 22 active+undersized+degraded, 36 active+undersized, 103 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 71/627 objects degraded (11.324%) 2024-04-14T14:22:07.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:22:07 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:22:07.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:22:07 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T14:22:07.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:22:07 smithi122 ceph-mon[67998]: from='osd.0 [v2:172.21.15.69:6802/3545396991,v1:172.21.15.69:6803/3545396991]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-04-14T14:22:07.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:22:07 smithi122 ceph-mon[67998]: from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-04-14T14:22:08.567 INFO:journalctl@ceph.osd.0.smithi069.stdout:Apr 14 14:22:08 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-osd-0[95733]: 2024-04-14T14:22:08.124+0000 7feef23f4700 -1 osd.0 111 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-14T14:22:09.067 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:22:08 smithi069 ceph-mon[83680]: from='client.25417 -' entity='client.iscsi.foo.smithi069.yuumtr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T14:22:09.067 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:22:08 smithi069 ceph-mon[83680]: from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-04-14T14:22:09.067 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:22:08 smithi069 ceph-mon[83680]: osdmap e114: 8 total, 7 up, 8 in 2024-04-14T14:22:09.067 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:22:08 smithi069 ceph-mon[83680]: from='osd.0 [v2:172.21.15.69:6802/3545396991,v1:172.21.15.69:6803/3545396991]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi069", "root=default"]}]: dispatch 2024-04-14T14:22:09.067 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:22:08 smithi069 ceph-mon[83680]: from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi069", "root=default"]}]: dispatch 2024-04-14T14:22:09.067 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:22:08 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:22:09.067 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:22:08 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:22:09.067 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:22:08 smithi069 ceph-mon[83680]: from='osd.0 ' entity='osd.0' 2024-04-14T14:22:09.068 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:22:08 smithi069 ceph-mon[88636]: from='client.25417 -' entity='client.iscsi.foo.smithi069.yuumtr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T14:22:09.068 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:22:08 smithi069 ceph-mon[88636]: from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-04-14T14:22:09.068 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:22:08 smithi069 ceph-mon[88636]: osdmap e114: 8 total, 7 up, 8 in 2024-04-14T14:22:09.068 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:22:08 smithi069 ceph-mon[88636]: from='osd.0 [v2:172.21.15.69:6802/3545396991,v1:172.21.15.69:6803/3545396991]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi069", "root=default"]}]: dispatch 2024-04-14T14:22:09.068 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:22:08 smithi069 ceph-mon[88636]: from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi069", "root=default"]}]: dispatch 2024-04-14T14:22:09.068 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:22:08 smithi069 ceph-mon[88636]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:22:09.068 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:22:08 smithi069 ceph-mon[88636]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:22:09.068 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:22:08 smithi069 ceph-mon[88636]: from='osd.0 ' entity='osd.0' 2024-04-14T14:22:09.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:22:08 smithi122 ceph-mon[67998]: from='client.25417 -' entity='client.iscsi.foo.smithi069.yuumtr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T14:22:09.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:22:08 smithi122 ceph-mon[67998]: from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-04-14T14:22:09.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:22:08 smithi122 ceph-mon[67998]: osdmap e114: 8 total, 7 up, 8 in 2024-04-14T14:22:09.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:22:08 smithi122 ceph-mon[67998]: from='osd.0 [v2:172.21.15.69:6802/3545396991,v1:172.21.15.69:6803/3545396991]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi069", "root=default"]}]: dispatch 2024-04-14T14:22:09.096 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:22:08 smithi122 ceph-mon[67998]: from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi069", "root=default"]}]: dispatch 2024-04-14T14:22:09.096 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:22:08 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:22:09.096 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:22:08 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:22:09.096 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:22:08 smithi122 ceph-mon[67998]: from='osd.0 ' entity='osd.0' 2024-04-14T14:22:09.965 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:22:09 smithi069 ceph-mon[83680]: pgmap v52: 161 pgs: 22 active+undersized+degraded, 36 active+undersized, 103 active+clean; 585 KiB data, 114 MiB used, 715 GiB / 715 GiB avail; 71/627 objects degraded (11.324%) 2024-04-14T14:22:09.965 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:22:09 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:22:09.965 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:22:09 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:22:09.965 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:22:09 smithi069 ceph-mon[83680]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-04-14T14:22:09.965 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:22:09 smithi069 ceph-mon[83680]: osd.0 [v2:172.21.15.69:6802/3545396991,v1:172.21.15.69:6803/3545396991] boot 2024-04-14T14:22:09.965 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:22:09 smithi069 ceph-mon[83680]: osdmap e115: 8 total, 8 up, 8 in 2024-04-14T14:22:09.965 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:22:09 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-14T14:22:09.966 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:22:09 smithi069 ceph-mon[88636]: pgmap v52: 161 pgs: 22 active+undersized+degraded, 36 active+undersized, 103 active+clean; 585 KiB data, 114 MiB used, 715 GiB / 715 GiB avail; 71/627 objects degraded (11.324%) 2024-04-14T14:22:09.966 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:22:09 smithi069 ceph-mon[88636]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:22:09.966 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:22:09 smithi069 ceph-mon[88636]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:22:09.966 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:22:09 smithi069 ceph-mon[88636]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-04-14T14:22:09.966 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:22:09 smithi069 ceph-mon[88636]: osd.0 [v2:172.21.15.69:6802/3545396991,v1:172.21.15.69:6803/3545396991] boot 2024-04-14T14:22:09.966 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:22:09 smithi069 ceph-mon[88636]: osdmap e115: 8 total, 8 up, 8 in 2024-04-14T14:22:09.966 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:22:09 smithi069 ceph-mon[88636]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-14T14:22:10.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:22:09 smithi122 ceph-mon[67998]: pgmap v52: 161 pgs: 22 active+undersized+degraded, 36 active+undersized, 103 active+clean; 585 KiB data, 114 MiB used, 715 GiB / 715 GiB avail; 71/627 objects degraded (11.324%) 2024-04-14T14:22:10.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:22:09 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:22:10.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:22:09 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:22:10.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:22:09 smithi122 ceph-mon[67998]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-04-14T14:22:10.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:22:09 smithi122 ceph-mon[67998]: osd.0 [v2:172.21.15.69:6802/3545396991,v1:172.21.15.69:6803/3545396991] boot 2024-04-14T14:22:10.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:22:09 smithi122 ceph-mon[67998]: osdmap e115: 8 total, 8 up, 8 in 2024-04-14T14:22:10.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:22:09 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-14T14:22:11.262 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:22:11 smithi069 ceph-mon[83680]: osdmap e116: 8 total, 8 up, 8 in 2024-04-14T14:22:11.262 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:22:11 smithi069 ceph-mon[83680]: pgmap v55: 161 pgs: 22 active+undersized+degraded, 36 active+undersized, 103 active+clean; 585 KiB data, 114 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 71/627 objects degraded (11.324%) 2024-04-14T14:22:11.262 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:22:11 smithi069 ceph-mon[88636]: osdmap e116: 8 total, 8 up, 8 in 2024-04-14T14:22:11.263 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:22:11 smithi069 ceph-mon[88636]: pgmap v55: 161 pgs: 22 active+undersized+degraded, 36 active+undersized, 103 active+clean; 585 KiB data, 114 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 71/627 objects degraded (11.324%) 2024-04-14T14:22:11.595 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:22:11 smithi122 ceph-mon[67998]: osdmap e116: 8 total, 8 up, 8 in 2024-04-14T14:22:11.595 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:22:11 smithi122 ceph-mon[67998]: pgmap v55: 161 pgs: 22 active+undersized+degraded, 36 active+undersized, 103 active+clean; 585 KiB data, 114 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 71/627 objects degraded (11.324%) 2024-04-14T14:22:12.649 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:22:12 smithi069 ceph-mon[88636]: Health check update: Degraded data redundancy: 31/627 objects degraded (4.944%), 9 pgs degraded (PG_DEGRADED) 2024-04-14T14:22:12.650 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:22:12 smithi069 ceph-mon[83680]: Health check update: Degraded data redundancy: 31/627 objects degraded (4.944%), 9 pgs degraded (PG_DEGRADED) 2024-04-14T14:22:13.067 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 14 14:22:12 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-y[63038]: 2024-04-14T14:22:12.838+0000 7fdd38467700 -1 mgr.server reply reply (16) Device or resource busy unsafe to stop osd(s) at this time (4 PGs are or would become offline) 2024-04-14T14:22:13.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:22:12 smithi122 ceph-mon[67998]: Health check update: Degraded data redundancy: 31/627 objects degraded (4.944%), 9 pgs degraded (PG_DEGRADED) 2024-04-14T14:22:14.067 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:22:13 smithi069 ceph-mon[83680]: pgmap v56: 161 pgs: 9 active+undersized+degraded, 12 active+undersized, 140 active+clean; 585 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s; 31/627 objects degraded (4.944%) 2024-04-14T14:22:14.068 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:22:13 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:22:14.068 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:22:13 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:22:14.068 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:22:13 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T14:22:14.068 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:22:13 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-14T14:22:14.068 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:22:13 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:22:14.068 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:22:13 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-14T14:22:14.068 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:22:13 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T14:22:14.068 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:22:13 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T14:22:14.068 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:22:13 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T14:22:14.068 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:22:13 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2024-04-14T14:22:14.069 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:22:13 smithi069 ceph-mon[88636]: pgmap v56: 161 pgs: 9 active+undersized+degraded, 12 active+undersized, 140 active+clean; 585 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s; 31/627 objects degraded (4.944%) 2024-04-14T14:22:14.069 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:22:13 smithi069 ceph-mon[88636]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:22:14.069 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:22:13 smithi069 ceph-mon[88636]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:22:14.069 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:22:13 smithi069 ceph-mon[88636]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T14:22:14.069 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:22:13 smithi069 ceph-mon[88636]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-14T14:22:14.069 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:22:13 smithi069 ceph-mon[88636]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:22:14.069 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:22:13 smithi069 ceph-mon[88636]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-14T14:22:14.069 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:22:13 smithi069 ceph-mon[88636]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T14:22:14.069 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:22:13 smithi069 ceph-mon[88636]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T14:22:14.069 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:22:13 smithi069 ceph-mon[88636]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T14:22:14.069 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:22:13 smithi069 ceph-mon[88636]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2024-04-14T14:22:14.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:22:13 smithi122 ceph-mon[67998]: pgmap v56: 161 pgs: 9 active+undersized+degraded, 12 active+undersized, 140 active+clean; 585 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s; 31/627 objects degraded (4.944%) 2024-04-14T14:22:14.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:22:13 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:22:14.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:22:13 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:22:14.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:22:13 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T14:22:14.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:22:13 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-14T14:22:14.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:22:13 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:22:14.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:22:13 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-14T14:22:14.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:22:13 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T14:22:14.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:22:13 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T14:22:14.096 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:22:13 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T14:22:14.096 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:22:13 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2024-04-14T14:22:15.067 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:22:14 smithi069 ceph-mon[83680]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2024-04-14T14:22:15.067 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:22:14 smithi069 ceph-mon[83680]: Upgrade: unsafe to stop osd(s) at this time (4 PGs are or would become offline) 2024-04-14T14:22:15.067 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:22:14 smithi069 ceph-mon[88636]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2024-04-14T14:22:15.067 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:22:14 smithi069 ceph-mon[88636]: Upgrade: unsafe to stop osd(s) at this time (4 PGs are or would become offline) 2024-04-14T14:22:15.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:22:14 smithi122 ceph-mon[67998]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2024-04-14T14:22:15.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:22:14 smithi122 ceph-mon[67998]: Upgrade: unsafe to stop osd(s) at this time (4 PGs are or would become offline) 2024-04-14T14:22:16.067 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:22:15 smithi069 ceph-mon[83680]: pgmap v57: 161 pgs: 161 active+clean; 585 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 744 B/s rd, 0 op/s 2024-04-14T14:22:16.067 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:22:15 smithi069 ceph-mon[83680]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 31/627 objects degraded (4.944%), 9 pgs degraded) 2024-04-14T14:22:16.067 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:22:15 smithi069 ceph-mon[83680]: Cluster is now healthy 2024-04-14T14:22:16.067 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:22:15 smithi069 ceph-mon[88636]: pgmap v57: 161 pgs: 161 active+clean; 585 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 744 B/s rd, 0 op/s 2024-04-14T14:22:16.067 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:22:15 smithi069 ceph-mon[88636]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 31/627 objects degraded (4.944%), 9 pgs degraded) 2024-04-14T14:22:16.067 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:22:15 smithi069 ceph-mon[88636]: Cluster is now healthy 2024-04-14T14:22:16.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:22:15 smithi122 ceph-mon[67998]: pgmap v57: 161 pgs: 161 active+clean; 585 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 744 B/s rd, 0 op/s 2024-04-14T14:22:16.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:22:15 smithi122 ceph-mon[67998]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 31/627 objects degraded (4.944%), 9 pgs degraded) 2024-04-14T14:22:16.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:22:15 smithi122 ceph-mon[67998]: Cluster is now healthy 2024-04-14T14:22:16.817 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 14 14:22:16 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-y[63038]: ::ffff:172.21.15.122 - - [14/Apr/2024:14:22:16] "GET /metrics HTTP/1.1" 200 38151 "" "Prometheus/2.43.0" 2024-04-14T14:22:18.067 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:22:17 smithi069 ceph-mon[83680]: pgmap v58: 161 pgs: 161 active+clean; 585 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-04-14T14:22:18.067 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:22:17 smithi069 ceph-mon[88636]: pgmap v58: 161 pgs: 161 active+clean; 585 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-04-14T14:22:18.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:22:17 smithi122 ceph-mon[67998]: pgmap v58: 161 pgs: 161 active+clean; 585 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-04-14T14:22:18.969 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:22:18 smithi069 ceph-mon[83680]: from='client.25417 -' entity='client.iscsi.foo.smithi069.yuumtr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T14:22:18.969 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:22:18 smithi069 ceph-mon[88636]: from='client.25417 -' entity='client.iscsi.foo.smithi069.yuumtr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T14:22:19.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:22:18 smithi122 ceph-mon[67998]: from='client.25417 -' entity='client.iscsi.foo.smithi069.yuumtr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T14:22:19.980 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:22:19 smithi069 ceph-mon[83680]: pgmap v59: 161 pgs: 161 active+clean; 585 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-04-14T14:22:19.980 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:22:19 smithi069 ceph-mon[88636]: pgmap v59: 161 pgs: 161 active+clean; 585 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-04-14T14:22:20.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:22:19 smithi122 ceph-mon[67998]: pgmap v59: 161 pgs: 161 active+clean; 585 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-04-14T14:22:22.067 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:22:21 smithi069 ceph-mon[83680]: pgmap v60: 161 pgs: 161 active+clean; 585 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 984 B/s rd, 0 op/s 2024-04-14T14:22:22.067 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:22:21 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:22:22.067 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:22:21 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T14:22:22.067 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:22:21 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:22:22.068 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:22:21 smithi069 ceph-mon[88636]: pgmap v60: 161 pgs: 161 active+clean; 585 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 984 B/s rd, 0 op/s 2024-04-14T14:22:22.068 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:22:21 smithi069 ceph-mon[88636]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:22:22.068 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:22:21 smithi069 ceph-mon[88636]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T14:22:22.068 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:22:21 smithi069 ceph-mon[88636]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:22:22.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:22:21 smithi122 ceph-mon[67998]: pgmap v60: 161 pgs: 161 active+clean; 585 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 984 B/s rd, 0 op/s 2024-04-14T14:22:22.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:22:21 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:22:22.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:22:21 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T14:22:22.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:22:21 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:22:23.067 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:22:22 smithi069 ceph-mon[83680]: pgmap v61: 161 pgs: 161 active+clean; 585 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:22:23.067 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:22:22 smithi069 ceph-mon[88636]: pgmap v61: 161 pgs: 161 active+clean; 585 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:22:23.094 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:22:22 smithi122 ceph-mon[67998]: pgmap v61: 161 pgs: 161 active+clean; 585 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:22:23.436 INFO:teuthology.orchestra.run.smithi069.stdout:true 2024-04-14T14:22:23.817 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:22:23 smithi069 ceph-mon[83680]: from='client.44250 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T14:22:23.817 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:22:23 smithi069 ceph-mon[88636]: from='client.44250 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T14:22:24.092 INFO:teuthology.orchestra.run.smithi069.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-14T14:22:24.092 INFO:teuthology.orchestra.run.smithi069.stdout:alertmanager.a smithi069 *:9093,9094 running (4m) 16s ago 25m 17.4M - 0.25.0 c8568f914cd2 7524f106e7bd 2024-04-14T14:22:24.092 INFO:teuthology.orchestra.run.smithi069.stdout:grafana.a smithi122 *:3000 running (3m) 102s ago 25m 81.0M - 9.4.12 2bacad6d85d8 5685178e3eba 2024-04-14T14:22:24.092 INFO:teuthology.orchestra.run.smithi069.stdout:iscsi.foo.smithi069.yuumtr smithi069 running (5m) 16s ago 25m 59.3M - 3.5 e1d6a67b021e 38cc81ea8cfa 2024-04-14T14:22:24.092 INFO:teuthology.orchestra.run.smithi069.stdout:mgr.x smithi122 *:8443,9283,8765 running (5m) 102s ago 30m 450M - 19.0.0-2778-ga971fd5f b96c509eef84 e29977590b43 2024-04-14T14:22:24.092 INFO:teuthology.orchestra.run.smithi069.stdout:mgr.y smithi069 *:8443,9283,8765 running (14m) 16s ago 31m 499M - 19.0.0-2778-ga971fd5f b96c509eef84 7af3d29d0e5f 2024-04-14T14:22:24.092 INFO:teuthology.orchestra.run.smithi069.stdout:mon.a smithi069 running (116s) 16s ago 32m 47.2M 2048M 19.0.0-2778-ga971fd5f b96c509eef84 8bcd6b3d3846 2024-04-14T14:22:24.092 INFO:teuthology.orchestra.run.smithi069.stdout:mon.b smithi122 running (2m) 102s ago 30m 33.9M 2048M 19.0.0-2778-ga971fd5f b96c509eef84 05f437a1a9a1 2024-04-14T14:22:24.092 INFO:teuthology.orchestra.run.smithi069.stdout:mon.c smithi069 running (92s) 16s ago 30m 31.9M 2048M 19.0.0-2778-ga971fd5f b96c509eef84 84aa95f11d0c 2024-04-14T14:22:24.092 INFO:teuthology.orchestra.run.smithi069.stdout:node-exporter.a smithi069 *:9100 running (4m) 16s ago 26m 8686k - 1.5.0 0da6a335fe13 0de0eb2171b9 2024-04-14T14:22:24.092 INFO:teuthology.orchestra.run.smithi069.stdout:node-exporter.b smithi122 *:9100 running (4m) 102s ago 26m 16.1M - 1.5.0 0da6a335fe13 67877bf02d8e 2024-04-14T14:22:24.092 INFO:teuthology.orchestra.run.smithi069.stdout:osd.0 smithi069 running (20s) 16s ago 30m 11.6M 2992M 19.0.0-2778-ga971fd5f b96c509eef84 a0e57e3fb494 2024-04-14T14:22:24.093 INFO:teuthology.orchestra.run.smithi069.stdout:osd.1 smithi069 running (29m) 16s ago 29m 55.6M 2992M 17.2.0 e1d6a67b021e cdba54a59609 2024-04-14T14:22:24.093 INFO:teuthology.orchestra.run.smithi069.stdout:osd.2 smithi069 running (29m) 16s ago 29m 51.6M 2992M 17.2.0 e1d6a67b021e bc37a1a9cba0 2024-04-14T14:22:24.093 INFO:teuthology.orchestra.run.smithi069.stdout:osd.3 smithi069 running (28m) 16s ago 28m 56.2M 2992M 17.2.0 e1d6a67b021e 0925ba3da801 2024-04-14T14:22:24.093 INFO:teuthology.orchestra.run.smithi069.stdout:osd.4 smithi122 running (28m) 102s ago 28m 56.1M 3248M 17.2.0 e1d6a67b021e c00975caf771 2024-04-14T14:22:24.093 INFO:teuthology.orchestra.run.smithi069.stdout:osd.5 smithi122 running (28m) 102s ago 28m 55.2M 3248M 17.2.0 e1d6a67b021e 5f52d217ceb8 2024-04-14T14:22:24.093 INFO:teuthology.orchestra.run.smithi069.stdout:osd.6 smithi122 running (27m) 102s ago 27m 52.4M 3248M 17.2.0 e1d6a67b021e a2acfec61475 2024-04-14T14:22:24.093 INFO:teuthology.orchestra.run.smithi069.stdout:osd.7 smithi122 running (27m) 102s ago 27m 57.1M 3248M 17.2.0 e1d6a67b021e 62411cd06801 2024-04-14T14:22:24.093 INFO:teuthology.orchestra.run.smithi069.stdout:prometheus.a smithi122 *:9095 running (4m) 102s ago 26m 66.4M - 2.43.0 a07b618ecd1d 5c1f41d3bbe6 2024-04-14T14:22:24.093 INFO:teuthology.orchestra.run.smithi069.stdout:rgw.foo.smithi069.wesedz smithi069 *:8000 running (25m) 16s ago 25m 79.1M - 17.2.0 e1d6a67b021e 16de491440f3 2024-04-14T14:22:24.093 INFO:teuthology.orchestra.run.smithi069.stdout:rgw.foo.smithi122.jiobyv smithi122 *:8000 running (25m) 102s ago 25m 75.8M - 17.2.0 e1d6a67b021e 6d16de445dba 2024-04-14T14:22:24.094 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:22:23 smithi122 ceph-mon[67998]: from='client.44250 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T14:22:24.488 INFO:teuthology.orchestra.run.smithi069.stdout:{ 2024-04-14T14:22:24.488 INFO:teuthology.orchestra.run.smithi069.stdout: "mon": { 2024-04-14T14:22:24.488 INFO:teuthology.orchestra.run.smithi069.stdout: "ceph version 19.0.0-2778-ga971fd5f (a971fd5f5719038c70b817bafacd6d5400b1d888) squid (dev)": 3 2024-04-14T14:22:24.488 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-14T14:22:24.488 INFO:teuthology.orchestra.run.smithi069.stdout: "mgr": { 2024-04-14T14:22:24.488 INFO:teuthology.orchestra.run.smithi069.stdout: "ceph version 19.0.0-2778-ga971fd5f (a971fd5f5719038c70b817bafacd6d5400b1d888) squid (dev)": 2 2024-04-14T14:22:24.488 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-14T14:22:24.488 INFO:teuthology.orchestra.run.smithi069.stdout: "osd": { 2024-04-14T14:22:24.488 INFO:teuthology.orchestra.run.smithi069.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 7, 2024-04-14T14:22:24.489 INFO:teuthology.orchestra.run.smithi069.stdout: "ceph version 19.0.0-2778-ga971fd5f (a971fd5f5719038c70b817bafacd6d5400b1d888) squid (dev)": 1 2024-04-14T14:22:24.489 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-14T14:22:24.489 INFO:teuthology.orchestra.run.smithi069.stdout: "rgw": { 2024-04-14T14:22:24.489 INFO:teuthology.orchestra.run.smithi069.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-04-14T14:22:24.489 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-14T14:22:24.489 INFO:teuthology.orchestra.run.smithi069.stdout: "overall": { 2024-04-14T14:22:24.489 INFO:teuthology.orchestra.run.smithi069.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 9, 2024-04-14T14:22:24.489 INFO:teuthology.orchestra.run.smithi069.stdout: "ceph version 19.0.0-2778-ga971fd5f (a971fd5f5719038c70b817bafacd6d5400b1d888) squid (dev)": 6 2024-04-14T14:22:24.489 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-14T14:22:24.489 INFO:teuthology.orchestra.run.smithi069.stdout:} 2024-04-14T14:22:24.817 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:22:24 smithi069 ceph-mon[88636]: from='client.54193 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T14:22:24.817 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:22:24 smithi069 ceph-mon[88636]: from='client.44262 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T14:22:24.817 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:22:24 smithi069 ceph-mon[88636]: from='client.? 172.21.15.69:0/1132436637' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T14:22:24.817 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:22:24 smithi069 ceph-mon[88636]: pgmap v62: 161 pgs: 161 active+clean; 585 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:22:24.817 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:22:24 smithi069 ceph-mon[83680]: from='client.54193 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T14:22:24.817 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:22:24 smithi069 ceph-mon[83680]: from='client.44262 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T14:22:24.817 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:22:24 smithi069 ceph-mon[83680]: from='client.? 172.21.15.69:0/1132436637' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T14:22:24.818 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:22:24 smithi069 ceph-mon[83680]: pgmap v62: 161 pgs: 161 active+clean; 585 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:22:24.821 INFO:teuthology.orchestra.run.smithi069.stdout:{ 2024-04-14T14:22:24.821 INFO:teuthology.orchestra.run.smithi069.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888", 2024-04-14T14:22:24.822 INFO:teuthology.orchestra.run.smithi069.stdout: "in_progress": true, 2024-04-14T14:22:24.822 INFO:teuthology.orchestra.run.smithi069.stdout: "which": "Upgrading daemons of type(s) osd. Upgrade limited to 2 daemons (1 remaining).", 2024-04-14T14:22:24.822 INFO:teuthology.orchestra.run.smithi069.stdout: "services_complete": [], 2024-04-14T14:22:24.822 INFO:teuthology.orchestra.run.smithi069.stdout: "progress": "1/8 daemons upgraded", 2024-04-14T14:22:24.822 INFO:teuthology.orchestra.run.smithi069.stdout: "message": "Currently upgrading osd daemons", 2024-04-14T14:22:24.822 INFO:teuthology.orchestra.run.smithi069.stdout: "is_paused": false 2024-04-14T14:22:24.822 INFO:teuthology.orchestra.run.smithi069.stdout:} 2024-04-14T14:22:25.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:22:24 smithi122 ceph-mon[67998]: from='client.54193 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T14:22:25.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:22:24 smithi122 ceph-mon[67998]: from='client.44262 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T14:22:25.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:22:24 smithi122 ceph-mon[67998]: from='client.? 172.21.15.69:0/1132436637' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T14:22:25.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:22:24 smithi122 ceph-mon[67998]: pgmap v62: 161 pgs: 161 active+clean; 585 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:22:26.067 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:22:25 smithi069 ceph-mon[83680]: from='client.44274 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T14:22:26.067 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:22:25 smithi069 ceph-mon[88636]: from='client.44274 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T14:22:26.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:22:25 smithi122 ceph-mon[67998]: from='client.44274 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T14:22:26.817 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:22:26 smithi069 ceph-mon[83680]: pgmap v63: 161 pgs: 161 active+clean; 585 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:22:26.817 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:22:26 smithi069 ceph-mon[88636]: pgmap v63: 161 pgs: 161 active+clean; 585 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:22:26.817 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 14 14:22:26 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-y[63038]: ::ffff:172.21.15.122 - - [14/Apr/2024:14:22:26] "GET /metrics HTTP/1.1" 200 38158 "" "Prometheus/2.43.0" 2024-04-14T14:22:27.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:22:26 smithi122 ceph-mon[67998]: pgmap v63: 161 pgs: 161 active+clean; 585 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:22:28.067 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:22:27 smithi069 ceph-mon[83680]: from='client.25417 -' entity='client.iscsi.foo.smithi069.yuumtr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T14:22:28.067 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:22:27 smithi069 ceph-mon[88636]: from='client.25417 -' entity='client.iscsi.foo.smithi069.yuumtr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T14:22:28.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:22:27 smithi122 ceph-mon[67998]: from='client.25417 -' entity='client.iscsi.foo.smithi069.yuumtr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T14:22:28.771 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:22:28 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2024-04-14T14:22:28.771 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:22:28 smithi069 ceph-mon[83680]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2024-04-14T14:22:28.771 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:22:28 smithi069 ceph-mon[83680]: Upgrade: osd.1 is safe to restart 2024-04-14T14:22:28.771 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:22:28 smithi069 ceph-mon[83680]: pgmap v64: 161 pgs: 161 active+clean; 585 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:22:28.771 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:22:28 smithi069 ceph-mon[88636]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2024-04-14T14:22:28.771 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:22:28 smithi069 ceph-mon[88636]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2024-04-14T14:22:28.771 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:22:28 smithi069 ceph-mon[88636]: Upgrade: osd.1 is safe to restart 2024-04-14T14:22:28.772 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:22:28 smithi069 ceph-mon[88636]: pgmap v64: 161 pgs: 161 active+clean; 585 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:22:29.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:22:28 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2024-04-14T14:22:29.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:22:28 smithi122 ceph-mon[67998]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2024-04-14T14:22:29.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:22:28 smithi122 ceph-mon[67998]: Upgrade: osd.1 is safe to restart 2024-04-14T14:22:29.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:22:28 smithi122 ceph-mon[67998]: pgmap v64: 161 pgs: 161 active+clean; 585 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:22:30.799 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:22:30 smithi069 ceph-mon[83680]: Upgrade: Updating osd.1 2024-04-14T14:22:30.799 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:22:30 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:22:30.799 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:22:30 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-04-14T14:22:30.799 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:22:30 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T14:22:30.799 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:22:30 smithi069 ceph-mon[83680]: Deploying daemon osd.1 on smithi069 2024-04-14T14:22:30.800 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:22:30 smithi069 ceph-mon[88636]: Upgrade: Updating osd.1 2024-04-14T14:22:30.800 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:22:30 smithi069 ceph-mon[88636]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:22:30.800 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:22:30 smithi069 ceph-mon[88636]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-04-14T14:22:30.800 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:22:30 smithi069 ceph-mon[88636]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T14:22:30.800 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:22:30 smithi069 ceph-mon[88636]: Deploying daemon osd.1 on smithi069 2024-04-14T14:22:30.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:22:30 smithi122 ceph-mon[67998]: Upgrade: Updating osd.1 2024-04-14T14:22:30.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:22:30 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:22:30.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:22:30 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-04-14T14:22:30.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:22:30 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T14:22:30.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:22:30 smithi122 ceph-mon[67998]: Deploying daemon osd.1 on smithi069 2024-04-14T14:22:31.804 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:22:31 smithi069 ceph-mon[83680]: pgmap v65: 161 pgs: 161 active+clean; 585 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:22:31.804 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:22:31 smithi069 ceph-mon[88636]: pgmap v65: 161 pgs: 161 active+clean; 585 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:22:31.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:22:31 smithi122 ceph-mon[67998]: pgmap v65: 161 pgs: 161 active+clean; 585 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:22:33.151 INFO:journalctl@ceph.osd.1.smithi069.stdout:Apr 14 14:22:32 smithi069 systemd[1]: Stopping Ceph osd.1 for dc326fc2-fa65-11ee-bc8e-c7b262605968... 2024-04-14T14:22:33.539 INFO:journalctl@ceph.osd.1.smithi069.stdout:Apr 14 14:22:33 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-osd-1[35300]: 2024-04-14T14:22:33.149+0000 7ff3a428a700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.1 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-14T14:22:33.539 INFO:journalctl@ceph.osd.1.smithi069.stdout:Apr 14 14:22:33 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-osd-1[35300]: 2024-04-14T14:22:33.149+0000 7ff3a428a700 -1 osd.1 116 *** Got signal Terminated *** 2024-04-14T14:22:33.539 INFO:journalctl@ceph.osd.1.smithi069.stdout:Apr 14 14:22:33 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-osd-1[35300]: 2024-04-14T14:22:33.149+0000 7ff3a428a700 -1 osd.1 116 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-14T14:22:33.817 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:22:33 smithi069 ceph-mon[83680]: pgmap v66: 161 pgs: 161 active+clean; 585 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:22:33.817 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:22:33 smithi069 ceph-mon[83680]: osd.1 marked itself down and dead 2024-04-14T14:22:33.817 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:22:33 smithi069 ceph-mon[88636]: pgmap v66: 161 pgs: 161 active+clean; 585 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:22:33.817 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:22:33 smithi069 ceph-mon[88636]: osd.1 marked itself down and dead 2024-04-14T14:22:33.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:22:33 smithi122 ceph-mon[67998]: pgmap v66: 161 pgs: 161 active+clean; 585 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:22:33.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:22:33 smithi122 ceph-mon[67998]: osd.1 marked itself down and dead 2024-04-14T14:22:34.211 INFO:journalctl@ceph.osd.1.smithi069.stdout:Apr 14 14:22:33 smithi069 podman[100488]: 2024-04-14 14:22:33.922139321 +0000 UTC m=+0.899864447 container died cdba54a59609f5f3ea0e5f52bf86192492201dd173107ea42b840beb2429de77 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-osd-1, io.openshift.tags=base centos centos-stream, ceph=True, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, io.k8s.display-name=CentOS Stream 8, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, vcs-type=git, release=754, CEPH_POINT_RELEASE=-17.2.0, vendor=Red Hat, Inc., description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, build-date=2022-05-03T08:36:31.336870, io.openshift.expose-services=, RELEASE=HEAD, name=centos-stream, io.buildah.version=1.19.8, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, version=8, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, GIT_BRANCH=HEAD, architecture=x86_64, GIT_REPO=https://github.com/ceph/ceph-container.git, distribution-scope=public, GIT_CLEAN=True, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, com.redhat.component=centos-stream-container, maintainer=Guillaume Abrioux ) 2024-04-14T14:22:34.540 INFO:journalctl@ceph.osd.1.smithi069.stdout:Apr 14 14:22:34 smithi069 podman[100488]: 2024-04-14 14:22:34.299607809 +0000 UTC m=+1.277332945 container cleanup cdba54a59609f5f3ea0e5f52bf86192492201dd173107ea42b840beb2429de77 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-osd-1, io.openshift.expose-services=, name=centos-stream, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, vcs-type=git, GIT_BRANCH=HEAD, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, ceph=True, RELEASE=HEAD, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., CEPH_POINT_RELEASE=-17.2.0, GIT_CLEAN=True, io.k8s.display-name=CentOS Stream 8, build-date=2022-05-03T08:36:31.336870, maintainer=Guillaume Abrioux , description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, com.redhat.component=centos-stream-container, architecture=x86_64, io.buildah.version=1.19.8, GIT_REPO=https://github.com/ceph/ceph-container.git, io.openshift.tags=base centos centos-stream, version=8, release=754, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, distribution-scope=public, vendor=Red Hat, Inc., GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb) 2024-04-14T14:22:34.541 INFO:journalctl@ceph.osd.1.smithi069.stdout:Apr 14 14:22:34 smithi069 bash[100488]: ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-osd-1 2024-04-14T14:22:34.817 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:22:34 smithi069 ceph-mon[83680]: Health check failed: 1 osds down (OSD_DOWN) 2024-04-14T14:22:34.817 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:22:34 smithi069 ceph-mon[83680]: osdmap e117: 8 total, 7 up, 8 in 2024-04-14T14:22:34.817 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:22:34 smithi069 ceph-mon[88636]: Health check failed: 1 osds down (OSD_DOWN) 2024-04-14T14:22:34.817 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:22:34 smithi069 ceph-mon[88636]: osdmap e117: 8 total, 7 up, 8 in 2024-04-14T14:22:34.817 INFO:journalctl@ceph.osd.1.smithi069.stdout:Apr 14 14:22:34 smithi069 podman[100515]: 2024-04-14 14:22:34.542546396 +0000 UTC m=+0.621787139 container remove cdba54a59609f5f3ea0e5f52bf86192492201dd173107ea42b840beb2429de77 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-osd-1, GIT_CLEAN=True, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, maintainer=Guillaume Abrioux , vcs-type=git, version=8, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, com.redhat.component=centos-stream-container, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, io.buildah.version=1.19.8, ceph=True, build-date=2022-05-03T08:36:31.336870, GIT_REPO=https://github.com/ceph/ceph-container.git, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, RELEASE=HEAD, release=754, GIT_BRANCH=HEAD, distribution-scope=public, io.k8s.display-name=CentOS Stream 8, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, name=centos-stream, io.openshift.tags=base centos centos-stream, architecture=x86_64, vendor=Red Hat, Inc., io.openshift.expose-services=, CEPH_POINT_RELEASE=-17.2.0, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac) 2024-04-14T14:22:34.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:22:34 smithi122 ceph-mon[67998]: Health check failed: 1 osds down (OSD_DOWN) 2024-04-14T14:22:34.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:22:34 smithi122 ceph-mon[67998]: osdmap e117: 8 total, 7 up, 8 in 2024-04-14T14:22:35.148 INFO:journalctl@ceph.osd.1.smithi069.stdout:Apr 14 14:22:34 smithi069 podman[100608]: 2024-04-14 14:22:34.770471975 +0000 UTC m=+0.017862112 image pull b96c509eef841460b3271f5fdd331860da87b004524f4ed59414aa3680183b53 quay.ceph.io/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888 2024-04-14T14:22:35.148 INFO:journalctl@ceph.osd.1.smithi069.stdout:Apr 14 14:22:34 smithi069 podman[100608]: 2024-04-14 14:22:34.868530035 +0000 UTC m=+0.115920168 container create 8f92f55b564c73682bbc826c9cad1084851021e3637cf83af0590f96a52a625e (image=quay.ceph.io/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888, name=ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-osd-1-deactivate, org.label-schema.build-date=20240403, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, RELEASE=main-a971fd5, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux ) 2024-04-14T14:22:35.148 INFO:journalctl@ceph.osd.1.smithi069.stdout:Apr 14 14:22:35 smithi069 podman[100608]: 2024-04-14 14:22:35.071001073 +0000 UTC m=+0.318391207 container init 8f92f55b564c73682bbc826c9cad1084851021e3637cf83af0590f96a52a625e (image=quay.ceph.io/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888, name=ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-osd-1-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, maintainer=Guillaume Abrioux , ceph=True, GIT_CLEAN=True, RELEASE=main-a971fd5, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.build-date=20240403, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-14T14:22:35.148 INFO:journalctl@ceph.osd.1.smithi069.stdout:Apr 14 14:22:35 smithi069 podman[100608]: 2024-04-14 14:22:35.074203167 +0000 UTC m=+0.321593309 container start 8f92f55b564c73682bbc826c9cad1084851021e3637cf83af0590f96a52a625e (image=quay.ceph.io/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888, name=ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-osd-1-deactivate, RELEASE=main-a971fd5, maintainer=Guillaume Abrioux , GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240403, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True) 2024-04-14T14:22:35.148 INFO:journalctl@ceph.osd.1.smithi069.stdout:Apr 14 14:22:35 smithi069 podman[100608]: 2024-04-14 14:22:35.145995627 +0000 UTC m=+0.393385783 container attach 8f92f55b564c73682bbc826c9cad1084851021e3637cf83af0590f96a52a625e (image=quay.ceph.io/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888, name=ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-osd-1-deactivate, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.build-date=20240403, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, ceph=True, GIT_CLEAN=True, RELEASE=main-a971fd5) 2024-04-14T14:22:35.551 INFO:journalctl@ceph.osd.1.smithi069.stdout:Apr 14 14:22:35 smithi069 podman[100608]: 2024-04-14 14:22:35.363285682 +0000 UTC m=+0.610675816 container died 8f92f55b564c73682bbc826c9cad1084851021e3637cf83af0590f96a52a625e (image=quay.ceph.io/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888, name=ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-osd-1-deactivate, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.build-date=20240403, GIT_CLEAN=True, RELEASE=main-a971fd5, maintainer=Guillaume Abrioux ) 2024-04-14T14:22:35.817 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:22:35 smithi069 ceph-mon[83680]: pgmap v68: 161 pgs: 21 stale+active+clean, 140 active+clean; 585 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-14T14:22:35.817 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:22:35 smithi069 ceph-mon[83680]: osdmap e118: 8 total, 7 up, 8 in 2024-04-14T14:22:35.817 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:22:35 smithi069 ceph-mon[88636]: pgmap v68: 161 pgs: 21 stale+active+clean, 140 active+clean; 585 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-14T14:22:35.817 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:22:35 smithi069 ceph-mon[88636]: osdmap e118: 8 total, 7 up, 8 in 2024-04-14T14:22:35.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:22:35 smithi122 ceph-mon[67998]: pgmap v68: 161 pgs: 21 stale+active+clean, 140 active+clean; 585 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-14T14:22:35.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:22:35 smithi122 ceph-mon[67998]: osdmap e118: 8 total, 7 up, 8 in 2024-04-14T14:22:36.317 INFO:journalctl@ceph.osd.1.smithi069.stdout:Apr 14 14:22:36 smithi069 podman[100634]: 2024-04-14 14:22:36.008015263 +0000 UTC m=+0.639260075 container remove 8f92f55b564c73682bbc826c9cad1084851021e3637cf83af0590f96a52a625e (image=quay.ceph.io/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888, name=ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-osd-1-deactivate, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.build-date=20240403, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=main-a971fd5, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, ceph=True) 2024-04-14T14:22:36.317 INFO:journalctl@ceph.osd.1.smithi069.stdout:Apr 14 14:22:36 smithi069 systemd[1]: ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968@osd.1.service: Deactivated successfully. 2024-04-14T14:22:36.318 INFO:journalctl@ceph.osd.1.smithi069.stdout:Apr 14 14:22:36 smithi069 systemd[1]: Stopped Ceph osd.1 for dc326fc2-fa65-11ee-bc8e-c7b262605968. 2024-04-14T14:22:36.318 INFO:journalctl@ceph.osd.1.smithi069.stdout:Apr 14 14:22:36 smithi069 systemd[1]: ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968@osd.1.service: Consumed 14.303s CPU time. 2024-04-14T14:22:36.318 INFO:journalctl@ceph.osd.1.smithi069.stdout:Apr 14 14:22:36 smithi069 systemd[1]: Starting Ceph osd.1 for dc326fc2-fa65-11ee-bc8e-c7b262605968... 2024-04-14T14:22:36.696 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 14 14:22:36 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-y[63038]: ::ffff:172.21.15.122 - - [14/Apr/2024:14:22:36] "GET /metrics HTTP/1.1" 200 38158 "" "Prometheus/2.43.0" 2024-04-14T14:22:36.697 INFO:journalctl@ceph.osd.1.smithi069.stdout:Apr 14 14:22:36 smithi069 podman[100734]: 2024-04-14 14:22:36.346406746 +0000 UTC m=+0.017762190 image pull b96c509eef841460b3271f5fdd331860da87b004524f4ed59414aa3680183b53 quay.ceph.io/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888 2024-04-14T14:22:36.697 INFO:journalctl@ceph.osd.1.smithi069.stdout:Apr 14 14:22:36 smithi069 podman[100734]: 2024-04-14 14:22:36.467741971 +0000 UTC m=+0.139097405 container create 320a586860c5ac3f5b5b6fceca321000a625c4fe7d4aefdc1e52688e2499b488 (image=quay.ceph.io/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888, name=ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-osd-1-activate, CEPH_POINT_RELEASE=, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, RELEASE=main-a971fd5, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.license=GPLv2, ceph=True, GIT_BRANCH=HEAD, org.label-schema.build-date=20240403, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux ) 2024-04-14T14:22:36.697 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:22:36 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.9", "id": [7, 2]}]: dispatch 2024-04-14T14:22:36.697 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:22:36 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.b", "id": [7, 1]}]: dispatch 2024-04-14T14:22:36.697 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:22:36 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1a", "id": [7, 1]}]: dispatch 2024-04-14T14:22:36.697 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:22:36 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "6.3", "id": [7, 1]}]: dispatch 2024-04-14T14:22:36.697 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:22:36 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "6.7", "id": [3, 0]}]: dispatch 2024-04-14T14:22:36.697 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:22:36 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "6.a", "id": [5, 1]}]: dispatch 2024-04-14T14:22:36.697 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:22:36 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "6.c", "id": [3, 1]}]: dispatch 2024-04-14T14:22:36.697 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:22:36 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "6.1b", "id": [3, 1]}]: dispatch 2024-04-14T14:22:36.698 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:22:36 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "6.1e", "id": [5, 1]}]: dispatch 2024-04-14T14:22:36.698 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:22:36 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "6.1f", "id": [3, 1]}]: dispatch 2024-04-14T14:22:36.698 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:22:36 smithi069 ceph-mon[88636]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.9", "id": [7, 2]}]: dispatch 2024-04-14T14:22:36.698 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:22:36 smithi069 ceph-mon[88636]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.b", "id": [7, 1]}]: dispatch 2024-04-14T14:22:36.698 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:22:36 smithi069 ceph-mon[88636]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1a", "id": [7, 1]}]: dispatch 2024-04-14T14:22:36.698 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:22:36 smithi069 ceph-mon[88636]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "6.3", "id": [7, 1]}]: dispatch 2024-04-14T14:22:36.698 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:22:36 smithi069 ceph-mon[88636]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "6.7", "id": [3, 0]}]: dispatch 2024-04-14T14:22:36.698 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:22:36 smithi069 ceph-mon[88636]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "6.a", "id": [5, 1]}]: dispatch 2024-04-14T14:22:36.698 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:22:36 smithi069 ceph-mon[88636]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "6.c", "id": [3, 1]}]: dispatch 2024-04-14T14:22:36.698 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:22:36 smithi069 ceph-mon[88636]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "6.1b", "id": [3, 1]}]: dispatch 2024-04-14T14:22:36.698 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:22:36 smithi069 ceph-mon[88636]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "6.1e", "id": [5, 1]}]: dispatch 2024-04-14T14:22:36.698 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:22:36 smithi069 ceph-mon[88636]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "6.1f", "id": [3, 1]}]: dispatch 2024-04-14T14:22:36.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:22:36 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.9", "id": [7, 2]}]: dispatch 2024-04-14T14:22:36.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:22:36 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.b", "id": [7, 1]}]: dispatch 2024-04-14T14:22:36.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:22:36 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1a", "id": [7, 1]}]: dispatch 2024-04-14T14:22:36.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:22:36 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "6.3", "id": [7, 1]}]: dispatch 2024-04-14T14:22:36.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:22:36 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "6.7", "id": [3, 0]}]: dispatch 2024-04-14T14:22:36.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:22:36 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "6.a", "id": [5, 1]}]: dispatch 2024-04-14T14:22:36.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:22:36 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "6.c", "id": [3, 1]}]: dispatch 2024-04-14T14:22:36.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:22:36 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "6.1b", "id": [3, 1]}]: dispatch 2024-04-14T14:22:36.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:22:36 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "6.1e", "id": [5, 1]}]: dispatch 2024-04-14T14:22:36.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:22:36 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "6.1f", "id": [3, 1]}]: dispatch 2024-04-14T14:22:37.067 INFO:journalctl@ceph.osd.1.smithi069.stdout:Apr 14 14:22:36 smithi069 podman[100734]: 2024-04-14 14:22:36.695410708 +0000 UTC m=+0.366766139 container init 320a586860c5ac3f5b5b6fceca321000a625c4fe7d4aefdc1e52688e2499b488 (image=quay.ceph.io/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888, name=ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-osd-1-activate, maintainer=Guillaume Abrioux , ceph=True, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240403, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, RELEASE=main-a971fd5, org.label-schema.vendor=CentOS) 2024-04-14T14:22:37.067 INFO:journalctl@ceph.osd.1.smithi069.stdout:Apr 14 14:22:36 smithi069 podman[100734]: 2024-04-14 14:22:36.698910654 +0000 UTC m=+0.370266089 container start 320a586860c5ac3f5b5b6fceca321000a625c4fe7d4aefdc1e52688e2499b488 (image=quay.ceph.io/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888, name=ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-osd-1-activate, GIT_BRANCH=HEAD, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, GIT_CLEAN=True, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , RELEASE=main-a971fd5, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.build-date=20240403, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-14T14:22:37.067 INFO:journalctl@ceph.osd.1.smithi069.stdout:Apr 14 14:22:36 smithi069 podman[100734]: 2024-04-14 14:22:36.778642562 +0000 UTC m=+0.449998005 container attach 320a586860c5ac3f5b5b6fceca321000a625c4fe7d4aefdc1e52688e2499b488 (image=quay.ceph.io/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888, name=ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-osd-1-activate, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, RELEASE=main-a971fd5, org.label-schema.license=GPLv2, org.label-schema.build-date=20240403, ceph=True, GIT_BRANCH=HEAD, GIT_CLEAN=True, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-14T14:22:37.484 INFO:journalctl@ceph.osd.1.smithi069.stdout:Apr 14 14:22:37 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-osd-1-activate[100750]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-04-14T14:22:37.484 INFO:journalctl@ceph.osd.1.smithi069.stdout:Apr 14 14:22:37 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-osd-1-activate[100750]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-1 --no-mon-config --dev /dev/mapper/vg_nvme-lv_3 2024-04-14T14:22:37.484 INFO:journalctl@ceph.osd.1.smithi069.stdout:Apr 14 14:22:37 smithi069 bash[100734]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-04-14T14:22:37.485 INFO:journalctl@ceph.osd.1.smithi069.stdout:Apr 14 14:22:37 smithi069 bash[100734]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-1 --no-mon-config --dev /dev/mapper/vg_nvme-lv_3 2024-04-14T14:22:37.485 INFO:journalctl@ceph.osd.1.smithi069.stdout:Apr 14 14:22:37 smithi069 bash[100734]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_3 2024-04-14T14:22:37.485 INFO:journalctl@ceph.osd.1.smithi069.stdout:Apr 14 14:22:37 smithi069 bash[100734]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-2 2024-04-14T14:22:37.485 INFO:journalctl@ceph.osd.1.smithi069.stdout:Apr 14 14:22:37 smithi069 bash[100734]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-1/block 2024-04-14T14:22:37.485 INFO:journalctl@ceph.osd.1.smithi069.stdout:Apr 14 14:22:37 smithi069 bash[100734]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-04-14T14:22:37.485 INFO:journalctl@ceph.osd.1.smithi069.stdout:Apr 14 14:22:37 smithi069 bash[100734]: --> ceph-volume raw activate successful for osd ID: 1 2024-04-14T14:22:37.485 INFO:journalctl@ceph.osd.1.smithi069.stdout:Apr 14 14:22:37 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-osd-1-activate[100750]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_3 2024-04-14T14:22:37.485 INFO:journalctl@ceph.osd.1.smithi069.stdout:Apr 14 14:22:37 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-osd-1-activate[100750]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-2 2024-04-14T14:22:37.485 INFO:journalctl@ceph.osd.1.smithi069.stdout:Apr 14 14:22:37 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-osd-1-activate[100750]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-1/block 2024-04-14T14:22:37.485 INFO:journalctl@ceph.osd.1.smithi069.stdout:Apr 14 14:22:37 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-osd-1-activate[100750]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-04-14T14:22:37.486 INFO:journalctl@ceph.osd.1.smithi069.stdout:Apr 14 14:22:37 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-osd-1-activate[100750]: --> ceph-volume raw activate successful for osd ID: 1 2024-04-14T14:22:37.486 INFO:journalctl@ceph.osd.1.smithi069.stdout:Apr 14 14:22:37 smithi069 podman[100734]: 2024-04-14 14:22:37.174327424 +0000 UTC m=+0.845682853 container died 320a586860c5ac3f5b5b6fceca321000a625c4fe7d4aefdc1e52688e2499b488 (image=quay.ceph.io/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888, name=ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-osd-1-activate, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240403, GIT_CLEAN=True, RELEASE=main-a971fd5, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux ) 2024-04-14T14:22:37.783 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:22:37 smithi069 ceph-mon[88636]: pgmap v70: 161 pgs: 21 stale+active+clean, 140 active+clean; 585 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-04-14T14:22:37.784 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:22:37 smithi069 ceph-mon[88636]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.9", "id": [7, 2]}]': finished 2024-04-14T14:22:37.784 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:22:37 smithi069 ceph-mon[88636]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.b", "id": [7, 1]}]': finished 2024-04-14T14:22:37.784 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:22:37 smithi069 ceph-mon[88636]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1a", "id": [7, 1]}]': finished 2024-04-14T14:22:37.784 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:22:37 smithi069 ceph-mon[88636]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "6.3", "id": [7, 1]}]': finished 2024-04-14T14:22:37.784 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:22:37 smithi069 ceph-mon[88636]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "6.7", "id": [3, 0]}]': finished 2024-04-14T14:22:37.784 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:22:37 smithi069 ceph-mon[88636]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "6.a", "id": [5, 1]}]': finished 2024-04-14T14:22:37.784 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:22:37 smithi069 ceph-mon[88636]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "6.c", "id": [3, 1]}]': finished 2024-04-14T14:22:37.784 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:22:37 smithi069 ceph-mon[88636]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "6.1b", "id": [3, 1]}]': finished 2024-04-14T14:22:37.784 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:22:37 smithi069 ceph-mon[88636]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "6.1e", "id": [5, 1]}]': finished 2024-04-14T14:22:37.784 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:22:37 smithi069 ceph-mon[88636]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "6.1f", "id": [3, 1]}]': finished 2024-04-14T14:22:37.784 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:22:37 smithi069 ceph-mon[88636]: osdmap e119: 8 total, 7 up, 8 in 2024-04-14T14:22:37.784 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:22:37 smithi069 ceph-mon[88636]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:22:37.784 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:22:37 smithi069 ceph-mon[88636]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T14:22:37.785 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:22:37 smithi069 ceph-mon[83680]: pgmap v70: 161 pgs: 21 stale+active+clean, 140 active+clean; 585 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-04-14T14:22:37.785 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:22:37 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.9", "id": [7, 2]}]': finished 2024-04-14T14:22:37.785 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:22:37 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.b", "id": [7, 1]}]': finished 2024-04-14T14:22:37.785 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:22:37 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1a", "id": [7, 1]}]': finished 2024-04-14T14:22:37.785 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:22:37 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "6.3", "id": [7, 1]}]': finished 2024-04-14T14:22:37.785 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:22:37 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "6.7", "id": [3, 0]}]': finished 2024-04-14T14:22:37.785 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:22:37 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "6.a", "id": [5, 1]}]': finished 2024-04-14T14:22:37.785 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:22:37 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "6.c", "id": [3, 1]}]': finished 2024-04-14T14:22:37.785 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:22:37 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "6.1b", "id": [3, 1]}]': finished 2024-04-14T14:22:37.785 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:22:37 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "6.1e", "id": [5, 1]}]': finished 2024-04-14T14:22:37.785 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:22:37 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "6.1f", "id": [3, 1]}]': finished 2024-04-14T14:22:37.785 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:22:37 smithi069 ceph-mon[83680]: osdmap e119: 8 total, 7 up, 8 in 2024-04-14T14:22:37.786 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:22:37 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:22:37.786 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:22:37 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T14:22:37.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:22:37 smithi122 ceph-mon[67998]: pgmap v70: 161 pgs: 21 stale+active+clean, 140 active+clean; 585 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-04-14T14:22:37.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:22:37 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.9", "id": [7, 2]}]': finished 2024-04-14T14:22:37.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:22:37 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.b", "id": [7, 1]}]': finished 2024-04-14T14:22:37.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:22:37 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1a", "id": [7, 1]}]': finished 2024-04-14T14:22:37.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:22:37 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "6.3", "id": [7, 1]}]': finished 2024-04-14T14:22:37.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:22:37 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "6.7", "id": [3, 0]}]': finished 2024-04-14T14:22:37.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:22:37 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "6.a", "id": [5, 1]}]': finished 2024-04-14T14:22:37.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:22:37 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "6.c", "id": [3, 1]}]': finished 2024-04-14T14:22:37.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:22:37 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "6.1b", "id": [3, 1]}]': finished 2024-04-14T14:22:37.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:22:37 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "6.1e", "id": [5, 1]}]': finished 2024-04-14T14:22:37.846 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:22:37 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "6.1f", "id": [3, 1]}]': finished 2024-04-14T14:22:37.846 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:22:37 smithi122 ceph-mon[67998]: osdmap e119: 8 total, 7 up, 8 in 2024-04-14T14:22:37.846 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:22:37 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:22:37.846 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:22:37 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T14:22:38.067 INFO:journalctl@ceph.osd.1.smithi069.stdout:Apr 14 14:22:37 smithi069 podman[100875]: 2024-04-14 14:22:37.782513116 +0000 UTC m=+0.602604270 container remove 320a586860c5ac3f5b5b6fceca321000a625c4fe7d4aefdc1e52688e2499b488 (image=quay.ceph.io/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888, name=ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-osd-1-activate, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, ceph=True, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240403, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, RELEASE=main-a971fd5, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-14T14:22:38.067 INFO:journalctl@ceph.osd.1.smithi069.stdout:Apr 14 14:22:37 smithi069 podman[100941]: 2024-04-14 14:22:37.991989726 +0000 UTC m=+0.113307776 container create cf73d17919bb431b2654fadb0d2d50ef708ede157cd76d8d28d56fed6c7affdc (image=quay.ceph.io/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888, name=ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-osd-1, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240403, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, ceph=True, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, RELEASE=main-a971fd5) 2024-04-14T14:22:38.067 INFO:journalctl@ceph.osd.1.smithi069.stdout:Apr 14 14:22:37 smithi069 podman[100941]: 2024-04-14 14:22:37.897312017 +0000 UTC m=+0.018630041 image pull b96c509eef841460b3271f5fdd331860da87b004524f4ed59414aa3680183b53 quay.ceph.io/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888 2024-04-14T14:22:38.567 INFO:journalctl@ceph.osd.1.smithi069.stdout:Apr 14 14:22:38 smithi069 podman[100941]: 2024-04-14 14:22:38.196313252 +0000 UTC m=+0.317631270 container init cf73d17919bb431b2654fadb0d2d50ef708ede157cd76d8d28d56fed6c7affdc (image=quay.ceph.io/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888, name=ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-osd-1, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240403, RELEASE=main-a971fd5, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, ceph=True, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD) 2024-04-14T14:22:38.567 INFO:journalctl@ceph.osd.1.smithi069.stdout:Apr 14 14:22:38 smithi069 podman[100941]: 2024-04-14 14:22:38.199323307 +0000 UTC m=+0.320641316 container start cf73d17919bb431b2654fadb0d2d50ef708ede157cd76d8d28d56fed6c7affdc (image=quay.ceph.io/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888, name=ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-osd-1, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_CLEAN=True, io.buildah.version=1.33.5, ceph=True, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240403, RELEASE=main-a971fd5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD) 2024-04-14T14:22:38.567 INFO:journalctl@ceph.osd.1.smithi069.stdout:Apr 14 14:22:38 smithi069 ceph-osd[100961]: -- 172.21.15.69:0/3204866924 <== mon.1 v2:172.21.15.69:3301/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55f2a6f1d680 con 0x55f2a6ee7000 2024-04-14T14:22:38.568 INFO:journalctl@ceph.osd.1.smithi069.stdout:Apr 14 14:22:38 smithi069 bash[100941]: cf73d17919bb431b2654fadb0d2d50ef708ede157cd76d8d28d56fed6c7affdc 2024-04-14T14:22:38.568 INFO:journalctl@ceph.osd.1.smithi069.stdout:Apr 14 14:22:38 smithi069 systemd[1]: Started Ceph osd.1 for dc326fc2-fa65-11ee-bc8e-c7b262605968. 2024-04-14T14:22:38.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:22:38 smithi122 ceph-mon[67998]: osdmap e120: 8 total, 7 up, 8 in 2024-04-14T14:22:38.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:22:38 smithi122 ceph-mon[67998]: from='client.25417 -' entity='client.iscsi.foo.smithi069.yuumtr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T14:22:38.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:22:38 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:22:38.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:22:38 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:22:38.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:22:38 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:22:38.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:22:38 smithi122 ceph-mon[67998]: osdmap e121: 8 total, 7 up, 8 in 2024-04-14T14:22:39.067 INFO:journalctl@ceph.osd.1.smithi069.stdout:Apr 14 14:22:38 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-osd-1[100957]: 2024-04-14T14:22:38.736+0000 7f45178c07c0 -1 Falling back to public interface 2024-04-14T14:22:39.067 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:22:38 smithi069 ceph-mon[83680]: osdmap e120: 8 total, 7 up, 8 in 2024-04-14T14:22:39.067 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:22:38 smithi069 ceph-mon[83680]: from='client.25417 -' entity='client.iscsi.foo.smithi069.yuumtr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T14:22:39.068 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:22:38 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:22:39.068 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:22:38 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:22:39.068 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:22:38 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:22:39.068 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:22:38 smithi069 ceph-mon[83680]: osdmap e121: 8 total, 7 up, 8 in 2024-04-14T14:22:39.068 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:22:38 smithi069 ceph-mon[88636]: osdmap e120: 8 total, 7 up, 8 in 2024-04-14T14:22:39.068 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:22:38 smithi069 ceph-mon[88636]: from='client.25417 -' entity='client.iscsi.foo.smithi069.yuumtr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T14:22:39.068 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:22:38 smithi069 ceph-mon[88636]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:22:39.068 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:22:38 smithi069 ceph-mon[88636]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:22:39.068 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:22:38 smithi069 ceph-mon[88636]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:22:39.068 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:22:38 smithi069 ceph-mon[88636]: osdmap e121: 8 total, 7 up, 8 in 2024-04-14T14:22:39.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:22:39 smithi122 ceph-mon[67998]: pgmap v73: 161 pgs: 9 peering, 21 active+undersized+degraded, 1 stale+active+clean, 41 active+undersized, 89 active+clean; 585 KiB data, 120 MiB used, 715 GiB / 715 GiB avail; 84/627 objects degraded (13.397%) 2024-04-14T14:22:39.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:22:39 smithi122 ceph-mon[67998]: Health check failed: Reduced data availability: 4 pgs inactive, 5 pgs peering (PG_AVAILABILITY) 2024-04-14T14:22:39.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:22:39 smithi122 ceph-mon[67998]: Health check failed: Degraded data redundancy: 84/627 objects degraded (13.397%), 21 pgs degraded (PG_DEGRADED) 2024-04-14T14:22:40.067 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:22:39 smithi069 ceph-mon[83680]: pgmap v73: 161 pgs: 9 peering, 21 active+undersized+degraded, 1 stale+active+clean, 41 active+undersized, 89 active+clean; 585 KiB data, 120 MiB used, 715 GiB / 715 GiB avail; 84/627 objects degraded (13.397%) 2024-04-14T14:22:40.067 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:22:39 smithi069 ceph-mon[83680]: Health check failed: Reduced data availability: 4 pgs inactive, 5 pgs peering (PG_AVAILABILITY) 2024-04-14T14:22:40.067 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:22:39 smithi069 ceph-mon[83680]: Health check failed: Degraded data redundancy: 84/627 objects degraded (13.397%), 21 pgs degraded (PG_DEGRADED) 2024-04-14T14:22:40.067 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:22:39 smithi069 ceph-mon[88636]: pgmap v73: 161 pgs: 9 peering, 21 active+undersized+degraded, 1 stale+active+clean, 41 active+undersized, 89 active+clean; 585 KiB data, 120 MiB used, 715 GiB / 715 GiB avail; 84/627 objects degraded (13.397%) 2024-04-14T14:22:40.067 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:22:39 smithi069 ceph-mon[88636]: Health check failed: Reduced data availability: 4 pgs inactive, 5 pgs peering (PG_AVAILABILITY) 2024-04-14T14:22:40.067 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:22:39 smithi069 ceph-mon[88636]: Health check failed: Degraded data redundancy: 84/627 objects degraded (13.397%), 21 pgs degraded (PG_DEGRADED) 2024-04-14T14:22:41.802 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:22:41 smithi069 ceph-mon[83680]: pgmap v75: 161 pgs: 9 peering, 21 active+undersized+degraded, 1 stale+active+clean, 41 active+undersized, 89 active+clean; 585 KiB data, 120 MiB used, 715 GiB / 715 GiB avail; 84/627 objects degraded (13.397%) 2024-04-14T14:22:41.803 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:22:41 smithi069 ceph-mon[83680]: from='osd.1 [v2:172.21.15.69:6810/2322984068,v1:172.21.15.69:6811/2322984068]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-04-14T14:22:41.803 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:22:41 smithi069 ceph-mon[83680]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-04-14T14:22:41.803 INFO:journalctl@ceph.osd.1.smithi069.stdout:Apr 14 14:22:41 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-osd-1[100957]: 2024-04-14T14:22:41.476+0000 7f45178c07c0 -1 osd.1 0 read_superblock omap replica is missing. 2024-04-14T14:22:41.803 INFO:journalctl@ceph.osd.1.smithi069.stdout:Apr 14 14:22:41 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-osd-1[100957]: 2024-04-14T14:22:41.512+0000 7f45178c07c0 -1 osd.1 116 log_to_monitors true 2024-04-14T14:22:41.804 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:22:41 smithi069 ceph-mon[88636]: pgmap v75: 161 pgs: 9 peering, 21 active+undersized+degraded, 1 stale+active+clean, 41 active+undersized, 89 active+clean; 585 KiB data, 120 MiB used, 715 GiB / 715 GiB avail; 84/627 objects degraded (13.397%) 2024-04-14T14:22:41.804 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:22:41 smithi069 ceph-mon[88636]: from='osd.1 [v2:172.21.15.69:6810/2322984068,v1:172.21.15.69:6811/2322984068]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-04-14T14:22:41.804 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:22:41 smithi069 ceph-mon[88636]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-04-14T14:22:41.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:22:41 smithi122 ceph-mon[67998]: pgmap v75: 161 pgs: 9 peering, 21 active+undersized+degraded, 1 stale+active+clean, 41 active+undersized, 89 active+clean; 585 KiB data, 120 MiB used, 715 GiB / 715 GiB avail; 84/627 objects degraded (13.397%) 2024-04-14T14:22:41.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:22:41 smithi122 ceph-mon[67998]: from='osd.1 [v2:172.21.15.69:6810/2322984068,v1:172.21.15.69:6811/2322984068]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-04-14T14:22:41.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:22:41 smithi122 ceph-mon[67998]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-04-14T14:22:42.066 INFO:journalctl@ceph.osd.1.smithi069.stdout:Apr 14 14:22:42 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-osd-1[100957]: 2024-04-14T14:22:42.040+0000 7f4504d10700 -1 osd.1 116 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-14T14:22:42.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:22:42 smithi122 ceph-mon[67998]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-04-14T14:22:42.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:22:42 smithi122 ceph-mon[67998]: osdmap e122: 8 total, 7 up, 8 in 2024-04-14T14:22:42.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:22:42 smithi122 ceph-mon[67998]: from='osd.1 [v2:172.21.15.69:6810/2322984068,v1:172.21.15.69:6811/2322984068]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi069", "root=default"]}]: dispatch 2024-04-14T14:22:42.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:22:42 smithi122 ceph-mon[67998]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi069", "root=default"]}]: dispatch 2024-04-14T14:22:42.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:22:42 smithi122 ceph-mon[67998]: from='osd.1 ' entity='osd.1' 2024-04-14T14:22:43.070 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:22:42 smithi069 ceph-mon[83680]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-04-14T14:22:43.070 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:22:42 smithi069 ceph-mon[83680]: osdmap e122: 8 total, 7 up, 8 in 2024-04-14T14:22:43.070 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:22:42 smithi069 ceph-mon[83680]: from='osd.1 [v2:172.21.15.69:6810/2322984068,v1:172.21.15.69:6811/2322984068]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi069", "root=default"]}]: dispatch 2024-04-14T14:22:43.070 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:22:42 smithi069 ceph-mon[83680]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi069", "root=default"]}]: dispatch 2024-04-14T14:22:43.070 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:22:42 smithi069 ceph-mon[83680]: from='osd.1 ' entity='osd.1' 2024-04-14T14:22:43.071 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:22:42 smithi069 ceph-mon[88636]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-04-14T14:22:43.071 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:22:42 smithi069 ceph-mon[88636]: osdmap e122: 8 total, 7 up, 8 in 2024-04-14T14:22:43.071 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:22:42 smithi069 ceph-mon[88636]: from='osd.1 [v2:172.21.15.69:6810/2322984068,v1:172.21.15.69:6811/2322984068]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi069", "root=default"]}]: dispatch 2024-04-14T14:22:43.071 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:22:42 smithi069 ceph-mon[88636]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi069", "root=default"]}]: dispatch 2024-04-14T14:22:43.071 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:22:42 smithi069 ceph-mon[88636]: from='osd.1 ' entity='osd.1' 2024-04-14T14:22:43.817 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:22:43 smithi069 ceph-mon[83680]: pgmap v77: 161 pgs: 9 active+clean+remapped, 21 active+undersized+degraded, 41 active+undersized, 90 active+clean; 585 KiB data, 136 MiB used, 715 GiB / 715 GiB avail; 84/627 objects degraded (13.397%); 3/627 objects misplaced (0.478%); 138 B/s, 0 objects/s recovering 2024-04-14T14:22:43.817 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:22:43 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:22:43.817 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:22:43 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:22:43.817 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:22:43 smithi069 ceph-mon[83680]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 4 pgs inactive, 5 pgs peering) 2024-04-14T14:22:43.817 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:22:43 smithi069 ceph-mon[83680]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-04-14T14:22:43.817 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:22:43 smithi069 ceph-mon[83680]: osd.1 [v2:172.21.15.69:6810/2322984068,v1:172.21.15.69:6811/2322984068] boot 2024-04-14T14:22:43.818 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:22:43 smithi069 ceph-mon[83680]: osdmap e123: 8 total, 8 up, 8 in 2024-04-14T14:22:43.818 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:22:43 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-14T14:22:43.818 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:22:43 smithi069 ceph-mon[83680]: osdmap e124: 8 total, 8 up, 8 in 2024-04-14T14:22:43.818 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:22:43 smithi069 ceph-mon[88636]: pgmap v77: 161 pgs: 9 active+clean+remapped, 21 active+undersized+degraded, 41 active+undersized, 90 active+clean; 585 KiB data, 136 MiB used, 715 GiB / 715 GiB avail; 84/627 objects degraded (13.397%); 3/627 objects misplaced (0.478%); 138 B/s, 0 objects/s recovering 2024-04-14T14:22:43.818 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:22:43 smithi069 ceph-mon[88636]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:22:43.818 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:22:43 smithi069 ceph-mon[88636]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:22:43.818 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:22:43 smithi069 ceph-mon[88636]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 4 pgs inactive, 5 pgs peering) 2024-04-14T14:22:43.818 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:22:43 smithi069 ceph-mon[88636]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-04-14T14:22:43.818 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:22:43 smithi069 ceph-mon[88636]: osd.1 [v2:172.21.15.69:6810/2322984068,v1:172.21.15.69:6811/2322984068] boot 2024-04-14T14:22:43.818 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:22:43 smithi069 ceph-mon[88636]: osdmap e123: 8 total, 8 up, 8 in 2024-04-14T14:22:43.818 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:22:43 smithi069 ceph-mon[88636]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-14T14:22:43.818 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:22:43 smithi069 ceph-mon[88636]: osdmap e124: 8 total, 8 up, 8 in 2024-04-14T14:22:43.846 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:22:43 smithi122 ceph-mon[67998]: pgmap v77: 161 pgs: 9 active+clean+remapped, 21 active+undersized+degraded, 41 active+undersized, 90 active+clean; 585 KiB data, 136 MiB used, 715 GiB / 715 GiB avail; 84/627 objects degraded (13.397%); 3/627 objects misplaced (0.478%); 138 B/s, 0 objects/s recovering 2024-04-14T14:22:43.846 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:22:43 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:22:43.846 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:22:43 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:22:43.846 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:22:43 smithi122 ceph-mon[67998]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 4 pgs inactive, 5 pgs peering) 2024-04-14T14:22:43.846 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:22:43 smithi122 ceph-mon[67998]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-04-14T14:22:43.846 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:22:43 smithi122 ceph-mon[67998]: osd.1 [v2:172.21.15.69:6810/2322984068,v1:172.21.15.69:6811/2322984068] boot 2024-04-14T14:22:43.847 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:22:43 smithi122 ceph-mon[67998]: osdmap e123: 8 total, 8 up, 8 in 2024-04-14T14:22:43.847 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:22:43 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-14T14:22:43.847 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:22:43 smithi122 ceph-mon[67998]: osdmap e124: 8 total, 8 up, 8 in 2024-04-14T14:22:45.317 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:22:44 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:22:45.317 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:22:44 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:22:45.317 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:22:44 smithi069 ceph-mon[83680]: osdmap e125: 8 total, 8 up, 8 in 2024-04-14T14:22:45.317 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:22:44 smithi069 ceph-mon[83680]: pgmap v81: 161 pgs: 9 active+clean+remapped, 21 active+undersized+degraded, 41 active+undersized, 90 active+clean; 585 KiB data, 136 MiB used, 715 GiB / 715 GiB avail; 84/627 objects degraded (13.397%); 3/627 objects misplaced (0.478%); 207 B/s, 0 objects/s recovering 2024-04-14T14:22:45.317 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:22:44 smithi069 ceph-mon[88636]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:22:45.317 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:22:44 smithi069 ceph-mon[88636]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:22:45.317 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:22:44 smithi069 ceph-mon[88636]: osdmap e125: 8 total, 8 up, 8 in 2024-04-14T14:22:45.317 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:22:44 smithi069 ceph-mon[88636]: pgmap v81: 161 pgs: 9 active+clean+remapped, 21 active+undersized+degraded, 41 active+undersized, 90 active+clean; 585 KiB data, 136 MiB used, 715 GiB / 715 GiB avail; 84/627 objects degraded (13.397%); 3/627 objects misplaced (0.478%); 207 B/s, 0 objects/s recovering 2024-04-14T14:22:45.345 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:22:44 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:22:45.345 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:22:44 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:22:45.345 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:22:44 smithi122 ceph-mon[67998]: osdmap e125: 8 total, 8 up, 8 in 2024-04-14T14:22:45.345 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:22:44 smithi122 ceph-mon[67998]: pgmap v81: 161 pgs: 9 active+clean+remapped, 21 active+undersized+degraded, 41 active+undersized, 90 active+clean; 585 KiB data, 136 MiB used, 715 GiB / 715 GiB avail; 84/627 objects degraded (13.397%); 3/627 objects misplaced (0.478%); 207 B/s, 0 objects/s recovering 2024-04-14T14:22:46.817 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 14 14:22:46 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-y[63038]: ::ffff:172.21.15.122 - - [14/Apr/2024:14:22:46] "GET /metrics HTTP/1.1" 200 38153 "" "Prometheus/2.43.0" 2024-04-14T14:22:47.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:22:47 smithi122 ceph-mon[67998]: pgmap v82: 161 pgs: 9 active+clean+remapped, 21 active+undersized+degraded, 41 active+undersized, 90 active+clean; 585 KiB data, 136 MiB used, 715 GiB / 715 GiB avail; 84/627 objects degraded (13.397%); 3/627 objects misplaced (0.478%); 167 B/s, 0 objects/s recovering 2024-04-14T14:22:47.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:22:47 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:22:47.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:22:47 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:22:47.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:22:47 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T14:22:47.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:22:47 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-14T14:22:47.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:22:47 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:22:47.846 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:22:47 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-14T14:22:47.846 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:22:47 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T14:22:47.846 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:22:47 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T14:22:47.846 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:22:47 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:22:47.846 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:22:47 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T14:22:47.846 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:22:47 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T14:22:47.846 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:22:47 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T14:22:47.846 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:22:47 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T14:22:47.846 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:22:47 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T14:22:47.846 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:22:47 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T14:22:47.846 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:22:47 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]: dispatch 2024-04-14T14:22:47.846 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:22:47 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]': finished 2024-04-14T14:22:47.846 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:22:47 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-14T14:22:47.847 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:22:47 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon"}]': finished 2024-04-14T14:22:47.847 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:22:47 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]: dispatch 2024-04-14T14:22:47.847 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:22:47 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]': finished 2024-04-14T14:22:47.847 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:22:47 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd"}]: dispatch 2024-04-14T14:22:47.847 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:22:47 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mds"}]: dispatch 2024-04-14T14:22:47.847 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:22:47 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw"}]: dispatch 2024-04-14T14:22:47.847 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:22:47 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"}]: dispatch 2024-04-14T14:22:47.847 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:22:47 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-14T14:22:47.847 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:22:47 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter"}]: dispatch 2024-04-14T14:22:47.847 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:22:47 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.iscsi"}]: dispatch 2024-04-14T14:22:47.847 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:22:47 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nfs"}]: dispatch 2024-04-14T14:22:47.847 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:22:47 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]: dispatch 2024-04-14T14:22:47.847 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:22:47 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]': finished 2024-04-14T14:22:47.847 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:22:47 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-14T14:22:47.847 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:22:47 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-14T14:22:47.848 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:22:47 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-14T14:22:47.848 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:22:47 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-14T14:22:47.848 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:22:47 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-14T14:22:47.848 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:22:47 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-14T14:22:47.848 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:22:47 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]: dispatch 2024-04-14T14:22:47.848 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:22:47 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd='[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]': finished 2024-04-14T14:22:48.067 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:22:47 smithi069 ceph-mon[83680]: pgmap v82: 161 pgs: 9 active+clean+remapped, 21 active+undersized+degraded, 41 active+undersized, 90 active+clean; 585 KiB data, 136 MiB used, 715 GiB / 715 GiB avail; 84/627 objects degraded (13.397%); 3/627 objects misplaced (0.478%); 167 B/s, 0 objects/s recovering 2024-04-14T14:22:48.068 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:22:47 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:22:48.068 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:22:47 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:22:48.068 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:22:47 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T14:22:48.068 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:22:47 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-14T14:22:48.068 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:22:47 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:22:48.068 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:22:47 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-14T14:22:48.069 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:22:47 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T14:22:48.069 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:22:47 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T14:22:48.069 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:22:47 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:22:48.069 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:22:47 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T14:22:48.069 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:22:47 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T14:22:48.069 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:22:47 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T14:22:48.069 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:22:47 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T14:22:48.069 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:22:47 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T14:22:48.069 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:22:47 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T14:22:48.070 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:22:47 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]: dispatch 2024-04-14T14:22:48.070 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:22:47 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]': finished 2024-04-14T14:22:48.070 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:22:47 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-14T14:22:48.070 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:22:47 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon"}]': finished 2024-04-14T14:22:48.070 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:22:47 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]: dispatch 2024-04-14T14:22:48.070 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:22:47 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]': finished 2024-04-14T14:22:48.070 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:22:47 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd"}]: dispatch 2024-04-14T14:22:48.070 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:22:47 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mds"}]: dispatch 2024-04-14T14:22:48.070 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:22:47 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw"}]: dispatch 2024-04-14T14:22:48.070 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:22:47 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"}]: dispatch 2024-04-14T14:22:48.070 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:22:47 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-14T14:22:48.070 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:22:47 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter"}]: dispatch 2024-04-14T14:22:48.071 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:22:47 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.iscsi"}]: dispatch 2024-04-14T14:22:48.071 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:22:47 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nfs"}]: dispatch 2024-04-14T14:22:48.071 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:22:47 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]: dispatch 2024-04-14T14:22:48.071 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:22:47 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]': finished 2024-04-14T14:22:48.071 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:22:47 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-14T14:22:48.071 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:22:47 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-14T14:22:48.071 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:22:47 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-14T14:22:48.071 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:22:47 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-14T14:22:48.071 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:22:47 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-14T14:22:48.071 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:22:47 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-14T14:22:48.071 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:22:47 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]: dispatch 2024-04-14T14:22:48.071 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:22:47 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd='[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]': finished 2024-04-14T14:22:48.072 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:22:47 smithi069 ceph-mon[88636]: pgmap v82: 161 pgs: 9 active+clean+remapped, 21 active+undersized+degraded, 41 active+undersized, 90 active+clean; 585 KiB data, 136 MiB used, 715 GiB / 715 GiB avail; 84/627 objects degraded (13.397%); 3/627 objects misplaced (0.478%); 167 B/s, 0 objects/s recovering 2024-04-14T14:22:48.072 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:22:47 smithi069 ceph-mon[88636]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:22:48.072 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:22:47 smithi069 ceph-mon[88636]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:22:48.072 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:22:47 smithi069 ceph-mon[88636]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T14:22:48.072 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:22:47 smithi069 ceph-mon[88636]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-14T14:22:48.072 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:22:47 smithi069 ceph-mon[88636]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:22:48.072 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:22:47 smithi069 ceph-mon[88636]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-14T14:22:48.072 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:22:47 smithi069 ceph-mon[88636]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T14:22:48.072 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:22:47 smithi069 ceph-mon[88636]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T14:22:48.072 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:22:47 smithi069 ceph-mon[88636]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:22:48.072 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:22:47 smithi069 ceph-mon[88636]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T14:22:48.072 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:22:47 smithi069 ceph-mon[88636]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T14:22:48.073 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:22:47 smithi069 ceph-mon[88636]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T14:22:48.073 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:22:47 smithi069 ceph-mon[88636]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T14:22:48.073 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:22:47 smithi069 ceph-mon[88636]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T14:22:48.073 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:22:47 smithi069 ceph-mon[88636]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T14:22:48.073 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:22:47 smithi069 ceph-mon[88636]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]: dispatch 2024-04-14T14:22:48.073 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:22:47 smithi069 ceph-mon[88636]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]': finished 2024-04-14T14:22:48.073 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:22:47 smithi069 ceph-mon[88636]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-14T14:22:48.073 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:22:47 smithi069 ceph-mon[88636]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon"}]': finished 2024-04-14T14:22:48.073 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:22:47 smithi069 ceph-mon[88636]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]: dispatch 2024-04-14T14:22:48.073 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:22:47 smithi069 ceph-mon[88636]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]': finished 2024-04-14T14:22:48.073 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:22:47 smithi069 ceph-mon[88636]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd"}]: dispatch 2024-04-14T14:22:48.073 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:22:47 smithi069 ceph-mon[88636]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mds"}]: dispatch 2024-04-14T14:22:48.073 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:22:47 smithi069 ceph-mon[88636]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw"}]: dispatch 2024-04-14T14:22:48.073 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:22:47 smithi069 ceph-mon[88636]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"}]: dispatch 2024-04-14T14:22:48.073 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:22:47 smithi069 ceph-mon[88636]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-14T14:22:48.074 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:22:47 smithi069 ceph-mon[88636]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter"}]: dispatch 2024-04-14T14:22:48.074 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:22:47 smithi069 ceph-mon[88636]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.iscsi"}]: dispatch 2024-04-14T14:22:48.074 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:22:47 smithi069 ceph-mon[88636]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nfs"}]: dispatch 2024-04-14T14:22:48.074 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:22:47 smithi069 ceph-mon[88636]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]: dispatch 2024-04-14T14:22:48.074 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:22:47 smithi069 ceph-mon[88636]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]': finished 2024-04-14T14:22:48.074 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:22:47 smithi069 ceph-mon[88636]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-14T14:22:48.074 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:22:47 smithi069 ceph-mon[88636]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-14T14:22:48.074 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:22:47 smithi069 ceph-mon[88636]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-14T14:22:48.074 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:22:47 smithi069 ceph-mon[88636]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-14T14:22:48.074 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:22:47 smithi069 ceph-mon[88636]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-14T14:22:48.074 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:22:47 smithi069 ceph-mon[88636]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-14T14:22:48.074 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:22:47 smithi069 ceph-mon[88636]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]: dispatch 2024-04-14T14:22:48.074 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:22:47 smithi069 ceph-mon[88636]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd='[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]': finished 2024-04-14T14:22:48.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:22:48 smithi122 ceph-mon[67998]: Upgrade: Setting container_image for all nvmeof 2024-04-14T14:22:48.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:22:48 smithi122 ceph-mon[67998]: Upgrade: Finalizing container_image settings 2024-04-14T14:22:48.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:22:48 smithi122 ceph-mon[67998]: Upgrade: Complete! 2024-04-14T14:22:48.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:22:48 smithi122 ceph-mon[67998]: from='client.25417 -' entity='client.iscsi.foo.smithi069.yuumtr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T14:22:48.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:22:48 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:22:48.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:22:48 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:22:48.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:22:48 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T14:22:48.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:22:48 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-14T14:22:48.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:22:48 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:22:48.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:22:48 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T14:22:48.846 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:22:48 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-14T14:22:48.846 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:22:48 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:22:49.067 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:22:48 smithi069 ceph-mon[88636]: Upgrade: Setting container_image for all nvmeof 2024-04-14T14:22:49.067 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:22:48 smithi069 ceph-mon[88636]: Upgrade: Finalizing container_image settings 2024-04-14T14:22:49.067 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:22:48 smithi069 ceph-mon[88636]: Upgrade: Complete! 2024-04-14T14:22:49.067 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:22:48 smithi069 ceph-mon[88636]: from='client.25417 -' entity='client.iscsi.foo.smithi069.yuumtr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T14:22:49.067 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:22:48 smithi069 ceph-mon[88636]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:22:49.067 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:22:48 smithi069 ceph-mon[88636]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:22:49.067 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:22:48 smithi069 ceph-mon[88636]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T14:22:49.067 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:22:48 smithi069 ceph-mon[88636]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-14T14:22:49.067 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:22:48 smithi069 ceph-mon[88636]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:22:49.067 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:22:48 smithi069 ceph-mon[88636]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T14:22:49.067 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:22:48 smithi069 ceph-mon[88636]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-14T14:22:49.067 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:22:48 smithi069 ceph-mon[88636]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:22:49.068 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:22:48 smithi069 ceph-mon[83680]: Upgrade: Setting container_image for all nvmeof 2024-04-14T14:22:49.068 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:22:48 smithi069 ceph-mon[83680]: Upgrade: Finalizing container_image settings 2024-04-14T14:22:49.068 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:22:48 smithi069 ceph-mon[83680]: Upgrade: Complete! 2024-04-14T14:22:49.068 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:22:48 smithi069 ceph-mon[83680]: from='client.25417 -' entity='client.iscsi.foo.smithi069.yuumtr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T14:22:49.068 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:22:48 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:22:49.068 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:22:48 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:22:49.068 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:22:48 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T14:22:49.068 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:22:48 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-14T14:22:49.068 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:22:48 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:22:49.068 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:22:48 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T14:22:49.068 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:22:48 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-14T14:22:49.069 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:22:48 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:22:50.067 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:22:49 smithi069 ceph-mon[83680]: pgmap v83: 161 pgs: 161 active+clean; 585 KiB data, 137 MiB used, 715 GiB / 715 GiB avail; 245 KiB/s rd, 246 op/s; 64 B/s, 0 objects/s recovering 2024-04-14T14:22:50.067 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:22:49 smithi069 ceph-mon[83680]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 84/627 objects degraded (13.397%), 21 pgs degraded) 2024-04-14T14:22:50.067 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:22:49 smithi069 ceph-mon[83680]: Cluster is now healthy 2024-04-14T14:22:50.067 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:22:49 smithi069 ceph-mon[88636]: pgmap v83: 161 pgs: 161 active+clean; 585 KiB data, 137 MiB used, 715 GiB / 715 GiB avail; 245 KiB/s rd, 246 op/s; 64 B/s, 0 objects/s recovering 2024-04-14T14:22:50.067 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:22:49 smithi069 ceph-mon[88636]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 84/627 objects degraded (13.397%), 21 pgs degraded) 2024-04-14T14:22:50.067 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:22:49 smithi069 ceph-mon[88636]: Cluster is now healthy 2024-04-14T14:22:50.094 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:22:49 smithi122 ceph-mon[67998]: pgmap v83: 161 pgs: 161 active+clean; 585 KiB data, 137 MiB used, 715 GiB / 715 GiB avail; 245 KiB/s rd, 246 op/s; 64 B/s, 0 objects/s recovering 2024-04-14T14:22:50.094 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:22:49 smithi122 ceph-mon[67998]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 84/627 objects degraded (13.397%), 21 pgs degraded) 2024-04-14T14:22:50.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:22:49 smithi122 ceph-mon[67998]: Cluster is now healthy 2024-04-14T14:22:52.070 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:22:51 smithi069 ceph-mon[88636]: pgmap v84: 161 pgs: 161 active+clean; 585 KiB data, 137 MiB used, 715 GiB / 715 GiB avail; 196 KiB/s rd, 197 op/s; 51 B/s, 0 objects/s recovering 2024-04-14T14:22:52.070 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:22:51 smithi069 ceph-mon[88636]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:22:52.070 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:22:51 smithi069 ceph-mon[88636]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T14:22:52.070 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:22:51 smithi069 ceph-mon[83680]: pgmap v84: 161 pgs: 161 active+clean; 585 KiB data, 137 MiB used, 715 GiB / 715 GiB avail; 196 KiB/s rd, 197 op/s; 51 B/s, 0 objects/s recovering 2024-04-14T14:22:52.070 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:22:51 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:22:52.070 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:22:51 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T14:22:52.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:22:51 smithi122 ceph-mon[67998]: pgmap v84: 161 pgs: 161 active+clean; 585 KiB data, 137 MiB used, 715 GiB / 715 GiB avail; 196 KiB/s rd, 197 op/s; 51 B/s, 0 objects/s recovering 2024-04-14T14:22:52.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:22:51 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:22:52.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:22:51 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T14:22:53.067 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:22:52 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:22:53.067 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:22:52 smithi069 ceph-mon[88636]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:22:53.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:22:52 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:22:54.067 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:22:53 smithi069 ceph-mon[83680]: pgmap v85: 161 pgs: 161 active+clean; 585 KiB data, 137 MiB used, 715 GiB / 715 GiB avail; 157 KiB/s rd, 158 op/s; 41 B/s, 0 objects/s recovering 2024-04-14T14:22:54.067 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:22:53 smithi069 ceph-mon[88636]: pgmap v85: 161 pgs: 161 active+clean; 585 KiB data, 137 MiB used, 715 GiB / 715 GiB avail; 157 KiB/s rd, 158 op/s; 41 B/s, 0 objects/s recovering 2024-04-14T14:22:54.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:22:53 smithi122 ceph-mon[67998]: pgmap v85: 161 pgs: 161 active+clean; 585 KiB data, 137 MiB used, 715 GiB / 715 GiB avail; 157 KiB/s rd, 158 op/s; 41 B/s, 0 objects/s recovering 2024-04-14T14:22:55.817 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:22:55 smithi069 ceph-mon[83680]: pgmap v86: 161 pgs: 161 active+clean; 585 KiB data, 137 MiB used, 715 GiB / 715 GiB avail; 142 KiB/s rd, 143 op/s; 37 B/s, 0 objects/s recovering 2024-04-14T14:22:55.817 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:22:55 smithi069 ceph-mon[88636]: pgmap v86: 161 pgs: 161 active+clean; 585 KiB data, 137 MiB used, 715 GiB / 715 GiB avail; 142 KiB/s rd, 143 op/s; 37 B/s, 0 objects/s recovering 2024-04-14T14:22:55.975 DEBUG:teuthology.orchestra.run.smithi069:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid dc326fc2-fa65-11ee-bc8e-c7b262605968 -e sha1=a971fd5f5719038c70b817bafacd6d5400b1d888 -- bash -c 'ceph orch ps' 2024-04-14T14:22:56.094 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:22:55 smithi122 ceph-mon[67998]: pgmap v86: 161 pgs: 161 active+clean; 585 KiB data, 137 MiB used, 715 GiB / 715 GiB avail; 142 KiB/s rd, 143 op/s; 37 B/s, 0 objects/s recovering 2024-04-14T14:22:56.561 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 14 14:22:56 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-y[63038]: ::ffff:172.21.15.122 - - [14/Apr/2024:14:22:56] "GET /metrics HTTP/1.1" 200 38178 "" "Prometheus/2.43.0" 2024-04-14T14:22:56.817 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:22:56 smithi069 ceph-mon[83680]: from='client.54223 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T14:22:56.817 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:22:56 smithi069 ceph-mon[88636]: from='client.54223 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T14:22:57.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:22:56 smithi122 ceph-mon[67998]: from='client.54223 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T14:22:58.067 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:22:57 smithi069 ceph-mon[83680]: pgmap v87: 161 pgs: 161 active+clean; 585 KiB data, 137 MiB used, 715 GiB / 715 GiB avail; 123 KiB/s rd, 123 op/s; 32 B/s, 0 objects/s recovering 2024-04-14T14:22:58.067 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:22:57 smithi069 ceph-mon[88636]: pgmap v87: 161 pgs: 161 active+clean; 585 KiB data, 137 MiB used, 715 GiB / 715 GiB avail; 123 KiB/s rd, 123 op/s; 32 B/s, 0 objects/s recovering 2024-04-14T14:22:58.088 INFO:teuthology.orchestra.run.smithi069.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-14T14:22:58.088 INFO:teuthology.orchestra.run.smithi069.stdout:alertmanager.a smithi069 *:9093,9094 running (4m) 15s ago 26m 17.4M - 0.25.0 c8568f914cd2 7524f106e7bd 2024-04-14T14:22:58.088 INFO:teuthology.orchestra.run.smithi069.stdout:grafana.a smithi122 *:3000 running (4m) 2m ago 26m 81.0M - 9.4.12 2bacad6d85d8 5685178e3eba 2024-04-14T14:22:58.088 INFO:teuthology.orchestra.run.smithi069.stdout:iscsi.foo.smithi069.yuumtr smithi069 running (5m) 15s ago 25m 59.4M - 3.5 e1d6a67b021e 38cc81ea8cfa 2024-04-14T14:22:58.088 INFO:teuthology.orchestra.run.smithi069.stdout:mgr.x smithi122 *:8443,9283,8765 running (5m) 2m ago 30m 450M - 19.0.0-2778-ga971fd5f b96c509eef84 e29977590b43 2024-04-14T14:22:58.088 INFO:teuthology.orchestra.run.smithi069.stdout:mgr.y smithi069 *:8443,9283,8765 running (15m) 15s ago 32m 502M - 19.0.0-2778-ga971fd5f b96c509eef84 7af3d29d0e5f 2024-04-14T14:22:58.088 INFO:teuthology.orchestra.run.smithi069.stdout:mon.a smithi069 running (2m) 15s ago 32m 50.6M 2048M 19.0.0-2778-ga971fd5f b96c509eef84 8bcd6b3d3846 2024-04-14T14:22:58.088 INFO:teuthology.orchestra.run.smithi069.stdout:mon.b smithi122 running (3m) 2m ago 31m 33.9M 2048M 19.0.0-2778-ga971fd5f b96c509eef84 05f437a1a9a1 2024-04-14T14:22:58.088 INFO:teuthology.orchestra.run.smithi069.stdout:mon.c smithi069 running (2m) 15s ago 31m 34.8M 2048M 19.0.0-2778-ga971fd5f b96c509eef84 84aa95f11d0c 2024-04-14T14:22:58.088 INFO:teuthology.orchestra.run.smithi069.stdout:node-exporter.a smithi069 *:9100 running (5m) 15s ago 27m 8434k - 1.5.0 0da6a335fe13 0de0eb2171b9 2024-04-14T14:22:58.088 INFO:teuthology.orchestra.run.smithi069.stdout:node-exporter.b smithi122 *:9100 running (5m) 2m ago 27m 16.1M - 1.5.0 0da6a335fe13 67877bf02d8e 2024-04-14T14:22:58.088 INFO:teuthology.orchestra.run.smithi069.stdout:osd.0 smithi069 running (54s) 15s ago 30m 51.5M 2992M 19.0.0-2778-ga971fd5f b96c509eef84 a0e57e3fb494 2024-04-14T14:22:58.089 INFO:teuthology.orchestra.run.smithi069.stdout:osd.1 smithi069 running (20s) 15s ago 30m 11.7M 2992M 19.0.0-2778-ga971fd5f b96c509eef84 cf73d17919bb 2024-04-14T14:22:58.089 INFO:teuthology.orchestra.run.smithi069.stdout:osd.2 smithi069 running (29m) 15s ago 29m 52.0M 2992M 17.2.0 e1d6a67b021e bc37a1a9cba0 2024-04-14T14:22:58.089 INFO:teuthology.orchestra.run.smithi069.stdout:osd.3 smithi069 running (29m) 15s ago 29m 57.1M 2992M 17.2.0 e1d6a67b021e 0925ba3da801 2024-04-14T14:22:58.089 INFO:teuthology.orchestra.run.smithi069.stdout:osd.4 smithi122 running (29m) 2m ago 29m 56.1M 3248M 17.2.0 e1d6a67b021e c00975caf771 2024-04-14T14:22:58.089 INFO:teuthology.orchestra.run.smithi069.stdout:osd.5 smithi122 running (28m) 2m ago 28m 55.2M 3248M 17.2.0 e1d6a67b021e 5f52d217ceb8 2024-04-14T14:22:58.089 INFO:teuthology.orchestra.run.smithi069.stdout:osd.6 smithi122 running (28m) 2m ago 28m 52.4M 3248M 17.2.0 e1d6a67b021e a2acfec61475 2024-04-14T14:22:58.089 INFO:teuthology.orchestra.run.smithi069.stdout:osd.7 smithi122 running (27m) 2m ago 27m 57.1M 3248M 17.2.0 e1d6a67b021e 62411cd06801 2024-04-14T14:22:58.089 INFO:teuthology.orchestra.run.smithi069.stdout:prometheus.a smithi122 *:9095 running (4m) 2m ago 27m 66.4M - 2.43.0 a07b618ecd1d 5c1f41d3bbe6 2024-04-14T14:22:58.089 INFO:teuthology.orchestra.run.smithi069.stdout:rgw.foo.smithi069.wesedz smithi069 *:8000 running (25m) 15s ago 25m 79.2M - 17.2.0 e1d6a67b021e 16de491440f3 2024-04-14T14:22:58.089 INFO:teuthology.orchestra.run.smithi069.stdout:rgw.foo.smithi122.jiobyv smithi122 *:8000 running (25m) 2m ago 25m 75.8M - 17.2.0 e1d6a67b021e 6d16de445dba 2024-04-14T14:22:58.094 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:22:57 smithi122 ceph-mon[67998]: pgmap v87: 161 pgs: 161 active+clean; 585 KiB data, 137 MiB used, 715 GiB / 715 GiB avail; 123 KiB/s rd, 123 op/s; 32 B/s, 0 objects/s recovering 2024-04-14T14:22:58.771 DEBUG:teuthology.orchestra.run.smithi069:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid dc326fc2-fa65-11ee-bc8e-c7b262605968 -e sha1=a971fd5f5719038c70b817bafacd6d5400b1d888 -- bash -c 'ceph versions | jq -e '"'"'.osd | length == 2'"'"'' 2024-04-14T14:22:58.816 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:22:58 smithi069 ceph-mon[83680]: from='client.25417 -' entity='client.iscsi.foo.smithi069.yuumtr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T14:22:58.817 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:22:58 smithi069 ceph-mon[88636]: from='client.25417 -' entity='client.iscsi.foo.smithi069.yuumtr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T14:22:59.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:22:58 smithi122 ceph-mon[67998]: from='client.25417 -' entity='client.iscsi.foo.smithi069.yuumtr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T14:22:59.754 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:22:59 smithi069 ceph-mon[88636]: from='client.54229 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T14:22:59.755 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:22:59 smithi069 ceph-mon[88636]: pgmap v88: 161 pgs: 161 active+clean; 585 KiB data, 137 MiB used, 715 GiB / 715 GiB avail; 123 KiB/s rd, 124 op/s; 32 B/s, 0 objects/s recovering 2024-04-14T14:22:59.755 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:22:59 smithi069 ceph-mon[83680]: from='client.54229 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T14:22:59.755 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:22:59 smithi069 ceph-mon[83680]: pgmap v88: 161 pgs: 161 active+clean; 585 KiB data, 137 MiB used, 715 GiB / 715 GiB avail; 123 KiB/s rd, 124 op/s; 32 B/s, 0 objects/s recovering 2024-04-14T14:23:00.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:22:59 smithi122 ceph-mon[67998]: from='client.54229 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T14:23:00.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:22:59 smithi122 ceph-mon[67998]: pgmap v88: 161 pgs: 161 active+clean; 585 KiB data, 137 MiB used, 715 GiB / 715 GiB avail; 123 KiB/s rd, 124 op/s; 32 B/s, 0 objects/s recovering 2024-04-14T14:23:00.768 INFO:teuthology.orchestra.run.smithi069.stdout:true 2024-04-14T14:23:01.420 DEBUG:teuthology.orchestra.run.smithi069:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid dc326fc2-fa65-11ee-bc8e-c7b262605968 -e sha1=a971fd5f5719038c70b817bafacd6d5400b1d888 -- bash -c 'ceph orch upgrade check quay.ceph.io/ceph-ci/ceph:$sha1 | jq -e '"'"'.up_to_date | length == 7'"'"'' 2024-04-14T14:23:01.647 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:23:01 smithi069 ceph-mon[83680]: pgmap v89: 161 pgs: 161 active+clean; 585 KiB data, 137 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:23:01.647 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:23:01 smithi069 ceph-mon[83680]: from='client.? 172.21.15.69:0/1883382385' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T14:23:01.897 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:23:01 smithi122 ceph-mon[67998]: pgmap v89: 161 pgs: 161 active+clean; 585 KiB data, 137 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:23:01.897 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:23:01 smithi122 ceph-mon[67998]: from='client.? 172.21.15.69:0/1883382385' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T14:23:01.939 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:23:01 smithi069 ceph-mon[88636]: pgmap v89: 161 pgs: 161 active+clean; 585 KiB data, 137 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:23:01.939 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:23:01 smithi069 ceph-mon[88636]: from='client.? 172.21.15.69:0/1883382385' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T14:23:03.930 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:23:03 smithi069 ceph-mon[83680]: pgmap v90: 161 pgs: 161 active+clean; 585 KiB data, 137 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:23:03.930 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:23:03 smithi069 ceph-mon[88636]: pgmap v90: 161 pgs: 161 active+clean; 585 KiB data, 137 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:23:04.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:23:03 smithi122 ceph-mon[67998]: pgmap v90: 161 pgs: 161 active+clean; 585 KiB data, 137 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:23:04.817 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:23:04 smithi069 ceph-mon[83680]: from='client.44304 -' entity='client.admin' cmd=[{"prefix": "orch upgrade check", "image": "quay.ceph.io/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T14:23:04.817 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:23:04 smithi069 ceph-mon[88636]: from='client.44304 -' entity='client.admin' cmd=[{"prefix": "orch upgrade check", "image": "quay.ceph.io/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T14:23:04.960 INFO:teuthology.orchestra.run.smithi069.stdout:true 2024-04-14T14:23:05.094 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:23:04 smithi122 ceph-mon[67998]: from='client.44304 -' entity='client.admin' cmd=[{"prefix": "orch upgrade check", "image": "quay.ceph.io/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T14:23:05.597 DEBUG:teuthology.orchestra.run.smithi069:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid dc326fc2-fa65-11ee-bc8e-c7b262605968 -e sha1=a971fd5f5719038c70b817bafacd6d5400b1d888 -- bash -c 'ceph orch upgrade status' 2024-04-14T14:23:05.903 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:23:05 smithi069 ceph-mon[83680]: pgmap v91: 161 pgs: 161 active+clean; 585 KiB data, 137 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:23:05.904 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:23:05 smithi069 ceph-mon[88636]: pgmap v91: 161 pgs: 161 active+clean; 585 KiB data, 137 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:23:06.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:23:05 smithi122 ceph-mon[67998]: pgmap v91: 161 pgs: 161 active+clean; 585 KiB data, 137 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:23:06.805 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:23:06 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T14:23:06.805 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 14 14:23:06 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-y[63038]: ::ffff:172.21.15.122 - - [14/Apr/2024:14:23:06] "GET /metrics HTTP/1.1" 200 38178 "" "Prometheus/2.43.0" 2024-04-14T14:23:06.806 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:23:06 smithi069 ceph-mon[88636]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T14:23:07.094 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:23:06 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T14:23:07.495 INFO:teuthology.orchestra.run.smithi069.stdout:{ 2024-04-14T14:23:07.496 INFO:teuthology.orchestra.run.smithi069.stdout: "target_image": null, 2024-04-14T14:23:07.496 INFO:teuthology.orchestra.run.smithi069.stdout: "in_progress": false, 2024-04-14T14:23:07.496 INFO:teuthology.orchestra.run.smithi069.stdout: "which": "", 2024-04-14T14:23:07.496 INFO:teuthology.orchestra.run.smithi069.stdout: "services_complete": [], 2024-04-14T14:23:07.496 INFO:teuthology.orchestra.run.smithi069.stdout: "progress": null, 2024-04-14T14:23:07.496 INFO:teuthology.orchestra.run.smithi069.stdout: "message": "", 2024-04-14T14:23:07.496 INFO:teuthology.orchestra.run.smithi069.stdout: "is_paused": false 2024-04-14T14:23:07.496 INFO:teuthology.orchestra.run.smithi069.stdout:} 2024-04-14T14:23:07.817 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:23:07 smithi069 ceph-mon[83680]: pgmap v92: 161 pgs: 161 active+clean; 585 KiB data, 137 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:23:07.817 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:23:07 smithi069 ceph-mon[88636]: pgmap v92: 161 pgs: 161 active+clean; 585 KiB data, 137 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:23:08.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:23:07 smithi122 ceph-mon[67998]: pgmap v92: 161 pgs: 161 active+clean; 585 KiB data, 137 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:23:08.145 DEBUG:teuthology.orchestra.run.smithi069:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid dc326fc2-fa65-11ee-bc8e-c7b262605968 -e sha1=a971fd5f5719038c70b817bafacd6d5400b1d888 -- bash -c 'ceph health detail' 2024-04-14T14:23:08.724 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:23:08 smithi069 ceph-mon[88636]: from='client.44310 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T14:23:08.724 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:23:08 smithi069 ceph-mon[88636]: from='client.25417 -' entity='client.iscsi.foo.smithi069.yuumtr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T14:23:08.724 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:23:08 smithi069 ceph-mon[88636]: pgmap v93: 161 pgs: 161 active+clean; 585 KiB data, 137 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:23:08.724 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:23:08 smithi069 ceph-mon[83680]: from='client.44310 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T14:23:08.725 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:23:08 smithi069 ceph-mon[83680]: from='client.25417 -' entity='client.iscsi.foo.smithi069.yuumtr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T14:23:08.725 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:23:08 smithi069 ceph-mon[83680]: pgmap v93: 161 pgs: 161 active+clean; 585 KiB data, 137 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:23:09.094 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:23:08 smithi122 ceph-mon[67998]: from='client.44310 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T14:23:09.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:23:08 smithi122 ceph-mon[67998]: from='client.25417 -' entity='client.iscsi.foo.smithi069.yuumtr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T14:23:09.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:23:08 smithi122 ceph-mon[67998]: pgmap v93: 161 pgs: 161 active+clean; 585 KiB data, 137 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:23:10.216 INFO:teuthology.orchestra.run.smithi069.stdout:HEALTH_OK 2024-04-14T14:23:10.567 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:23:10 smithi069 ceph-mon[83680]: from='client.? 172.21.15.69:0/1133783396' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-14T14:23:10.567 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:23:10 smithi069 ceph-mon[88636]: from='client.? 172.21.15.69:0/1133783396' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-14T14:23:10.595 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:23:10 smithi122 ceph-mon[67998]: from='client.? 172.21.15.69:0/1133783396' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-14T14:23:10.914 DEBUG:teuthology.orchestra.run.smithi069:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid dc326fc2-fa65-11ee-bc8e-c7b262605968 -e sha1=a971fd5f5719038c70b817bafacd6d5400b1d888 -- bash -c 'ceph orch upgrade start --image quay.ceph.io/ceph-ci/ceph:$sha1 --daemon-types crash,osd --limit 1' 2024-04-14T14:23:11.560 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:23:11 smithi069 ceph-mon[83680]: pgmap v94: 161 pgs: 161 active+clean; 585 KiB data, 137 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:23:11.560 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:23:11 smithi069 ceph-mon[88636]: pgmap v94: 161 pgs: 161 active+clean; 585 KiB data, 137 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:23:11.594 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:23:11 smithi122 ceph-mon[67998]: pgmap v94: 161 pgs: 161 active+clean; 585 KiB data, 137 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:23:13.862 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:23:13 smithi069 ceph-mon[88636]: pgmap v95: 161 pgs: 161 active+clean; 585 KiB data, 137 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:23:13.863 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:23:13 smithi069 ceph-mon[83680]: pgmap v95: 161 pgs: 161 active+clean; 585 KiB data, 137 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:23:14.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:23:13 smithi122 ceph-mon[67998]: pgmap v95: 161 pgs: 161 active+clean; 585 KiB data, 137 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:23:14.712 INFO:teuthology.orchestra.run.smithi069.stdout:Initiating upgrade to quay.ceph.io/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888 2024-04-14T14:23:15.067 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:23:14 smithi069 ceph-mon[83680]: from='client.44322 -' entity='client.admin' cmd=[{"prefix": "orch upgrade start", "image": "quay.ceph.io/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888", "daemon_types": "crash,osd", "limit": 1, "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T14:23:15.067 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:23:14 smithi069 ceph-mon[88636]: from='client.44322 -' entity='client.admin' cmd=[{"prefix": "orch upgrade start", "image": "quay.ceph.io/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888", "daemon_types": "crash,osd", "limit": 1, "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T14:23:15.094 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:23:14 smithi122 ceph-mon[67998]: from='client.44322 -' entity='client.admin' cmd=[{"prefix": "orch upgrade start", "image": "quay.ceph.io/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888", "daemon_types": "crash,osd", "limit": 1, "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T14:23:15.489 DEBUG:teuthology.orchestra.run.smithi069:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid dc326fc2-fa65-11ee-bc8e-c7b262605968 -e sha1=a971fd5f5719038c70b817bafacd6d5400b1d888 -- bash -c 'while ceph orch upgrade status | jq '"'"'.in_progress'"'"' | grep true && ! ceph orch upgrade status | jq '"'"'.message'"'"' | grep Error ; do ceph orch ps ; ceph versions ; ceph orch upgrade status ; sleep 30 ; done' 2024-04-14T14:23:15.800 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:23:15 smithi069 ceph-mon[83680]: pgmap v96: 161 pgs: 161 active+clean; 585 KiB data, 137 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:23:15.800 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:23:15 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:23:15.800 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:23:15 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T14:23:15.800 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:23:15 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-14T14:23:15.800 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:23:15 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:23:15.801 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:23:15 smithi069 ceph-mon[88636]: pgmap v96: 161 pgs: 161 active+clean; 585 KiB data, 137 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:23:15.801 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:23:15 smithi069 ceph-mon[88636]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:23:15.801 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:23:15 smithi069 ceph-mon[88636]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T14:23:15.801 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:23:15 smithi069 ceph-mon[88636]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-14T14:23:15.801 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:23:15 smithi069 ceph-mon[88636]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:23:16.098 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:23:15 smithi122 ceph-mon[67998]: pgmap v96: 161 pgs: 161 active+clean; 585 KiB data, 137 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:23:16.098 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:23:15 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:23:16.098 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:23:15 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T14:23:16.098 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:23:15 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-14T14:23:16.098 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:23:15 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:23:16.778 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:23:16 smithi069 ceph-mon[83680]: Upgrade: Started with target quay.ceph.io/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888 2024-04-14T14:23:16.778 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:23:16 smithi069 ceph-mon[83680]: Upgrade: First pull of quay.ceph.io/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888 2024-04-14T14:23:16.778 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 14 14:23:16 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-y[63038]: ::ffff:172.21.15.122 - - [14/Apr/2024:14:23:16] "GET /metrics HTTP/1.1" 200 38177 "" "Prometheus/2.43.0" 2024-04-14T14:23:16.779 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:23:16 smithi069 ceph-mon[88636]: Upgrade: Started with target quay.ceph.io/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888 2024-04-14T14:23:16.779 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:23:16 smithi069 ceph-mon[88636]: Upgrade: First pull of quay.ceph.io/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888 2024-04-14T14:23:17.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:23:16 smithi122 ceph-mon[67998]: Upgrade: Started with target quay.ceph.io/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888 2024-04-14T14:23:17.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:23:16 smithi122 ceph-mon[67998]: Upgrade: First pull of quay.ceph.io/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888 2024-04-14T14:23:17.706 INFO:teuthology.orchestra.run.smithi069.stdout:true 2024-04-14T14:23:18.012 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:23:17 smithi069 ceph-mon[88636]: pgmap v97: 161 pgs: 161 active+clean; 585 KiB data, 137 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:23:18.013 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:23:17 smithi069 ceph-mon[83680]: pgmap v97: 161 pgs: 161 active+clean; 585 KiB data, 137 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:23:18.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:23:17 smithi122 ceph-mon[67998]: pgmap v97: 161 pgs: 161 active+clean; 585 KiB data, 137 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:23:18.400 INFO:teuthology.orchestra.run.smithi069.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-14T14:23:18.401 INFO:teuthology.orchestra.run.smithi069.stdout:alertmanager.a smithi069 *:9093,9094 running (5m) 35s ago 26m 17.4M - 0.25.0 c8568f914cd2 7524f106e7bd 2024-04-14T14:23:18.401 INFO:teuthology.orchestra.run.smithi069.stdout:grafana.a smithi122 *:3000 running (4m) 2m ago 26m 81.0M - 9.4.12 2bacad6d85d8 5685178e3eba 2024-04-14T14:23:18.401 INFO:teuthology.orchestra.run.smithi069.stdout:iscsi.foo.smithi069.yuumtr smithi069 running (6m) 35s ago 25m 59.4M - 3.5 e1d6a67b021e 38cc81ea8cfa 2024-04-14T14:23:18.401 INFO:teuthology.orchestra.run.smithi069.stdout:mgr.x smithi122 *:8443,9283,8765 running (5m) 2m ago 31m 450M - 19.0.0-2778-ga971fd5f b96c509eef84 e29977590b43 2024-04-14T14:23:18.401 INFO:teuthology.orchestra.run.smithi069.stdout:mgr.y smithi069 *:8443,9283,8765 running (15m) 35s ago 32m 502M - 19.0.0-2778-ga971fd5f b96c509eef84 7af3d29d0e5f 2024-04-14T14:23:18.401 INFO:teuthology.orchestra.run.smithi069.stdout:mon.a smithi069 running (2m) 35s ago 32m 50.6M 2048M 19.0.0-2778-ga971fd5f b96c509eef84 8bcd6b3d3846 2024-04-14T14:23:18.401 INFO:teuthology.orchestra.run.smithi069.stdout:mon.b smithi122 running (3m) 2m ago 31m 33.9M 2048M 19.0.0-2778-ga971fd5f b96c509eef84 05f437a1a9a1 2024-04-14T14:23:18.401 INFO:teuthology.orchestra.run.smithi069.stdout:mon.c smithi069 running (2m) 35s ago 31m 34.8M 2048M 19.0.0-2778-ga971fd5f b96c509eef84 84aa95f11d0c 2024-04-14T14:23:18.401 INFO:teuthology.orchestra.run.smithi069.stdout:node-exporter.a smithi069 *:9100 running (5m) 35s ago 27m 8434k - 1.5.0 0da6a335fe13 0de0eb2171b9 2024-04-14T14:23:18.401 INFO:teuthology.orchestra.run.smithi069.stdout:node-exporter.b smithi122 *:9100 running (5m) 2m ago 27m 16.1M - 1.5.0 0da6a335fe13 67877bf02d8e 2024-04-14T14:23:18.401 INFO:teuthology.orchestra.run.smithi069.stdout:osd.0 smithi069 running (75s) 35s ago 30m 51.5M 2992M 19.0.0-2778-ga971fd5f b96c509eef84 a0e57e3fb494 2024-04-14T14:23:18.401 INFO:teuthology.orchestra.run.smithi069.stdout:osd.1 smithi069 running (40s) 35s ago 30m 11.7M 2992M 19.0.0-2778-ga971fd5f b96c509eef84 cf73d17919bb 2024-04-14T14:23:18.401 INFO:teuthology.orchestra.run.smithi069.stdout:osd.2 smithi069 running (30m) 35s ago 30m 52.0M 2992M 17.2.0 e1d6a67b021e bc37a1a9cba0 2024-04-14T14:23:18.401 INFO:teuthology.orchestra.run.smithi069.stdout:osd.3 smithi069 running (29m) 35s ago 29m 57.1M 2992M 17.2.0 e1d6a67b021e 0925ba3da801 2024-04-14T14:23:18.402 INFO:teuthology.orchestra.run.smithi069.stdout:osd.4 smithi122 running (29m) 2m ago 29m 56.1M 3248M 17.2.0 e1d6a67b021e c00975caf771 2024-04-14T14:23:18.402 INFO:teuthology.orchestra.run.smithi069.stdout:osd.5 smithi122 running (29m) 2m ago 29m 55.2M 3248M 17.2.0 e1d6a67b021e 5f52d217ceb8 2024-04-14T14:23:18.402 INFO:teuthology.orchestra.run.smithi069.stdout:osd.6 smithi122 running (28m) 2m ago 28m 52.4M 3248M 17.2.0 e1d6a67b021e a2acfec61475 2024-04-14T14:23:18.402 INFO:teuthology.orchestra.run.smithi069.stdout:osd.7 smithi122 running (28m) 2m ago 28m 57.1M 3248M 17.2.0 e1d6a67b021e 62411cd06801 2024-04-14T14:23:18.402 INFO:teuthology.orchestra.run.smithi069.stdout:prometheus.a smithi122 *:9095 running (5m) 2m ago 27m 66.4M - 2.43.0 a07b618ecd1d 5c1f41d3bbe6 2024-04-14T14:23:18.402 INFO:teuthology.orchestra.run.smithi069.stdout:rgw.foo.smithi069.wesedz smithi069 *:8000 running (26m) 35s ago 26m 79.2M - 17.2.0 e1d6a67b021e 16de491440f3 2024-04-14T14:23:18.402 INFO:teuthology.orchestra.run.smithi069.stdout:rgw.foo.smithi122.jiobyv smithi122 *:8000 running (25m) 2m ago 25m 75.8M - 17.2.0 e1d6a67b021e 6d16de445dba 2024-04-14T14:23:18.815 INFO:teuthology.orchestra.run.smithi069.stdout:{ 2024-04-14T14:23:18.815 INFO:teuthology.orchestra.run.smithi069.stdout: "mon": { 2024-04-14T14:23:18.815 INFO:teuthology.orchestra.run.smithi069.stdout: "ceph version 19.0.0-2778-ga971fd5f (a971fd5f5719038c70b817bafacd6d5400b1d888) squid (dev)": 3 2024-04-14T14:23:18.816 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-14T14:23:18.816 INFO:teuthology.orchestra.run.smithi069.stdout: "mgr": { 2024-04-14T14:23:18.816 INFO:teuthology.orchestra.run.smithi069.stdout: "ceph version 19.0.0-2778-ga971fd5f (a971fd5f5719038c70b817bafacd6d5400b1d888) squid (dev)": 2 2024-04-14T14:23:18.816 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-14T14:23:18.816 INFO:teuthology.orchestra.run.smithi069.stdout: "osd": { 2024-04-14T14:23:18.816 INFO:teuthology.orchestra.run.smithi069.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 6, 2024-04-14T14:23:18.816 INFO:teuthology.orchestra.run.smithi069.stdout: "ceph version 19.0.0-2778-ga971fd5f (a971fd5f5719038c70b817bafacd6d5400b1d888) squid (dev)": 2 2024-04-14T14:23:18.816 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-14T14:23:18.816 INFO:teuthology.orchestra.run.smithi069.stdout: "rgw": { 2024-04-14T14:23:18.816 INFO:teuthology.orchestra.run.smithi069.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-04-14T14:23:18.816 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-14T14:23:18.816 INFO:teuthology.orchestra.run.smithi069.stdout: "overall": { 2024-04-14T14:23:18.816 INFO:teuthology.orchestra.run.smithi069.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 8, 2024-04-14T14:23:18.816 INFO:teuthology.orchestra.run.smithi069.stdout: "ceph version 19.0.0-2778-ga971fd5f (a971fd5f5719038c70b817bafacd6d5400b1d888) squid (dev)": 7 2024-04-14T14:23:18.817 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-14T14:23:18.817 INFO:teuthology.orchestra.run.smithi069.stdout:} 2024-04-14T14:23:18.994 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:23:18 smithi069 ceph-mon[88636]: from='client.25417 -' entity='client.iscsi.foo.smithi069.yuumtr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T14:23:18.994 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:23:18 smithi069 ceph-mon[88636]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:23:18.995 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:23:18 smithi069 ceph-mon[88636]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-14T14:23:18.995 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:23:18 smithi069 ceph-mon[88636]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T14:23:18.995 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:23:18 smithi069 ceph-mon[88636]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:23:18.995 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:23:18 smithi069 ceph-mon[88636]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T14:23:18.995 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:23:18 smithi069 ceph-mon[88636]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:23:18.995 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:23:18 smithi069 ceph-mon[88636]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T14:23:18.995 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:23:18 smithi069 ceph-mon[88636]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:23:18.995 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:23:18 smithi069 ceph-mon[88636]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16}]: dispatch 2024-04-14T14:23:18.995 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:23:18 smithi069 ceph-mon[83680]: from='client.25417 -' entity='client.iscsi.foo.smithi069.yuumtr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T14:23:18.995 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:23:18 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:23:18.996 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:23:18 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-14T14:23:18.996 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:23:18 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T14:23:18.996 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:23:18 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:23:18.996 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:23:18 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T14:23:18.996 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:23:18 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:23:18.996 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:23:18 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T14:23:18.996 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:23:18 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:23:18.996 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:23:18 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16}]: dispatch 2024-04-14T14:23:19.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:23:18 smithi122 ceph-mon[67998]: from='client.25417 -' entity='client.iscsi.foo.smithi069.yuumtr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T14:23:19.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:23:18 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:23:19.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:23:18 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-14T14:23:19.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:23:18 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T14:23:19.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:23:18 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:23:19.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:23:18 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T14:23:19.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:23:18 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:23:19.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:23:18 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T14:23:19.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:23:18 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:23:19.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:23:18 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16}]: dispatch 2024-04-14T14:23:19.166 INFO:teuthology.orchestra.run.smithi069.stdout:{ 2024-04-14T14:23:19.166 INFO:teuthology.orchestra.run.smithi069.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888", 2024-04-14T14:23:19.166 INFO:teuthology.orchestra.run.smithi069.stdout: "in_progress": true, 2024-04-14T14:23:19.166 INFO:teuthology.orchestra.run.smithi069.stdout: "which": "Upgrading daemons of type(s) crash,osd. Upgrade limited to 1 daemons (1 remaining).", 2024-04-14T14:23:19.166 INFO:teuthology.orchestra.run.smithi069.stdout: "services_complete": [], 2024-04-14T14:23:19.166 INFO:teuthology.orchestra.run.smithi069.stdout: "progress": "2/8 daemons upgraded", 2024-04-14T14:23:19.166 INFO:teuthology.orchestra.run.smithi069.stdout: "message": "Currently upgrading osd daemons", 2024-04-14T14:23:19.167 INFO:teuthology.orchestra.run.smithi069.stdout: "is_paused": false 2024-04-14T14:23:19.167 INFO:teuthology.orchestra.run.smithi069.stdout:} 2024-04-14T14:23:19.817 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:23:19 smithi069 ceph-mon[83680]: from='client.54265 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T14:23:19.817 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:23:19 smithi069 ceph-mon[83680]: from='client.54271 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T14:23:19.817 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:23:19 smithi069 ceph-mon[83680]: Upgrade: Target is version 19.0.0-2778-ga971fd5f (squid) 2024-04-14T14:23:19.817 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:23:19 smithi069 ceph-mon[83680]: Upgrade: Target container is quay.ceph.io/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888, digests ['quay.ceph.io/ceph-ci/ceph@sha256:02f0a191d4d693952654c64f93cc9c0152602c0915251b3c94c2c4e32855c608'] 2024-04-14T14:23:19.817 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:23:19 smithi069 ceph-mon[83680]: Upgrade: Setting container_image for all mgr 2024-04-14T14:23:19.817 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:23:19 smithi069 ceph-mon[83680]: Upgrade: Setting container_image for all mon 2024-04-14T14:23:19.817 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:23:19 smithi069 ceph-mon[83680]: Upgrade: Setting container_image for all crash 2024-04-14T14:23:19.817 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:23:19 smithi069 ceph-mon[83680]: from='client.44340 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T14:23:19.818 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:23:19 smithi069 ceph-mon[83680]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16}]: dispatch 2024-04-14T14:23:19.818 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:23:19 smithi069 ceph-mon[83680]: Upgrade: osd.2 is safe to restart 2024-04-14T14:23:19.818 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:23:19 smithi069 ceph-mon[83680]: pgmap v98: 161 pgs: 161 active+clean; 585 KiB data, 137 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:23:19.818 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:23:19 smithi069 ceph-mon[83680]: from='client.? 172.21.15.69:0/2467403684' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T14:23:19.818 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:23:19 smithi069 ceph-mon[88636]: from='client.54265 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T14:23:19.818 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:23:19 smithi069 ceph-mon[88636]: from='client.54271 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T14:23:19.818 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:23:19 smithi069 ceph-mon[88636]: Upgrade: Target is version 19.0.0-2778-ga971fd5f (squid) 2024-04-14T14:23:19.818 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:23:19 smithi069 ceph-mon[88636]: Upgrade: Target container is quay.ceph.io/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888, digests ['quay.ceph.io/ceph-ci/ceph@sha256:02f0a191d4d693952654c64f93cc9c0152602c0915251b3c94c2c4e32855c608'] 2024-04-14T14:23:19.818 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:23:19 smithi069 ceph-mon[88636]: Upgrade: Setting container_image for all mgr 2024-04-14T14:23:19.818 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:23:19 smithi069 ceph-mon[88636]: Upgrade: Setting container_image for all mon 2024-04-14T14:23:19.818 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:23:19 smithi069 ceph-mon[88636]: Upgrade: Setting container_image for all crash 2024-04-14T14:23:19.819 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:23:19 smithi069 ceph-mon[88636]: from='client.44340 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T14:23:19.819 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:23:19 smithi069 ceph-mon[88636]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16}]: dispatch 2024-04-14T14:23:19.819 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:23:19 smithi069 ceph-mon[88636]: Upgrade: osd.2 is safe to restart 2024-04-14T14:23:19.819 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:23:19 smithi069 ceph-mon[88636]: pgmap v98: 161 pgs: 161 active+clean; 585 KiB data, 137 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:23:19.819 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:23:19 smithi069 ceph-mon[88636]: from='client.? 172.21.15.69:0/2467403684' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T14:23:20.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:23:19 smithi122 ceph-mon[67998]: from='client.54265 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T14:23:20.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:23:19 smithi122 ceph-mon[67998]: from='client.54271 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T14:23:20.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:23:19 smithi122 ceph-mon[67998]: Upgrade: Target is version 19.0.0-2778-ga971fd5f (squid) 2024-04-14T14:23:20.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:23:19 smithi122 ceph-mon[67998]: Upgrade: Target container is quay.ceph.io/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888, digests ['quay.ceph.io/ceph-ci/ceph@sha256:02f0a191d4d693952654c64f93cc9c0152602c0915251b3c94c2c4e32855c608'] 2024-04-14T14:23:20.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:23:19 smithi122 ceph-mon[67998]: Upgrade: Setting container_image for all mgr 2024-04-14T14:23:20.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:23:19 smithi122 ceph-mon[67998]: Upgrade: Setting container_image for all mon 2024-04-14T14:23:20.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:23:19 smithi122 ceph-mon[67998]: Upgrade: Setting container_image for all crash 2024-04-14T14:23:20.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:23:19 smithi122 ceph-mon[67998]: from='client.44340 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T14:23:20.096 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:23:19 smithi122 ceph-mon[67998]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16}]: dispatch 2024-04-14T14:23:20.096 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:23:19 smithi122 ceph-mon[67998]: Upgrade: osd.2 is safe to restart 2024-04-14T14:23:20.096 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:23:19 smithi122 ceph-mon[67998]: pgmap v98: 161 pgs: 161 active+clean; 585 KiB data, 137 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:23:20.096 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:23:19 smithi122 ceph-mon[67998]: from='client.? 172.21.15.69:0/2467403684' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T14:23:20.869 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:23:20 smithi069 ceph-mon[83680]: from='client.54289 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T14:23:20.869 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:23:20 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:23:20.869 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:23:20 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-04-14T14:23:20.869 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:23:20 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T14:23:20.869 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:23:20 smithi069 ceph-mon[88636]: from='client.54289 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T14:23:20.869 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:23:20 smithi069 ceph-mon[88636]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:23:20.870 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:23:20 smithi069 ceph-mon[88636]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-04-14T14:23:20.870 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:23:20 smithi069 ceph-mon[88636]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T14:23:21.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:23:20 smithi122 ceph-mon[67998]: from='client.54289 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T14:23:21.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:23:20 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:23:21.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:23:20 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-04-14T14:23:21.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:23:20 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T14:23:21.798 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:23:21 smithi069 ceph-mon[83680]: Upgrade: Updating osd.2 2024-04-14T14:23:21.798 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:23:21 smithi069 ceph-mon[83680]: Deploying daemon osd.2 on smithi069 2024-04-14T14:23:21.798 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:23:21 smithi069 ceph-mon[83680]: pgmap v99: 161 pgs: 161 active+clean; 585 KiB data, 137 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:23:21.798 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:23:21 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T14:23:21.798 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:23:21 smithi069 ceph-mon[88636]: Upgrade: Updating osd.2 2024-04-14T14:23:21.798 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:23:21 smithi069 ceph-mon[88636]: Deploying daemon osd.2 on smithi069 2024-04-14T14:23:21.799 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:23:21 smithi069 ceph-mon[88636]: pgmap v99: 161 pgs: 161 active+clean; 585 KiB data, 137 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:23:21.799 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:23:21 smithi069 ceph-mon[88636]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T14:23:22.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:23:21 smithi122 ceph-mon[67998]: Upgrade: Updating osd.2 2024-04-14T14:23:22.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:23:21 smithi122 ceph-mon[67998]: Deploying daemon osd.2 on smithi069 2024-04-14T14:23:22.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:23:21 smithi122 ceph-mon[67998]: pgmap v99: 161 pgs: 161 active+clean; 585 KiB data, 137 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:23:22.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:23:21 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T14:23:23.817 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:23:23 smithi069 ceph-mon[88636]: pgmap v100: 161 pgs: 161 active+clean; 585 KiB data, 137 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:23:23.817 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:23:23 smithi069 ceph-mon[88636]: osd.2 marked itself down and dead 2024-04-14T14:23:23.817 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:23:23 smithi069 ceph-mon[83680]: pgmap v100: 161 pgs: 161 active+clean; 585 KiB data, 137 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:23:23.817 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:23:23 smithi069 ceph-mon[83680]: osd.2 marked itself down and dead 2024-04-14T14:23:23.818 INFO:journalctl@ceph.osd.2.smithi069.stdout:Apr 14 14:23:23 smithi069 systemd[1]: Stopping Ceph osd.2 for dc326fc2-fa65-11ee-bc8e-c7b262605968... 2024-04-14T14:23:23.818 INFO:journalctl@ceph.osd.2.smithi069.stdout:Apr 14 14:23:23 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-osd-2[38223]: 2024-04-14T14:23:23.462+0000 7fce07af8700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.2 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-14T14:23:23.818 INFO:journalctl@ceph.osd.2.smithi069.stdout:Apr 14 14:23:23 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-osd-2[38223]: 2024-04-14T14:23:23.462+0000 7fce07af8700 -1 osd.2 125 *** Got signal Terminated *** 2024-04-14T14:23:23.818 INFO:journalctl@ceph.osd.2.smithi069.stdout:Apr 14 14:23:23 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-osd-2[38223]: 2024-04-14T14:23:23.462+0000 7fce07af8700 -1 osd.2 125 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-14T14:23:24.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:23:23 smithi122 ceph-mon[67998]: pgmap v100: 161 pgs: 161 active+clean; 585 KiB data, 137 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:23:24.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:23:23 smithi122 ceph-mon[67998]: osd.2 marked itself down and dead 2024-04-14T14:23:24.989 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:23:24 smithi069 ceph-mon[83680]: Health check failed: 1 osds down (OSD_DOWN) 2024-04-14T14:23:24.989 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:23:24 smithi069 ceph-mon[83680]: osdmap e126: 8 total, 7 up, 8 in 2024-04-14T14:23:24.990 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:23:24 smithi069 ceph-mon[88636]: Health check failed: 1 osds down (OSD_DOWN) 2024-04-14T14:23:24.990 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:23:24 smithi069 ceph-mon[88636]: osdmap e126: 8 total, 7 up, 8 in 2024-04-14T14:23:24.990 INFO:journalctl@ceph.osd.2.smithi069.stdout:Apr 14 14:23:24 smithi069 podman[107136]: 2024-04-14 14:23:24.658667054 +0000 UTC m=+1.307967601 container stop bc37a1a9cba0ba0f6c9977cafce8d2c56a04c1722b52b4006c28d56ee6d8588a (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-osd-2, io.buildah.version=1.19.8, CEPH_POINT_RELEASE=-17.2.0, GIT_REPO=https://github.com/ceph/ceph-container.git, RELEASE=HEAD, name=centos-stream, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, io.k8s.display-name=CentOS Stream 8, release=754, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, version=8, io.openshift.expose-services=, vendor=Red Hat, Inc., description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, com.redhat.component=centos-stream-container, io.openshift.tags=base centos centos-stream, ceph=True, GIT_CLEAN=True, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, GIT_BRANCH=HEAD, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, maintainer=Guillaume Abrioux , vcs-type=git, architecture=x86_64, distribution-scope=public, build-date=2022-05-03T08:36:31.336870, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754) 2024-04-14T14:23:24.990 INFO:journalctl@ceph.osd.2.smithi069.stdout:Apr 14 14:23:24 smithi069 podman[107136]: 2024-04-14 14:23:24.685279334 +0000 UTC m=+1.334579879 container died bc37a1a9cba0ba0f6c9977cafce8d2c56a04c1722b52b4006c28d56ee6d8588a (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-osd-2, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, distribution-scope=public, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, io.buildah.version=1.19.8, maintainer=Guillaume Abrioux , GIT_REPO=https://github.com/ceph/ceph-container.git, architecture=x86_64, io.openshift.tags=base centos centos-stream, GIT_CLEAN=True, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, build-date=2022-05-03T08:36:31.336870, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, vendor=Red Hat, Inc., release=754, name=centos-stream, GIT_BRANCH=HEAD, io.k8s.display-name=CentOS Stream 8, vcs-type=git, io.openshift.expose-services=, com.redhat.component=centos-stream-container, ceph=True, version=8, RELEASE=HEAD, CEPH_POINT_RELEASE=-17.2.0, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly.) 2024-04-14T14:23:25.094 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:23:24 smithi122 ceph-mon[67998]: Health check failed: 1 osds down (OSD_DOWN) 2024-04-14T14:23:25.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:23:24 smithi122 ceph-mon[67998]: osdmap e126: 8 total, 7 up, 8 in 2024-04-14T14:23:25.316 INFO:journalctl@ceph.osd.2.smithi069.stdout:Apr 14 14:23:25 smithi069 podman[107136]: 2024-04-14 14:23:25.195728124 +0000 UTC m=+1.845028677 container cleanup bc37a1a9cba0ba0f6c9977cafce8d2c56a04c1722b52b4006c28d56ee6d8588a (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-osd-2, io.openshift.tags=base centos centos-stream, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , io.k8s.display-name=CentOS Stream 8, io.openshift.expose-services=, release=754, vcs-type=git, com.redhat.component=centos-stream-container, io.buildah.version=1.19.8, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., vendor=Red Hat, Inc., GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, distribution-scope=public, RELEASE=HEAD, name=centos-stream, version=8, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, architecture=x86_64, ceph=True, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, CEPH_POINT_RELEASE=-17.2.0, GIT_CLEAN=True, GIT_REPO=https://github.com/ceph/ceph-container.git, build-date=2022-05-03T08:36:31.336870, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb) 2024-04-14T14:23:25.317 INFO:journalctl@ceph.osd.2.smithi069.stdout:Apr 14 14:23:25 smithi069 bash[107136]: ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-osd-2 2024-04-14T14:23:25.815 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:23:25 smithi069 ceph-mon[83680]: pgmap v102: 161 pgs: 13 stale+active+clean, 148 active+clean; 585 KiB data, 137 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-14T14:23:25.816 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:23:25 smithi069 ceph-mon[83680]: osdmap e127: 8 total, 7 up, 8 in 2024-04-14T14:23:25.816 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:23:25 smithi069 ceph-mon[88636]: pgmap v102: 161 pgs: 13 stale+active+clean, 148 active+clean; 585 KiB data, 137 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-14T14:23:25.816 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:23:25 smithi069 ceph-mon[88636]: osdmap e127: 8 total, 7 up, 8 in 2024-04-14T14:23:25.816 INFO:journalctl@ceph.osd.2.smithi069.stdout:Apr 14 14:23:25 smithi069 podman[107153]: 2024-04-14 14:23:25.397162548 +0000 UTC m=+0.732049036 container remove bc37a1a9cba0ba0f6c9977cafce8d2c56a04c1722b52b4006c28d56ee6d8588a (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-osd-2, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, io.buildah.version=1.19.8, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, com.redhat.component=centos-stream-container, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., io.k8s.display-name=CentOS Stream 8, RELEASE=HEAD, vcs-type=git, maintainer=Guillaume Abrioux , com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, ceph=True, architecture=x86_64, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, GIT_BRANCH=HEAD, distribution-scope=public, version=8, io.openshift.expose-services=, build-date=2022-05-03T08:36:31.336870, GIT_CLEAN=True, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, GIT_REPO=https://github.com/ceph/ceph-container.git, io.openshift.tags=base centos centos-stream, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, vendor=Red Hat, Inc., release=754, CEPH_POINT_RELEASE=-17.2.0, name=centos-stream) 2024-04-14T14:23:26.067 INFO:journalctl@ceph.osd.2.smithi069.stdout:Apr 14 14:23:25 smithi069 podman[107244]: 2024-04-14 14:23:25.81473292 +0000 UTC m=+0.114480476 container create dd743fe1287e68cc014168549caa08c5cc3ce9431b2c5e93399400397681fd3f (image=quay.ceph.io/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888, name=ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-osd-2-deactivate, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_CLEAN=True, RELEASE=main-a971fd5, GIT_BRANCH=HEAD, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.build-date=20240403, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-14T14:23:26.067 INFO:journalctl@ceph.osd.2.smithi069.stdout:Apr 14 14:23:25 smithi069 podman[107244]: 2024-04-14 14:23:25.718380172 +0000 UTC m=+0.018127737 image pull b96c509eef841460b3271f5fdd331860da87b004524f4ed59414aa3680183b53 quay.ceph.io/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888 2024-04-14T14:23:26.067 INFO:journalctl@ceph.osd.2.smithi069.stdout:Apr 14 14:23:26 smithi069 podman[107244]: 2024-04-14 14:23:26.050386723 +0000 UTC m=+0.350134280 container init dd743fe1287e68cc014168549caa08c5cc3ce9431b2c5e93399400397681fd3f (image=quay.ceph.io/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888, name=ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-osd-2-deactivate, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, RELEASE=main-a971fd5, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.build-date=20240403, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-14T14:23:26.067 INFO:journalctl@ceph.osd.2.smithi069.stdout:Apr 14 14:23:26 smithi069 podman[107244]: 2024-04-14 14:23:26.053448842 +0000 UTC m=+0.353196401 container start dd743fe1287e68cc014168549caa08c5cc3ce9431b2c5e93399400397681fd3f (image=quay.ceph.io/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888, name=ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-osd-2-deactivate, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.build-date=20240403, ceph=True, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, RELEASE=main-a971fd5) 2024-04-14T14:23:26.094 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:23:25 smithi122 ceph-mon[67998]: pgmap v102: 161 pgs: 13 stale+active+clean, 148 active+clean; 585 KiB data, 137 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-14T14:23:26.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:23:25 smithi122 ceph-mon[67998]: osdmap e127: 8 total, 7 up, 8 in 2024-04-14T14:23:26.772 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 14 14:23:26 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-y[63038]: ::ffff:172.21.15.122 - - [14/Apr/2024:14:23:26] "GET /metrics HTTP/1.1" 200 38176 "" "Prometheus/2.43.0" 2024-04-14T14:23:26.773 INFO:journalctl@ceph.osd.2.smithi069.stdout:Apr 14 14:23:26 smithi069 podman[107244]: 2024-04-14 14:23:26.325947771 +0000 UTC m=+0.625695342 container attach dd743fe1287e68cc014168549caa08c5cc3ce9431b2c5e93399400397681fd3f (image=quay.ceph.io/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888, name=ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-osd-2-deactivate, io.buildah.version=1.33.5, org.label-schema.build-date=20240403, GIT_CLEAN=True, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=main-a971fd5, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, ceph=True) 2024-04-14T14:23:26.773 INFO:journalctl@ceph.osd.2.smithi069.stdout:Apr 14 14:23:26 smithi069 podman[107244]: 2024-04-14 14:23:26.416713325 +0000 UTC m=+0.716460885 container died dd743fe1287e68cc014168549caa08c5cc3ce9431b2c5e93399400397681fd3f (image=quay.ceph.io/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888, name=ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-osd-2-deactivate, GIT_CLEAN=True, org.label-schema.build-date=20240403, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, ceph=True, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=main-a971fd5, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.vendor=CentOS) 2024-04-14T14:23:27.026 INFO:journalctl@ceph.osd.2.smithi069.stdout:Apr 14 14:23:27 smithi069 podman[107270]: 2024-04-14 14:23:27.004531839 +0000 UTC m=+0.582983013 container remove dd743fe1287e68cc014168549caa08c5cc3ce9431b2c5e93399400397681fd3f (image=quay.ceph.io/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888, name=ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-osd-2-deactivate, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, ceph=True, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, RELEASE=main-a971fd5, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.build-date=20240403) 2024-04-14T14:23:27.317 INFO:journalctl@ceph.osd.2.smithi069.stdout:Apr 14 14:23:27 smithi069 systemd[1]: ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968@osd.2.service: Deactivated successfully. 2024-04-14T14:23:27.317 INFO:journalctl@ceph.osd.2.smithi069.stdout:Apr 14 14:23:27 smithi069 systemd[1]: Stopped Ceph osd.2 for dc326fc2-fa65-11ee-bc8e-c7b262605968. 2024-04-14T14:23:27.317 INFO:journalctl@ceph.osd.2.smithi069.stdout:Apr 14 14:23:27 smithi069 systemd[1]: ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968@osd.2.service: Consumed 12.912s CPU time. 2024-04-14T14:23:27.317 INFO:journalctl@ceph.osd.2.smithi069.stdout:Apr 14 14:23:27 smithi069 systemd[1]: Starting Ceph osd.2 for dc326fc2-fa65-11ee-bc8e-c7b262605968... 2024-04-14T14:23:27.751 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:23:27 smithi069 ceph-mon[83680]: pgmap v104: 161 pgs: 13 stale+active+clean, 148 active+clean; 585 KiB data, 137 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-04-14T14:23:27.751 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:23:27 smithi069 ceph-mon[88636]: pgmap v104: 161 pgs: 13 stale+active+clean, 148 active+clean; 585 KiB data, 137 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-04-14T14:23:27.751 INFO:journalctl@ceph.osd.2.smithi069.stdout:Apr 14 14:23:27 smithi069 podman[107372]: 2024-04-14 14:23:27.455752539 +0000 UTC m=+0.112397328 container create 012fbd15824253e6e8065c1b43520b7d4f0fe5168cd71b04ef424d6bd1dea7d5 (image=quay.ceph.io/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888, name=ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-osd-2-activate, org.label-schema.vendor=CentOS, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.build-date=20240403, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, ceph=True, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=main-a971fd5) 2024-04-14T14:23:27.751 INFO:journalctl@ceph.osd.2.smithi069.stdout:Apr 14 14:23:27 smithi069 podman[107372]: 2024-04-14 14:23:27.361428754 +0000 UTC m=+0.018073559 image pull b96c509eef841460b3271f5fdd331860da87b004524f4ed59414aa3680183b53 quay.ceph.io/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888 2024-04-14T14:23:28.066 INFO:journalctl@ceph.osd.2.smithi069.stdout:Apr 14 14:23:27 smithi069 podman[107372]: 2024-04-14 14:23:27.750138582 +0000 UTC m=+0.406783372 container init 012fbd15824253e6e8065c1b43520b7d4f0fe5168cd71b04ef424d6bd1dea7d5 (image=quay.ceph.io/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888, name=ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-osd-2-activate, org.label-schema.schema-version=1.0, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , RELEASE=main-a971fd5, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.build-date=20240403, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, ceph=True) 2024-04-14T14:23:28.067 INFO:journalctl@ceph.osd.2.smithi069.stdout:Apr 14 14:23:27 smithi069 podman[107372]: 2024-04-14 14:23:27.753653769 +0000 UTC m=+0.410298570 container start 012fbd15824253e6e8065c1b43520b7d4f0fe5168cd71b04ef424d6bd1dea7d5 (image=quay.ceph.io/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888, name=ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-osd-2-activate, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_CLEAN=True, maintainer=Guillaume Abrioux , RELEASE=main-a971fd5, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.build-date=20240403, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True) 2024-04-14T14:23:28.067 INFO:journalctl@ceph.osd.2.smithi069.stdout:Apr 14 14:23:27 smithi069 podman[107372]: 2024-04-14 14:23:27.849892642 +0000 UTC m=+0.506537445 container attach 012fbd15824253e6e8065c1b43520b7d4f0fe5168cd71b04ef424d6bd1dea7d5 (image=quay.ceph.io/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888, name=ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-osd-2-activate, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.build-date=20240403, RELEASE=main-a971fd5, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, io.buildah.version=1.33.5) 2024-04-14T14:23:28.094 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:23:27 smithi122 ceph-mon[67998]: pgmap v104: 161 pgs: 13 stale+active+clean, 148 active+clean; 585 KiB data, 137 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-04-14T14:23:28.554 INFO:journalctl@ceph.osd.2.smithi069.stdout:Apr 14 14:23:28 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-osd-2-activate[107387]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-04-14T14:23:28.554 INFO:journalctl@ceph.osd.2.smithi069.stdout:Apr 14 14:23:28 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-osd-2-activate[107387]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-2 --no-mon-config --dev /dev/mapper/vg_nvme-lv_2 2024-04-14T14:23:28.554 INFO:journalctl@ceph.osd.2.smithi069.stdout:Apr 14 14:23:28 smithi069 bash[107372]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-04-14T14:23:28.555 INFO:journalctl@ceph.osd.2.smithi069.stdout:Apr 14 14:23:28 smithi069 bash[107372]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-2 --no-mon-config --dev /dev/mapper/vg_nvme-lv_2 2024-04-14T14:23:28.555 INFO:journalctl@ceph.osd.2.smithi069.stdout:Apr 14 14:23:28 smithi069 bash[107372]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_2 2024-04-14T14:23:28.555 INFO:journalctl@ceph.osd.2.smithi069.stdout:Apr 14 14:23:28 smithi069 bash[107372]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-1 2024-04-14T14:23:28.555 INFO:journalctl@ceph.osd.2.smithi069.stdout:Apr 14 14:23:28 smithi069 bash[107372]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-2/block 2024-04-14T14:23:28.555 INFO:journalctl@ceph.osd.2.smithi069.stdout:Apr 14 14:23:28 smithi069 bash[107372]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-04-14T14:23:28.555 INFO:journalctl@ceph.osd.2.smithi069.stdout:Apr 14 14:23:28 smithi069 bash[107372]: --> ceph-volume raw activate successful for osd ID: 2 2024-04-14T14:23:28.555 INFO:journalctl@ceph.osd.2.smithi069.stdout:Apr 14 14:23:28 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-osd-2-activate[107387]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_2 2024-04-14T14:23:28.555 INFO:journalctl@ceph.osd.2.smithi069.stdout:Apr 14 14:23:28 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-osd-2-activate[107387]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-1 2024-04-14T14:23:28.555 INFO:journalctl@ceph.osd.2.smithi069.stdout:Apr 14 14:23:28 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-osd-2-activate[107387]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-2/block 2024-04-14T14:23:28.555 INFO:journalctl@ceph.osd.2.smithi069.stdout:Apr 14 14:23:28 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-osd-2-activate[107387]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-04-14T14:23:28.555 INFO:journalctl@ceph.osd.2.smithi069.stdout:Apr 14 14:23:28 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-osd-2-activate[107387]: --> ceph-volume raw activate successful for osd ID: 2 2024-04-14T14:23:28.555 INFO:journalctl@ceph.osd.2.smithi069.stdout:Apr 14 14:23:28 smithi069 podman[107372]: 2024-04-14 14:23:28.219944339 +0000 UTC m=+0.876589130 container died 012fbd15824253e6e8065c1b43520b7d4f0fe5168cd71b04ef424d6bd1dea7d5 (image=quay.ceph.io/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888, name=ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-osd-2-activate, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.vendor=CentOS, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=main-a971fd5, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240403, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=) 2024-04-14T14:23:28.816 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:23:28 smithi069 ceph-mon[83680]: from='client.25417 -' entity='client.iscsi.foo.smithi069.yuumtr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T14:23:28.817 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:23:28 smithi069 ceph-mon[88636]: from='client.25417 -' entity='client.iscsi.foo.smithi069.yuumtr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T14:23:29.094 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:23:28 smithi122 ceph-mon[67998]: from='client.25417 -' entity='client.iscsi.foo.smithi069.yuumtr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T14:23:29.150 INFO:journalctl@ceph.osd.2.smithi069.stdout:Apr 14 14:23:28 smithi069 podman[107513]: 2024-04-14 14:23:28.82864029 +0000 UTC m=+0.603678397 container remove 012fbd15824253e6e8065c1b43520b7d4f0fe5168cd71b04ef424d6bd1dea7d5 (image=quay.ceph.io/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888, name=ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-osd-2-activate, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, ceph=True, RELEASE=main-a971fd5, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.build-date=20240403, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, GIT_BRANCH=HEAD) 2024-04-14T14:23:29.151 INFO:journalctl@ceph.osd.2.smithi069.stdout:Apr 14 14:23:29 smithi069 podman[107578]: 2024-04-14 14:23:28.944748659 +0000 UTC m=+0.017760507 image pull b96c509eef841460b3271f5fdd331860da87b004524f4ed59414aa3680183b53 quay.ceph.io/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888 2024-04-14T14:23:29.151 INFO:journalctl@ceph.osd.2.smithi069.stdout:Apr 14 14:23:29 smithi069 podman[107578]: 2024-04-14 14:23:29.05448796 +0000 UTC m=+0.127499797 container create c0efba5ef45111ecb47224b1910bd2a0d09803805741e3787a63114789d33889 (image=quay.ceph.io/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888, name=ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-osd-2, RELEASE=main-a971fd5, maintainer=Guillaume Abrioux , ceph=True, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, org.label-schema.build-date=20240403, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, io.buildah.version=1.33.5, GIT_BRANCH=HEAD) 2024-04-14T14:23:29.566 INFO:journalctl@ceph.osd.2.smithi069.stdout:Apr 14 14:23:29 smithi069 podman[107578]: 2024-04-14 14:23:29.249139038 +0000 UTC m=+0.322150879 container init c0efba5ef45111ecb47224b1910bd2a0d09803805741e3787a63114789d33889 (image=quay.ceph.io/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888, name=ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-osd-2, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240403, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, RELEASE=main-a971fd5, io.buildah.version=1.33.5, GIT_BRANCH=HEAD) 2024-04-14T14:23:29.567 INFO:journalctl@ceph.osd.2.smithi069.stdout:Apr 14 14:23:29 smithi069 podman[107578]: 2024-04-14 14:23:29.252473175 +0000 UTC m=+0.325485016 container start c0efba5ef45111ecb47224b1910bd2a0d09803805741e3787a63114789d33889 (image=quay.ceph.io/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888, name=ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-osd-2, GIT_CLEAN=True, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240403, RELEASE=main-a971fd5, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS) 2024-04-14T14:23:29.567 INFO:journalctl@ceph.osd.2.smithi069.stdout:Apr 14 14:23:29 smithi069 ceph-osd[107597]: -- 172.21.15.69:0/3336236688 <== mon.0 v2:172.21.15.69:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55946b541680 con 0x55946b50ac00 2024-04-14T14:23:29.567 INFO:journalctl@ceph.osd.2.smithi069.stdout:Apr 14 14:23:29 smithi069 bash[107578]: c0efba5ef45111ecb47224b1910bd2a0d09803805741e3787a63114789d33889 2024-04-14T14:23:29.567 INFO:journalctl@ceph.osd.2.smithi069.stdout:Apr 14 14:23:29 smithi069 systemd[1]: Started Ceph osd.2 for dc326fc2-fa65-11ee-bc8e-c7b262605968. 2024-04-14T14:23:30.066 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:23:29 smithi069 ceph-mon[83680]: pgmap v105: 161 pgs: 16 active+undersized+degraded, 32 active+undersized, 113 active+clean; 585 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 383 B/s rd, 0 op/s; 60/627 objects degraded (9.569%) 2024-04-14T14:23:30.067 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:23:29 smithi069 ceph-mon[83680]: Health check failed: Degraded data redundancy: 60/627 objects degraded (9.569%), 16 pgs degraded (PG_DEGRADED) 2024-04-14T14:23:30.067 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:23:29 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:23:30.067 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:23:29 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:23:30.067 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:23:29 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:23:30.067 INFO:journalctl@ceph.osd.2.smithi069.stdout:Apr 14 14:23:29 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-osd-2[107592]: 2024-04-14T14:23:29.790+0000 7f5c08b4d7c0 -1 Falling back to public interface 2024-04-14T14:23:30.067 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:23:29 smithi069 ceph-mon[88636]: pgmap v105: 161 pgs: 16 active+undersized+degraded, 32 active+undersized, 113 active+clean; 585 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 383 B/s rd, 0 op/s; 60/627 objects degraded (9.569%) 2024-04-14T14:23:30.068 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:23:29 smithi069 ceph-mon[88636]: Health check failed: Degraded data redundancy: 60/627 objects degraded (9.569%), 16 pgs degraded (PG_DEGRADED) 2024-04-14T14:23:30.068 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:23:29 smithi069 ceph-mon[88636]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:23:30.068 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:23:29 smithi069 ceph-mon[88636]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:23:30.068 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:23:29 smithi069 ceph-mon[88636]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:23:30.094 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:23:29 smithi122 ceph-mon[67998]: pgmap v105: 161 pgs: 16 active+undersized+degraded, 32 active+undersized, 113 active+clean; 585 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 383 B/s rd, 0 op/s; 60/627 objects degraded (9.569%) 2024-04-14T14:23:30.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:23:29 smithi122 ceph-mon[67998]: Health check failed: Degraded data redundancy: 60/627 objects degraded (9.569%), 16 pgs degraded (PG_DEGRADED) 2024-04-14T14:23:30.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:23:29 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:23:30.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:23:29 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:23:30.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:23:29 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:23:31.897 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:23:31 smithi122 ceph-mon[67998]: pgmap v106: 161 pgs: 16 active+undersized+degraded, 32 active+undersized, 113 active+clean; 585 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 60/627 objects degraded (9.569%) 2024-04-14T14:23:31.943 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:23:31 smithi069 ceph-mon[83680]: pgmap v106: 161 pgs: 16 active+undersized+degraded, 32 active+undersized, 113 active+clean; 585 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 60/627 objects degraded (9.569%) 2024-04-14T14:23:31.944 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:23:31 smithi069 ceph-mon[88636]: pgmap v106: 161 pgs: 16 active+undersized+degraded, 32 active+undersized, 113 active+clean; 585 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 60/627 objects degraded (9.569%) 2024-04-14T14:23:32.639 INFO:journalctl@ceph.osd.2.smithi069.stdout:Apr 14 14:23:32 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-osd-2[107592]: 2024-04-14T14:23:32.497+0000 7f5c08b4d7c0 -1 osd.2 0 read_superblock omap replica is missing. 2024-04-14T14:23:32.639 INFO:journalctl@ceph.osd.2.smithi069.stdout:Apr 14 14:23:32 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-osd-2[107592]: 2024-04-14T14:23:32.519+0000 7f5c08b4d7c0 -1 osd.2 125 log_to_monitors true 2024-04-14T14:23:33.066 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:23:32 smithi069 ceph-mon[83680]: from='osd.2 [v2:172.21.15.69:6818/3243318726,v1:172.21.15.69:6819/3243318726]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-04-14T14:23:33.067 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:23:32 smithi069 ceph-mon[83680]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-04-14T14:23:33.067 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:23:32 smithi069 ceph-mon[88636]: from='osd.2 [v2:172.21.15.69:6818/3243318726,v1:172.21.15.69:6819/3243318726]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-04-14T14:23:33.067 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:23:32 smithi069 ceph-mon[88636]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-04-14T14:23:33.067 INFO:journalctl@ceph.osd.2.smithi069.stdout:Apr 14 14:23:32 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-osd-2[107592]: 2024-04-14T14:23:32.672+0000 7f5bf9fa5700 -1 osd.2 125 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-14T14:23:33.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:23:32 smithi122 ceph-mon[67998]: from='osd.2 [v2:172.21.15.69:6818/3243318726,v1:172.21.15.69:6819/3243318726]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-04-14T14:23:33.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:23:32 smithi122 ceph-mon[67998]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-04-14T14:23:34.067 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:23:33 smithi069 ceph-mon[83680]: pgmap v107: 161 pgs: 16 active+undersized+degraded, 32 active+undersized, 113 active+clean; 585 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 367 B/s rd, 0 op/s; 60/627 objects degraded (9.569%) 2024-04-14T14:23:34.067 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:23:33 smithi069 ceph-mon[83680]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-04-14T14:23:34.067 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:23:33 smithi069 ceph-mon[83680]: osdmap e128: 8 total, 7 up, 8 in 2024-04-14T14:23:34.067 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:23:33 smithi069 ceph-mon[83680]: from='osd.2 [v2:172.21.15.69:6818/3243318726,v1:172.21.15.69:6819/3243318726]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi069", "root=default"]}]: dispatch 2024-04-14T14:23:34.067 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:23:33 smithi069 ceph-mon[83680]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi069", "root=default"]}]: dispatch 2024-04-14T14:23:34.067 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:23:33 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:23:34.067 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:23:33 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:23:34.067 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:23:33 smithi069 ceph-mon[88636]: pgmap v107: 161 pgs: 16 active+undersized+degraded, 32 active+undersized, 113 active+clean; 585 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 367 B/s rd, 0 op/s; 60/627 objects degraded (9.569%) 2024-04-14T14:23:34.067 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:23:33 smithi069 ceph-mon[88636]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-04-14T14:23:34.067 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:23:33 smithi069 ceph-mon[88636]: osdmap e128: 8 total, 7 up, 8 in 2024-04-14T14:23:34.068 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:23:33 smithi069 ceph-mon[88636]: from='osd.2 [v2:172.21.15.69:6818/3243318726,v1:172.21.15.69:6819/3243318726]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi069", "root=default"]}]: dispatch 2024-04-14T14:23:34.068 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:23:33 smithi069 ceph-mon[88636]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi069", "root=default"]}]: dispatch 2024-04-14T14:23:34.068 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:23:33 smithi069 ceph-mon[88636]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:23:34.068 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:23:33 smithi069 ceph-mon[88636]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:23:34.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:23:33 smithi122 ceph-mon[67998]: pgmap v107: 161 pgs: 16 active+undersized+degraded, 32 active+undersized, 113 active+clean; 585 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 367 B/s rd, 0 op/s; 60/627 objects degraded (9.569%) 2024-04-14T14:23:34.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:23:33 smithi122 ceph-mon[67998]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-04-14T14:23:34.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:23:33 smithi122 ceph-mon[67998]: osdmap e128: 8 total, 7 up, 8 in 2024-04-14T14:23:34.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:23:33 smithi122 ceph-mon[67998]: from='osd.2 [v2:172.21.15.69:6818/3243318726,v1:172.21.15.69:6819/3243318726]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi069", "root=default"]}]: dispatch 2024-04-14T14:23:34.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:23:33 smithi122 ceph-mon[67998]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi069", "root=default"]}]: dispatch 2024-04-14T14:23:34.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:23:33 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:23:34.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:23:33 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:23:35.067 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:23:34 smithi069 ceph-mon[88636]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-04-14T14:23:35.067 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:23:34 smithi069 ceph-mon[88636]: osd.2 [v2:172.21.15.69:6818/3243318726,v1:172.21.15.69:6819/3243318726] boot 2024-04-14T14:23:35.067 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:23:34 smithi069 ceph-mon[88636]: osdmap e129: 8 total, 8 up, 8 in 2024-04-14T14:23:35.067 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:23:34 smithi069 ceph-mon[88636]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-14T14:23:35.067 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:23:34 smithi069 ceph-mon[88636]: osdmap e130: 8 total, 8 up, 8 in 2024-04-14T14:23:35.067 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:23:34 smithi069 ceph-mon[83680]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-04-14T14:23:35.067 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:23:34 smithi069 ceph-mon[83680]: osd.2 [v2:172.21.15.69:6818/3243318726,v1:172.21.15.69:6819/3243318726] boot 2024-04-14T14:23:35.067 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:23:34 smithi069 ceph-mon[83680]: osdmap e129: 8 total, 8 up, 8 in 2024-04-14T14:23:35.068 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:23:34 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-14T14:23:35.068 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:23:34 smithi069 ceph-mon[83680]: osdmap e130: 8 total, 8 up, 8 in 2024-04-14T14:23:35.094 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:23:34 smithi122 ceph-mon[67998]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-04-14T14:23:35.094 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:23:34 smithi122 ceph-mon[67998]: osd.2 [v2:172.21.15.69:6818/3243318726,v1:172.21.15.69:6819/3243318726] boot 2024-04-14T14:23:35.094 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:23:34 smithi122 ceph-mon[67998]: osdmap e129: 8 total, 8 up, 8 in 2024-04-14T14:23:35.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:23:34 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-14T14:23:35.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:23:34 smithi122 ceph-mon[67998]: osdmap e130: 8 total, 8 up, 8 in 2024-04-14T14:23:36.067 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:23:35 smithi069 ceph-mon[83680]: pgmap v110: 161 pgs: 16 active+undersized+degraded, 32 active+undersized, 113 active+clean; 585 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 383 B/s rd, 0 op/s; 60/627 objects degraded (9.569%) 2024-04-14T14:23:36.067 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:23:35 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:23:36.067 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:23:35 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:23:36.067 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:23:35 smithi069 ceph-mon[88636]: pgmap v110: 161 pgs: 16 active+undersized+degraded, 32 active+undersized, 113 active+clean; 585 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 383 B/s rd, 0 op/s; 60/627 objects degraded (9.569%) 2024-04-14T14:23:36.067 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:23:35 smithi069 ceph-mon[88636]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:23:36.067 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:23:35 smithi069 ceph-mon[88636]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:23:36.094 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:23:35 smithi122 ceph-mon[67998]: pgmap v110: 161 pgs: 16 active+undersized+degraded, 32 active+undersized, 113 active+clean; 585 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 383 B/s rd, 0 op/s; 60/627 objects degraded (9.569%) 2024-04-14T14:23:36.094 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:23:35 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:23:36.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:23:35 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:23:36.817 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 14 14:23:36 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-y[63038]: ::ffff:172.21.15.122 - - [14/Apr/2024:14:23:36] "GET /metrics HTTP/1.1" 200 38176 "" "Prometheus/2.43.0" 2024-04-14T14:23:37.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:23:37 smithi122 ceph-mon[67998]: pgmap v112: 161 pgs: 16 active+undersized+degraded, 32 active+undersized, 113 active+clean; 585 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 60/627 objects degraded (9.569%) 2024-04-14T14:23:37.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:23:37 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:23:37.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:23:37 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T14:23:37.899 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:23:37 smithi069 ceph-mon[83680]: pgmap v112: 161 pgs: 16 active+undersized+degraded, 32 active+undersized, 113 active+clean; 585 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 60/627 objects degraded (9.569%) 2024-04-14T14:23:37.899 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:23:37 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:23:37.899 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:23:37 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T14:23:37.900 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:23:37 smithi069 ceph-mon[88636]: pgmap v112: 161 pgs: 16 active+undersized+degraded, 32 active+undersized, 113 active+clean; 585 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 60/627 objects degraded (9.569%) 2024-04-14T14:23:37.900 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:23:37 smithi069 ceph-mon[88636]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:23:37.900 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:23:37 smithi069 ceph-mon[88636]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T14:23:39.317 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:23:38 smithi069 ceph-mon[83680]: from='client.25417 -' entity='client.iscsi.foo.smithi069.yuumtr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T14:23:39.317 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:23:38 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:23:39.317 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:23:38 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:23:39.317 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:23:38 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T14:23:39.318 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:23:38 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-14T14:23:39.318 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:23:38 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:23:39.318 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:23:38 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-14T14:23:39.318 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:23:38 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T14:23:39.318 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:23:38 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T14:23:39.318 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:23:38 smithi069 ceph-mon[83680]: Upgrade: Setting container_image for all nvmeof 2024-04-14T14:23:39.318 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:23:38 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:23:39.318 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:23:38 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T14:23:39.318 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:23:38 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T14:23:39.318 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:23:38 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T14:23:39.318 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:23:38 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T14:23:39.318 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:23:38 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T14:23:39.318 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:23:38 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T14:23:39.319 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:23:38 smithi069 ceph-mon[83680]: Upgrade: Finalizing container_image settings 2024-04-14T14:23:39.319 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:23:38 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]: dispatch 2024-04-14T14:23:39.319 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:23:38 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]': finished 2024-04-14T14:23:39.319 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:23:38 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-14T14:23:39.319 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:23:38 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon"}]': finished 2024-04-14T14:23:39.319 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:23:38 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]: dispatch 2024-04-14T14:23:39.319 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:23:38 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]': finished 2024-04-14T14:23:39.319 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:23:38 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd"}]: dispatch 2024-04-14T14:23:39.319 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:23:38 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mds"}]: dispatch 2024-04-14T14:23:39.319 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:23:38 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw"}]: dispatch 2024-04-14T14:23:39.319 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:23:38 smithi069 ceph-mon[88636]: from='client.25417 -' entity='client.iscsi.foo.smithi069.yuumtr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T14:23:39.320 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:23:38 smithi069 ceph-mon[88636]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:23:39.320 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:23:38 smithi069 ceph-mon[88636]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:23:39.320 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:23:38 smithi069 ceph-mon[88636]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T14:23:39.320 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:23:38 smithi069 ceph-mon[88636]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-14T14:23:39.320 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:23:38 smithi069 ceph-mon[88636]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:23:39.320 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:23:38 smithi069 ceph-mon[88636]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-14T14:23:39.320 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:23:38 smithi069 ceph-mon[88636]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T14:23:39.320 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:23:38 smithi069 ceph-mon[88636]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T14:23:39.320 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:23:38 smithi069 ceph-mon[88636]: Upgrade: Setting container_image for all nvmeof 2024-04-14T14:23:39.320 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:23:38 smithi069 ceph-mon[88636]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:23:39.320 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:23:38 smithi069 ceph-mon[88636]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T14:23:39.320 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:23:38 smithi069 ceph-mon[88636]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T14:23:39.320 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:23:38 smithi069 ceph-mon[88636]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T14:23:39.320 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:23:38 smithi069 ceph-mon[88636]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T14:23:39.321 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:23:38 smithi069 ceph-mon[88636]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T14:23:39.321 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:23:38 smithi069 ceph-mon[88636]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T14:23:39.321 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:23:38 smithi069 ceph-mon[88636]: Upgrade: Finalizing container_image settings 2024-04-14T14:23:39.321 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:23:38 smithi069 ceph-mon[88636]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]: dispatch 2024-04-14T14:23:39.321 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:23:38 smithi069 ceph-mon[88636]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]': finished 2024-04-14T14:23:39.321 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:23:38 smithi069 ceph-mon[88636]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-14T14:23:39.321 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:23:38 smithi069 ceph-mon[88636]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon"}]': finished 2024-04-14T14:23:39.321 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:23:38 smithi069 ceph-mon[88636]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]: dispatch 2024-04-14T14:23:39.321 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:23:38 smithi069 ceph-mon[88636]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]': finished 2024-04-14T14:23:39.321 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:23:38 smithi069 ceph-mon[88636]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd"}]: dispatch 2024-04-14T14:23:39.321 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:23:38 smithi069 ceph-mon[88636]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mds"}]: dispatch 2024-04-14T14:23:39.321 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:23:38 smithi069 ceph-mon[88636]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw"}]: dispatch 2024-04-14T14:23:39.322 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:23:38 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"}]: dispatch 2024-04-14T14:23:39.322 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:23:38 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-14T14:23:39.322 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:23:38 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter"}]: dispatch 2024-04-14T14:23:39.322 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:23:38 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.iscsi"}]: dispatch 2024-04-14T14:23:39.322 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:23:38 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nfs"}]: dispatch 2024-04-14T14:23:39.322 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:23:38 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]: dispatch 2024-04-14T14:23:39.322 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:23:38 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]': finished 2024-04-14T14:23:39.322 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:23:38 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-14T14:23:39.322 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:23:38 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-14T14:23:39.322 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:23:38 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-14T14:23:39.322 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:23:38 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-14T14:23:39.323 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:23:38 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-14T14:23:39.323 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:23:38 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-14T14:23:39.323 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:23:38 smithi069 ceph-mon[83680]: Upgrade: Complete! 2024-04-14T14:23:39.323 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:23:38 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]: dispatch 2024-04-14T14:23:39.323 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:23:38 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd='[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]': finished 2024-04-14T14:23:39.323 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:23:38 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T14:23:39.323 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:23:38 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-14T14:23:39.323 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:23:38 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:23:39.323 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:23:38 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T14:23:39.323 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:23:38 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-14T14:23:39.323 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:23:38 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:23:39.323 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:23:38 smithi069 ceph-mon[83680]: pgmap v113: 161 pgs: 161 active+clean; 585 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:23:39.324 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:23:38 smithi069 ceph-mon[88636]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"}]: dispatch 2024-04-14T14:23:39.324 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:23:38 smithi069 ceph-mon[88636]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-14T14:23:39.324 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:23:38 smithi069 ceph-mon[88636]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter"}]: dispatch 2024-04-14T14:23:39.324 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:23:38 smithi069 ceph-mon[88636]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.iscsi"}]: dispatch 2024-04-14T14:23:39.324 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:23:38 smithi069 ceph-mon[88636]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nfs"}]: dispatch 2024-04-14T14:23:39.324 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:23:38 smithi069 ceph-mon[88636]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]: dispatch 2024-04-14T14:23:39.324 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:23:38 smithi069 ceph-mon[88636]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]': finished 2024-04-14T14:23:39.324 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:23:38 smithi069 ceph-mon[88636]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-14T14:23:39.324 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:23:38 smithi069 ceph-mon[88636]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-14T14:23:39.324 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:23:38 smithi069 ceph-mon[88636]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-14T14:23:39.324 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:23:38 smithi069 ceph-mon[88636]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-14T14:23:39.324 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:23:38 smithi069 ceph-mon[88636]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-14T14:23:39.324 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:23:38 smithi069 ceph-mon[88636]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-14T14:23:39.325 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:23:38 smithi069 ceph-mon[88636]: Upgrade: Complete! 2024-04-14T14:23:39.325 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:23:38 smithi069 ceph-mon[88636]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]: dispatch 2024-04-14T14:23:39.325 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:23:38 smithi069 ceph-mon[88636]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd='[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]': finished 2024-04-14T14:23:39.325 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:23:38 smithi069 ceph-mon[88636]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T14:23:39.325 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:23:38 smithi069 ceph-mon[88636]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-14T14:23:39.325 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:23:38 smithi069 ceph-mon[88636]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:23:39.325 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:23:38 smithi069 ceph-mon[88636]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T14:23:39.325 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:23:38 smithi069 ceph-mon[88636]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-14T14:23:39.325 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:23:38 smithi069 ceph-mon[88636]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:23:39.325 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:23:38 smithi069 ceph-mon[88636]: pgmap v113: 161 pgs: 161 active+clean; 585 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:23:39.345 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:23:38 smithi122 ceph-mon[67998]: from='client.25417 -' entity='client.iscsi.foo.smithi069.yuumtr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T14:23:39.345 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:23:38 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:23:39.345 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:23:38 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:23:39.345 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:23:38 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T14:23:39.345 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:23:38 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-14T14:23:39.345 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:23:38 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:23:39.345 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:23:38 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-14T14:23:39.345 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:23:38 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T14:23:39.345 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:23:38 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T14:23:39.345 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:23:38 smithi122 ceph-mon[67998]: Upgrade: Setting container_image for all nvmeof 2024-04-14T14:23:39.346 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:23:38 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:23:39.346 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:23:38 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T14:23:39.346 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:23:38 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T14:23:39.346 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:23:38 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T14:23:39.346 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:23:38 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T14:23:39.346 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:23:38 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T14:23:39.346 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:23:38 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T14:23:39.346 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:23:38 smithi122 ceph-mon[67998]: Upgrade: Finalizing container_image settings 2024-04-14T14:23:39.346 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:23:38 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]: dispatch 2024-04-14T14:23:39.346 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:23:38 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]': finished 2024-04-14T14:23:39.346 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:23:38 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-14T14:23:39.346 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:23:38 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon"}]': finished 2024-04-14T14:23:39.346 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:23:38 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]: dispatch 2024-04-14T14:23:39.347 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:23:38 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]': finished 2024-04-14T14:23:39.347 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:23:38 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd"}]: dispatch 2024-04-14T14:23:39.347 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:23:38 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mds"}]: dispatch 2024-04-14T14:23:39.347 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:23:38 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw"}]: dispatch 2024-04-14T14:23:39.347 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:23:38 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"}]: dispatch 2024-04-14T14:23:39.347 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:23:38 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-14T14:23:39.347 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:23:38 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter"}]: dispatch 2024-04-14T14:23:39.347 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:23:38 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.iscsi"}]: dispatch 2024-04-14T14:23:39.347 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:23:38 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nfs"}]: dispatch 2024-04-14T14:23:39.347 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:23:38 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]: dispatch 2024-04-14T14:23:39.347 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:23:38 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]': finished 2024-04-14T14:23:39.347 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:23:38 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-14T14:23:39.347 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:23:38 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-14T14:23:39.347 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:23:38 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-14T14:23:39.347 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:23:38 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-14T14:23:39.348 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:23:38 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-14T14:23:39.348 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:23:38 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-14T14:23:39.348 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:23:38 smithi122 ceph-mon[67998]: Upgrade: Complete! 2024-04-14T14:23:39.348 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:23:38 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]: dispatch 2024-04-14T14:23:39.348 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:23:38 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd='[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]': finished 2024-04-14T14:23:39.348 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:23:38 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T14:23:39.348 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:23:38 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-14T14:23:39.348 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:23:38 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:23:39.348 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:23:38 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T14:23:39.348 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:23:38 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-14T14:23:39.348 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:23:38 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:23:39.348 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:23:38 smithi122 ceph-mon[67998]: pgmap v113: 161 pgs: 161 active+clean; 585 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:23:40.317 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:23:39 smithi069 ceph-mon[83680]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 60/627 objects degraded (9.569%), 16 pgs degraded) 2024-04-14T14:23:40.317 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:23:39 smithi069 ceph-mon[83680]: Cluster is now healthy 2024-04-14T14:23:40.317 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:23:39 smithi069 ceph-mon[88636]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 60/627 objects degraded (9.569%), 16 pgs degraded) 2024-04-14T14:23:40.317 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:23:39 smithi069 ceph-mon[88636]: Cluster is now healthy 2024-04-14T14:23:40.344 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:23:39 smithi122 ceph-mon[67998]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 60/627 objects degraded (9.569%), 16 pgs degraded) 2024-04-14T14:23:40.345 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:23:39 smithi122 ceph-mon[67998]: Cluster is now healthy 2024-04-14T14:23:41.316 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:23:40 smithi069 ceph-mon[83680]: pgmap v114: 161 pgs: 161 active+clean; 585 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 647 B/s rd, 0 op/s 2024-04-14T14:23:41.317 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:23:40 smithi069 ceph-mon[88636]: pgmap v114: 161 pgs: 161 active+clean; 585 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 647 B/s rd, 0 op/s 2024-04-14T14:23:41.344 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:23:40 smithi122 ceph-mon[67998]: pgmap v114: 161 pgs: 161 active+clean; 585 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 647 B/s rd, 0 op/s 2024-04-14T14:23:43.066 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:23:42 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:23:43.067 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:23:42 smithi069 ceph-mon[88636]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:23:43.094 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:23:42 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:23:44.066 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:23:43 smithi069 ceph-mon[83680]: pgmap v115: 161 pgs: 161 active+clean; 585 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-04-14T14:23:44.067 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:23:43 smithi069 ceph-mon[88636]: pgmap v115: 161 pgs: 161 active+clean; 585 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-04-14T14:23:44.094 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:23:43 smithi122 ceph-mon[67998]: pgmap v115: 161 pgs: 161 active+clean; 585 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-04-14T14:23:46.067 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:23:45 smithi069 ceph-mon[83680]: pgmap v116: 161 pgs: 161 active+clean; 585 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-14T14:23:46.067 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:23:45 smithi069 ceph-mon[88636]: pgmap v116: 161 pgs: 161 active+clean; 585 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-14T14:23:46.094 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:23:45 smithi122 ceph-mon[67998]: pgmap v116: 161 pgs: 161 active+clean; 585 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-14T14:23:46.817 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 14 14:23:46 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-y[63038]: ::ffff:172.21.15.122 - - [14/Apr/2024:14:23:46] "GET /metrics HTTP/1.1" 200 38039 "" "Prometheus/2.43.0" 2024-04-14T14:23:47.995 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:23:47 smithi069 ceph-mon[88636]: pgmap v117: 161 pgs: 161 active+clean; 585 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 860 B/s rd, 0 op/s 2024-04-14T14:23:47.995 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:23:47 smithi069 ceph-mon[88636]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:23:47.995 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:23:47 smithi069 ceph-mon[83680]: pgmap v117: 161 pgs: 161 active+clean; 585 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 860 B/s rd, 0 op/s 2024-04-14T14:23:47.995 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:23:47 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:23:48.094 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:23:47 smithi122 ceph-mon[67998]: pgmap v117: 161 pgs: 161 active+clean; 585 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 860 B/s rd, 0 op/s 2024-04-14T14:23:48.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:23:47 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:23:49.817 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:23:49 smithi069 ceph-mon[83680]: from='client.25417 -' entity='client.iscsi.foo.smithi069.yuumtr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T14:23:49.817 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:23:49 smithi069 ceph-mon[83680]: pgmap v118: 161 pgs: 161 active+clean; 585 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:23:49.817 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:23:49 smithi069 ceph-mon[88636]: from='client.25417 -' entity='client.iscsi.foo.smithi069.yuumtr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T14:23:49.817 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:23:49 smithi069 ceph-mon[88636]: pgmap v118: 161 pgs: 161 active+clean; 585 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:23:50.094 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:23:49 smithi122 ceph-mon[67998]: from='client.25417 -' entity='client.iscsi.foo.smithi069.yuumtr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T14:23:50.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:23:49 smithi122 ceph-mon[67998]: pgmap v118: 161 pgs: 161 active+clean; 585 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:23:50.135 DEBUG:teuthology.orchestra.run.smithi069:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid dc326fc2-fa65-11ee-bc8e-c7b262605968 -e sha1=a971fd5f5719038c70b817bafacd6d5400b1d888 -- bash -c 'ceph orch ps' 2024-04-14T14:23:50.693 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:23:50 smithi069 ceph-mon[88636]: from='client.44364 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T14:23:50.693 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:23:50 smithi069 ceph-mon[83680]: from='client.44364 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T14:23:51.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:23:50 smithi122 ceph-mon[67998]: from='client.44364 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T14:23:52.056 INFO:teuthology.orchestra.run.smithi069.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-14T14:23:52.056 INFO:teuthology.orchestra.run.smithi069.stdout:alertmanager.a smithi069 *:9093,9094 running (5m) 18s ago 27m 24.4M - 0.25.0 c8568f914cd2 7524f106e7bd 2024-04-14T14:23:52.056 INFO:teuthology.orchestra.run.smithi069.stdout:grafana.a smithi122 *:3000 running (5m) 3m ago 27m 81.0M - 9.4.12 2bacad6d85d8 5685178e3eba 2024-04-14T14:23:52.056 INFO:teuthology.orchestra.run.smithi069.stdout:iscsi.foo.smithi069.yuumtr smithi069 running (6m) 18s ago 26m 62.9M - 3.5 e1d6a67b021e 38cc81ea8cfa 2024-04-14T14:23:52.056 INFO:teuthology.orchestra.run.smithi069.stdout:mgr.x smithi122 *:8443,9283,8765 running (6m) 3m ago 31m 450M - 19.0.0-2778-ga971fd5f b96c509eef84 e29977590b43 2024-04-14T14:23:52.056 INFO:teuthology.orchestra.run.smithi069.stdout:mgr.y smithi069 *:8443,9283,8765 running (16m) 18s ago 33m 504M - 19.0.0-2778-ga971fd5f b96c509eef84 7af3d29d0e5f 2024-04-14T14:23:52.056 INFO:teuthology.orchestra.run.smithi069.stdout:mon.a smithi069 running (3m) 18s ago 33m 52.7M 2048M 19.0.0-2778-ga971fd5f b96c509eef84 8bcd6b3d3846 2024-04-14T14:23:52.056 INFO:teuthology.orchestra.run.smithi069.stdout:mon.b smithi122 running (4m) 3m ago 32m 33.9M 2048M 19.0.0-2778-ga971fd5f b96c509eef84 05f437a1a9a1 2024-04-14T14:23:52.056 INFO:teuthology.orchestra.run.smithi069.stdout:mon.c smithi069 running (3m) 18s ago 32m 36.8M 2048M 19.0.0-2778-ga971fd5f b96c509eef84 84aa95f11d0c 2024-04-14T14:23:52.056 INFO:teuthology.orchestra.run.smithi069.stdout:node-exporter.a smithi069 *:9100 running (6m) 18s ago 28m 15.5M - 1.5.0 0da6a335fe13 0de0eb2171b9 2024-04-14T14:23:52.057 INFO:teuthology.orchestra.run.smithi069.stdout:node-exporter.b smithi122 *:9100 running (6m) 3m ago 27m 16.1M - 1.5.0 0da6a335fe13 67877bf02d8e 2024-04-14T14:23:52.057 INFO:teuthology.orchestra.run.smithi069.stdout:osd.0 smithi069 running (108s) 18s ago 31m 53.4M 2992M 19.0.0-2778-ga971fd5f b96c509eef84 a0e57e3fb494 2024-04-14T14:23:52.057 INFO:teuthology.orchestra.run.smithi069.stdout:osd.1 smithi069 running (74s) 18s ago 31m 51.5M 2992M 19.0.0-2778-ga971fd5f b96c509eef84 cf73d17919bb 2024-04-14T14:23:52.057 INFO:teuthology.orchestra.run.smithi069.stdout:osd.2 smithi069 running (23s) 18s ago 30m 12.7M 2992M 19.0.0-2778-ga971fd5f b96c509eef84 c0efba5ef451 2024-04-14T14:23:52.057 INFO:teuthology.orchestra.run.smithi069.stdout:osd.3 smithi069 running (30m) 18s ago 30m 57.5M 2992M 17.2.0 e1d6a67b021e 0925ba3da801 2024-04-14T14:23:52.057 INFO:teuthology.orchestra.run.smithi069.stdout:osd.4 smithi122 running (29m) 3m ago 29m 56.1M 3248M 17.2.0 e1d6a67b021e c00975caf771 2024-04-14T14:23:52.057 INFO:teuthology.orchestra.run.smithi069.stdout:osd.5 smithi122 running (29m) 3m ago 29m 55.2M 3248M 17.2.0 e1d6a67b021e 5f52d217ceb8 2024-04-14T14:23:52.057 INFO:teuthology.orchestra.run.smithi069.stdout:osd.6 smithi122 running (29m) 3m ago 29m 52.4M 3248M 17.2.0 e1d6a67b021e a2acfec61475 2024-04-14T14:23:52.057 INFO:teuthology.orchestra.run.smithi069.stdout:osd.7 smithi122 running (28m) 3m ago 28m 57.1M 3248M 17.2.0 e1d6a67b021e 62411cd06801 2024-04-14T14:23:52.057 INFO:teuthology.orchestra.run.smithi069.stdout:prometheus.a smithi122 *:9095 running (5m) 3m ago 28m 66.4M - 2.43.0 a07b618ecd1d 5c1f41d3bbe6 2024-04-14T14:23:52.057 INFO:teuthology.orchestra.run.smithi069.stdout:rgw.foo.smithi069.wesedz smithi069 *:8000 running (26m) 18s ago 26m 79.4M - 17.2.0 e1d6a67b021e 16de491440f3 2024-04-14T14:23:52.057 INFO:teuthology.orchestra.run.smithi069.stdout:rgw.foo.smithi122.jiobyv smithi122 *:8000 running (26m) 3m ago 26m 75.8M - 17.2.0 e1d6a67b021e 6d16de445dba 2024-04-14T14:23:52.058 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:23:51 smithi069 ceph-mon[83680]: pgmap v119: 161 pgs: 161 active+clean; 585 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:23:52.058 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:23:51 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:23:52.058 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:23:51 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T14:23:52.058 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:23:51 smithi069 ceph-mon[88636]: pgmap v119: 161 pgs: 161 active+clean; 585 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:23:52.058 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:23:51 smithi069 ceph-mon[88636]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:23:52.058 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:23:51 smithi069 ceph-mon[88636]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T14:23:52.094 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:23:51 smithi122 ceph-mon[67998]: pgmap v119: 161 pgs: 161 active+clean; 585 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:23:52.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:23:51 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:23:52.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:23:51 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T14:23:52.735 DEBUG:teuthology.orchestra.run.smithi069:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid dc326fc2-fa65-11ee-bc8e-c7b262605968 -e sha1=a971fd5f5719038c70b817bafacd6d5400b1d888 -- bash -c 'ceph versions | jq -e '"'"'.osd | length == 2'"'"'' 2024-04-14T14:23:53.957 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:23:53 smithi069 ceph-mon[83680]: from='client.44370 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T14:23:53.957 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:23:53 smithi069 ceph-mon[83680]: pgmap v120: 161 pgs: 161 active+clean; 585 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:23:53.957 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:23:53 smithi069 ceph-mon[88636]: from='client.44370 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T14:23:53.958 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:23:53 smithi069 ceph-mon[88636]: pgmap v120: 161 pgs: 161 active+clean; 585 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:23:54.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:23:53 smithi122 ceph-mon[67998]: from='client.44370 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T14:23:54.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:23:53 smithi122 ceph-mon[67998]: pgmap v120: 161 pgs: 161 active+clean; 585 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:23:54.606 INFO:teuthology.orchestra.run.smithi069.stdout:true 2024-04-14T14:23:55.045 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:23:54 smithi069 ceph-mon[83680]: from='client.? 172.21.15.69:0/108679291' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T14:23:55.045 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:23:54 smithi069 ceph-mon[88636]: from='client.? 172.21.15.69:0/108679291' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T14:23:55.094 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:23:54 smithi122 ceph-mon[67998]: from='client.? 172.21.15.69:0/108679291' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T14:23:55.171 DEBUG:teuthology.orchestra.run.smithi069:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid dc326fc2-fa65-11ee-bc8e-c7b262605968 -e sha1=a971fd5f5719038c70b817bafacd6d5400b1d888 -- bash -c 'ceph orch upgrade check quay.ceph.io/ceph-ci/ceph:$sha1 | jq -e '"'"'.up_to_date | length == 8'"'"'' 2024-04-14T14:23:55.733 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:23:55 smithi069 ceph-mon[83680]: pgmap v121: 161 pgs: 161 active+clean; 585 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:23:55.733 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:23:55 smithi069 ceph-mon[88636]: pgmap v121: 161 pgs: 161 active+clean; 585 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:23:56.094 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:23:55 smithi122 ceph-mon[67998]: pgmap v121: 161 pgs: 161 active+clean; 585 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:23:56.722 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 14 14:23:56 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-y[63038]: ::ffff:172.21.15.122 - - [14/Apr/2024:14:23:56] "GET /metrics HTTP/1.1" 200 38181 "" "Prometheus/2.43.0" 2024-04-14T14:23:57.968 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:23:57 smithi069 ceph-mon[83680]: pgmap v122: 161 pgs: 161 active+clean; 585 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:23:57.968 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:23:57 smithi069 ceph-mon[88636]: pgmap v122: 161 pgs: 161 active+clean; 585 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:23:58.094 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:23:57 smithi122 ceph-mon[67998]: pgmap v122: 161 pgs: 161 active+clean; 585 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:23:58.796 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:23:58 smithi069 ceph-mon[83680]: from='client.44382 -' entity='client.admin' cmd=[{"prefix": "orch upgrade check", "image": "quay.ceph.io/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T14:23:58.797 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:23:58 smithi069 ceph-mon[88636]: from='client.44382 -' entity='client.admin' cmd=[{"prefix": "orch upgrade check", "image": "quay.ceph.io/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T14:23:58.877 INFO:teuthology.orchestra.run.smithi069.stdout:true 2024-04-14T14:23:59.094 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:23:58 smithi122 ceph-mon[67998]: from='client.44382 -' entity='client.admin' cmd=[{"prefix": "orch upgrade check", "image": "quay.ceph.io/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T14:23:59.553 DEBUG:teuthology.orchestra.run.smithi069:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid dc326fc2-fa65-11ee-bc8e-c7b262605968 -e sha1=a971fd5f5719038c70b817bafacd6d5400b1d888 -- bash -c 'ceph orch upgrade status' 2024-04-14T14:23:59.816 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:23:59 smithi069 ceph-mon[83680]: from='client.25417 -' entity='client.iscsi.foo.smithi069.yuumtr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T14:23:59.817 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:23:59 smithi069 ceph-mon[83680]: pgmap v123: 161 pgs: 161 active+clean; 585 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:23:59.817 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:23:59 smithi069 ceph-mon[88636]: from='client.25417 -' entity='client.iscsi.foo.smithi069.yuumtr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T14:23:59.817 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:23:59 smithi069 ceph-mon[88636]: pgmap v123: 161 pgs: 161 active+clean; 585 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:24:00.094 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:23:59 smithi122 ceph-mon[67998]: from='client.25417 -' entity='client.iscsi.foo.smithi069.yuumtr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T14:24:00.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:23:59 smithi122 ceph-mon[67998]: pgmap v123: 161 pgs: 161 active+clean; 585 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:24:01.507 INFO:teuthology.orchestra.run.smithi069.stdout:{ 2024-04-14T14:24:01.507 INFO:teuthology.orchestra.run.smithi069.stdout: "target_image": null, 2024-04-14T14:24:01.507 INFO:teuthology.orchestra.run.smithi069.stdout: "in_progress": false, 2024-04-14T14:24:01.507 INFO:teuthology.orchestra.run.smithi069.stdout: "which": "", 2024-04-14T14:24:01.507 INFO:teuthology.orchestra.run.smithi069.stdout: "services_complete": [], 2024-04-14T14:24:01.507 INFO:teuthology.orchestra.run.smithi069.stdout: "progress": null, 2024-04-14T14:24:01.507 INFO:teuthology.orchestra.run.smithi069.stdout: "message": "", 2024-04-14T14:24:01.507 INFO:teuthology.orchestra.run.smithi069.stdout: "is_paused": false 2024-04-14T14:24:01.507 INFO:teuthology.orchestra.run.smithi069.stdout:} 2024-04-14T14:24:01.816 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:24:01 smithi069 ceph-mon[88636]: pgmap v124: 161 pgs: 161 active+clean; 585 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:24:01.817 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:24:01 smithi069 ceph-mon[83680]: pgmap v124: 161 pgs: 161 active+clean; 585 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:24:02.073 DEBUG:teuthology.orchestra.run.smithi069:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid dc326fc2-fa65-11ee-bc8e-c7b262605968 -e sha1=a971fd5f5719038c70b817bafacd6d5400b1d888 -- bash -c 'ceph health detail' 2024-04-14T14:24:02.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:24:01 smithi122 ceph-mon[67998]: pgmap v124: 161 pgs: 161 active+clean; 585 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:24:03.029 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:24:02 smithi069 ceph-mon[83680]: from='client.44388 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T14:24:03.029 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:24:02 smithi069 ceph-mon[88636]: from='client.44388 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T14:24:03.094 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:24:02 smithi122 ceph-mon[67998]: from='client.44388 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T14:24:03.988 INFO:teuthology.orchestra.run.smithi069.stdout:HEALTH_OK 2024-04-14T14:24:03.988 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:24:03 smithi069 ceph-mon[83680]: pgmap v125: 161 pgs: 161 active+clean; 585 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:24:03.988 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:24:03 smithi069 ceph-mon[88636]: pgmap v125: 161 pgs: 161 active+clean; 585 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:24:04.094 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:24:03 smithi122 ceph-mon[67998]: pgmap v125: 161 pgs: 161 active+clean; 585 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:24:04.663 DEBUG:teuthology.orchestra.run.smithi069:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid dc326fc2-fa65-11ee-bc8e-c7b262605968 -e sha1=a971fd5f5719038c70b817bafacd6d5400b1d888 -- bash -c 'ceph orch upgrade start --image quay.ceph.io/ceph-ci/ceph:$sha1 --daemon-types crash,osd' 2024-04-14T14:24:04.971 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:24:04 smithi069 ceph-mon[83680]: from='client.? 172.21.15.69:0/694910954' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-14T14:24:04.971 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:24:04 smithi069 ceph-mon[83680]: pgmap v126: 161 pgs: 161 active+clean; 585 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:24:04.971 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:24:04 smithi069 ceph-mon[88636]: from='client.? 172.21.15.69:0/694910954' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-14T14:24:04.972 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:24:04 smithi069 ceph-mon[88636]: pgmap v126: 161 pgs: 161 active+clean; 585 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:24:05.094 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:24:04 smithi122 ceph-mon[67998]: from='client.? 172.21.15.69:0/694910954' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-14T14:24:05.094 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:24:04 smithi122 ceph-mon[67998]: pgmap v126: 161 pgs: 161 active+clean; 585 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:24:06.816 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:24:06 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T14:24:06.817 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 14 14:24:06 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-y[63038]: ::ffff:172.21.15.122 - - [14/Apr/2024:14:24:06] "GET /metrics HTTP/1.1" 200 38181 "" "Prometheus/2.43.0" 2024-04-14T14:24:06.817 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:24:06 smithi069 ceph-mon[88636]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T14:24:07.094 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:24:06 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T14:24:07.693 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:24:07 smithi069 ceph-mon[83680]: pgmap v127: 161 pgs: 161 active+clean; 585 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:24:07.693 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:24:07 smithi069 ceph-mon[83680]: from='client.44400 -' entity='client.admin' cmd=[{"prefix": "orch upgrade start", "image": "quay.ceph.io/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888", "daemon_types": "crash,osd", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T14:24:07.693 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:24:07 smithi069 ceph-mon[88636]: pgmap v127: 161 pgs: 161 active+clean; 585 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:24:07.694 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:24:07 smithi069 ceph-mon[88636]: from='client.44400 -' entity='client.admin' cmd=[{"prefix": "orch upgrade start", "image": "quay.ceph.io/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888", "daemon_types": "crash,osd", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T14:24:08.057 INFO:teuthology.orchestra.run.smithi069.stdout:Initiating upgrade to quay.ceph.io/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888 2024-04-14T14:24:08.094 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:24:07 smithi122 ceph-mon[67998]: pgmap v127: 161 pgs: 161 active+clean; 585 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:24:08.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:24:07 smithi122 ceph-mon[67998]: from='client.44400 -' entity='client.admin' cmd=[{"prefix": "orch upgrade start", "image": "quay.ceph.io/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888", "daemon_types": "crash,osd", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T14:24:08.788 DEBUG:teuthology.orchestra.run.smithi069:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid dc326fc2-fa65-11ee-bc8e-c7b262605968 -e sha1=a971fd5f5719038c70b817bafacd6d5400b1d888 -- bash -c 'while ceph orch upgrade status | jq '"'"'.in_progress'"'"' | grep true && ! ceph orch upgrade status | jq '"'"'.message'"'"' | grep Error ; do ceph orch ps ; ceph versions ; ceph orch upgrade status ; sleep 30 ; done' 2024-04-14T14:24:09.056 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:24:09 smithi069 ceph-mon[83680]: from='client.25417 -' entity='client.iscsi.foo.smithi069.yuumtr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T14:24:09.057 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:24:09 smithi069 ceph-mon[88636]: from='client.25417 -' entity='client.iscsi.foo.smithi069.yuumtr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T14:24:09.057 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:24:09 smithi069 ceph-mon[88636]: Upgrade: Started with target quay.ceph.io/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888 2024-04-14T14:24:09.057 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:24:09 smithi069 ceph-mon[88636]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:24:09.317 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:24:09 smithi069 ceph-mon[88636]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T14:24:09.317 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:24:09 smithi069 ceph-mon[88636]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-14T14:24:09.317 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:24:09 smithi069 ceph-mon[88636]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:24:09.317 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:24:09 smithi069 ceph-mon[88636]: Upgrade: First pull of quay.ceph.io/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888 2024-04-14T14:24:09.317 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:24:09 smithi069 ceph-mon[88636]: pgmap v128: 161 pgs: 161 active+clean; 585 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:24:09.317 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:24:09 smithi069 ceph-mon[83680]: Upgrade: Started with target quay.ceph.io/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888 2024-04-14T14:24:09.317 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:24:09 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:24:09.317 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:24:09 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T14:24:09.317 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:24:09 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-14T14:24:09.317 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:24:09 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:24:09.317 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:24:09 smithi069 ceph-mon[83680]: Upgrade: First pull of quay.ceph.io/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888 2024-04-14T14:24:09.318 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:24:09 smithi069 ceph-mon[83680]: pgmap v128: 161 pgs: 161 active+clean; 585 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:24:09.344 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:24:09 smithi122 ceph-mon[67998]: from='client.25417 -' entity='client.iscsi.foo.smithi069.yuumtr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T14:24:09.345 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:24:09 smithi122 ceph-mon[67998]: Upgrade: Started with target quay.ceph.io/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888 2024-04-14T14:24:09.345 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:24:09 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:24:09.345 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:24:09 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T14:24:09.345 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:24:09 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-14T14:24:09.345 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:24:09 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:24:09.345 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:24:09 smithi122 ceph-mon[67998]: Upgrade: First pull of quay.ceph.io/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888 2024-04-14T14:24:09.345 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:24:09 smithi122 ceph-mon[67998]: pgmap v128: 161 pgs: 161 active+clean; 585 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:24:11.306 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:24:11 smithi069 ceph-mon[88636]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:24:11.306 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:24:11 smithi069 ceph-mon[88636]: Upgrade: Target is version 19.0.0-2778-ga971fd5f (squid) 2024-04-14T14:24:11.306 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:24:11 smithi069 ceph-mon[88636]: Upgrade: Target container is quay.ceph.io/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888, digests ['quay.ceph.io/ceph-ci/ceph@sha256:02f0a191d4d693952654c64f93cc9c0152602c0915251b3c94c2c4e32855c608'] 2024-04-14T14:24:11.306 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:24:11 smithi069 ceph-mon[88636]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-14T14:24:11.306 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:24:11 smithi069 ceph-mon[88636]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T14:24:11.307 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:24:11 smithi069 ceph-mon[88636]: Upgrade: Setting container_image for all mgr 2024-04-14T14:24:11.307 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:24:11 smithi069 ceph-mon[88636]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:24:11.307 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:24:11 smithi069 ceph-mon[88636]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T14:24:11.307 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:24:11 smithi069 ceph-mon[88636]: Upgrade: Setting container_image for all mon 2024-04-14T14:24:11.307 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:24:11 smithi069 ceph-mon[88636]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:24:11.307 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:24:11 smithi069 ceph-mon[88636]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T14:24:11.307 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:24:11 smithi069 ceph-mon[88636]: Upgrade: Setting container_image for all crash 2024-04-14T14:24:11.307 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:24:11 smithi069 ceph-mon[88636]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:24:11.307 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:24:11 smithi069 ceph-mon[88636]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16}]: dispatch 2024-04-14T14:24:11.307 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:24:11 smithi069 ceph-mon[88636]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16}]: dispatch 2024-04-14T14:24:11.307 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:24:11 smithi069 ceph-mon[88636]: Upgrade: osd.3 is safe to restart 2024-04-14T14:24:11.307 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:24:11 smithi069 ceph-mon[88636]: pgmap v129: 161 pgs: 161 active+clean; 585 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:24:11.308 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:24:11 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:24:11.308 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:24:11 smithi069 ceph-mon[83680]: Upgrade: Target is version 19.0.0-2778-ga971fd5f (squid) 2024-04-14T14:24:11.308 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:24:11 smithi069 ceph-mon[83680]: Upgrade: Target container is quay.ceph.io/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888, digests ['quay.ceph.io/ceph-ci/ceph@sha256:02f0a191d4d693952654c64f93cc9c0152602c0915251b3c94c2c4e32855c608'] 2024-04-14T14:24:11.308 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:24:11 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-14T14:24:11.308 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:24:11 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T14:24:11.308 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:24:11 smithi069 ceph-mon[83680]: Upgrade: Setting container_image for all mgr 2024-04-14T14:24:11.308 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:24:11 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:24:11.308 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:24:11 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T14:24:11.308 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:24:11 smithi069 ceph-mon[83680]: Upgrade: Setting container_image for all mon 2024-04-14T14:24:11.308 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:24:11 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:24:11.308 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:24:11 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T14:24:11.308 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:24:11 smithi069 ceph-mon[83680]: Upgrade: Setting container_image for all crash 2024-04-14T14:24:11.308 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:24:11 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:24:11.308 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:24:11 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16}]: dispatch 2024-04-14T14:24:11.308 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:24:11 smithi069 ceph-mon[83680]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16}]: dispatch 2024-04-14T14:24:11.309 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:24:11 smithi069 ceph-mon[83680]: Upgrade: osd.3 is safe to restart 2024-04-14T14:24:11.309 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:24:11 smithi069 ceph-mon[83680]: pgmap v129: 161 pgs: 161 active+clean; 585 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:24:11.345 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:24:11 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:24:11.345 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:24:11 smithi122 ceph-mon[67998]: Upgrade: Target is version 19.0.0-2778-ga971fd5f (squid) 2024-04-14T14:24:11.345 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:24:11 smithi122 ceph-mon[67998]: Upgrade: Target container is quay.ceph.io/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888, digests ['quay.ceph.io/ceph-ci/ceph@sha256:02f0a191d4d693952654c64f93cc9c0152602c0915251b3c94c2c4e32855c608'] 2024-04-14T14:24:11.345 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:24:11 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-14T14:24:11.345 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:24:11 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T14:24:11.345 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:24:11 smithi122 ceph-mon[67998]: Upgrade: Setting container_image for all mgr 2024-04-14T14:24:11.345 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:24:11 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:24:11.345 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:24:11 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T14:24:11.345 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:24:11 smithi122 ceph-mon[67998]: Upgrade: Setting container_image for all mon 2024-04-14T14:24:11.345 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:24:11 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:24:11.345 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:24:11 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T14:24:11.345 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:24:11 smithi122 ceph-mon[67998]: Upgrade: Setting container_image for all crash 2024-04-14T14:24:11.346 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:24:11 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:24:11.346 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:24:11 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16}]: dispatch 2024-04-14T14:24:11.346 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:24:11 smithi122 ceph-mon[67998]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16}]: dispatch 2024-04-14T14:24:11.346 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:24:11 smithi122 ceph-mon[67998]: Upgrade: osd.3 is safe to restart 2024-04-14T14:24:11.346 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:24:11 smithi122 ceph-mon[67998]: pgmap v129: 161 pgs: 161 active+clean; 585 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:24:11.751 INFO:teuthology.orchestra.run.smithi069.stdout:true 2024-04-14T14:24:12.421 INFO:teuthology.orchestra.run.smithi069.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-14T14:24:12.422 INFO:teuthology.orchestra.run.smithi069.stdout:alertmanager.a smithi069 *:9093,9094 running (5m) 38s ago 27m 24.4M - 0.25.0 c8568f914cd2 7524f106e7bd 2024-04-14T14:24:12.422 INFO:teuthology.orchestra.run.smithi069.stdout:grafana.a smithi122 *:3000 running (5m) 3m ago 27m 81.0M - 9.4.12 2bacad6d85d8 5685178e3eba 2024-04-14T14:24:12.422 INFO:teuthology.orchestra.run.smithi069.stdout:iscsi.foo.smithi069.yuumtr smithi069 running (7m) 38s ago 26m 62.9M - 3.5 e1d6a67b021e 38cc81ea8cfa 2024-04-14T14:24:12.422 INFO:teuthology.orchestra.run.smithi069.stdout:mgr.x smithi122 *:8443,9283,8765 running (6m) 3m ago 32m 450M - 19.0.0-2778-ga971fd5f b96c509eef84 e29977590b43 2024-04-14T14:24:12.422 INFO:teuthology.orchestra.run.smithi069.stdout:mgr.y smithi069 *:8443,9283,8765 running (16m) 38s ago 33m 504M - 19.0.0-2778-ga971fd5f b96c509eef84 7af3d29d0e5f 2024-04-14T14:24:12.422 INFO:teuthology.orchestra.run.smithi069.stdout:mon.a smithi069 running (3m) 38s ago 33m 52.7M 2048M 19.0.0-2778-ga971fd5f b96c509eef84 8bcd6b3d3846 2024-04-14T14:24:12.422 INFO:teuthology.orchestra.run.smithi069.stdout:mon.b smithi122 running (4m) 3m ago 32m 33.9M 2048M 19.0.0-2778-ga971fd5f b96c509eef84 05f437a1a9a1 2024-04-14T14:24:12.422 INFO:teuthology.orchestra.run.smithi069.stdout:mon.c smithi069 running (3m) 38s ago 32m 36.8M 2048M 19.0.0-2778-ga971fd5f b96c509eef84 84aa95f11d0c 2024-04-14T14:24:12.422 INFO:teuthology.orchestra.run.smithi069.stdout:node-exporter.a smithi069 *:9100 running (6m) 38s ago 28m 15.5M - 1.5.0 0da6a335fe13 0de0eb2171b9 2024-04-14T14:24:12.422 INFO:teuthology.orchestra.run.smithi069.stdout:node-exporter.b smithi122 *:9100 running (6m) 3m ago 28m 16.1M - 1.5.0 0da6a335fe13 67877bf02d8e 2024-04-14T14:24:12.422 INFO:teuthology.orchestra.run.smithi069.stdout:osd.0 smithi069 running (2m) 38s ago 31m 53.4M 2992M 19.0.0-2778-ga971fd5f b96c509eef84 a0e57e3fb494 2024-04-14T14:24:12.422 INFO:teuthology.orchestra.run.smithi069.stdout:osd.1 smithi069 running (94s) 38s ago 31m 51.5M 2992M 19.0.0-2778-ga971fd5f b96c509eef84 cf73d17919bb 2024-04-14T14:24:12.422 INFO:teuthology.orchestra.run.smithi069.stdout:osd.2 smithi069 running (43s) 38s ago 31m 12.7M 2992M 19.0.0-2778-ga971fd5f b96c509eef84 c0efba5ef451 2024-04-14T14:24:12.423 INFO:teuthology.orchestra.run.smithi069.stdout:osd.3 smithi069 running (30m) 38s ago 30m 57.5M 2992M 17.2.0 e1d6a67b021e 0925ba3da801 2024-04-14T14:24:12.423 INFO:teuthology.orchestra.run.smithi069.stdout:osd.4 smithi122 running (30m) 3m ago 30m 56.1M 3248M 17.2.0 e1d6a67b021e c00975caf771 2024-04-14T14:24:12.423 INFO:teuthology.orchestra.run.smithi069.stdout:osd.5 smithi122 running (29m) 3m ago 29m 55.2M 3248M 17.2.0 e1d6a67b021e 5f52d217ceb8 2024-04-14T14:24:12.423 INFO:teuthology.orchestra.run.smithi069.stdout:osd.6 smithi122 running (29m) 3m ago 29m 52.4M 3248M 17.2.0 e1d6a67b021e a2acfec61475 2024-04-14T14:24:12.423 INFO:teuthology.orchestra.run.smithi069.stdout:osd.7 smithi122 running (29m) 3m ago 29m 57.1M 3248M 17.2.0 e1d6a67b021e 62411cd06801 2024-04-14T14:24:12.423 INFO:teuthology.orchestra.run.smithi069.stdout:prometheus.a smithi122 *:9095 running (6m) 3m ago 28m 66.4M - 2.43.0 a07b618ecd1d 5c1f41d3bbe6 2024-04-14T14:24:12.423 INFO:teuthology.orchestra.run.smithi069.stdout:rgw.foo.smithi069.wesedz smithi069 *:8000 running (26m) 38s ago 26m 79.4M - 17.2.0 e1d6a67b021e 16de491440f3 2024-04-14T14:24:12.423 INFO:teuthology.orchestra.run.smithi069.stdout:rgw.foo.smithi122.jiobyv smithi122 *:8000 running (26m) 3m ago 26m 75.8M - 17.2.0 e1d6a67b021e 6d16de445dba 2024-04-14T14:24:12.835 INFO:teuthology.orchestra.run.smithi069.stdout:{ 2024-04-14T14:24:12.835 INFO:teuthology.orchestra.run.smithi069.stdout: "mon": { 2024-04-14T14:24:12.835 INFO:teuthology.orchestra.run.smithi069.stdout: "ceph version 19.0.0-2778-ga971fd5f (a971fd5f5719038c70b817bafacd6d5400b1d888) squid (dev)": 3 2024-04-14T14:24:12.835 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-14T14:24:12.835 INFO:teuthology.orchestra.run.smithi069.stdout: "mgr": { 2024-04-14T14:24:12.835 INFO:teuthology.orchestra.run.smithi069.stdout: "ceph version 19.0.0-2778-ga971fd5f (a971fd5f5719038c70b817bafacd6d5400b1d888) squid (dev)": 2 2024-04-14T14:24:12.835 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-14T14:24:12.836 INFO:teuthology.orchestra.run.smithi069.stdout: "osd": { 2024-04-14T14:24:12.836 INFO:teuthology.orchestra.run.smithi069.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 5, 2024-04-14T14:24:12.836 INFO:teuthology.orchestra.run.smithi069.stdout: "ceph version 19.0.0-2778-ga971fd5f (a971fd5f5719038c70b817bafacd6d5400b1d888) squid (dev)": 3 2024-04-14T14:24:12.836 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-14T14:24:12.836 INFO:teuthology.orchestra.run.smithi069.stdout: "rgw": { 2024-04-14T14:24:12.836 INFO:teuthology.orchestra.run.smithi069.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-04-14T14:24:12.836 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-14T14:24:12.836 INFO:teuthology.orchestra.run.smithi069.stdout: "overall": { 2024-04-14T14:24:12.836 INFO:teuthology.orchestra.run.smithi069.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 7, 2024-04-14T14:24:12.836 INFO:teuthology.orchestra.run.smithi069.stdout: "ceph version 19.0.0-2778-ga971fd5f (a971fd5f5719038c70b817bafacd6d5400b1d888) squid (dev)": 8 2024-04-14T14:24:12.836 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-14T14:24:12.836 INFO:teuthology.orchestra.run.smithi069.stdout:} 2024-04-14T14:24:13.185 INFO:teuthology.orchestra.run.smithi069.stdout:{ 2024-04-14T14:24:13.185 INFO:teuthology.orchestra.run.smithi069.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888", 2024-04-14T14:24:13.185 INFO:teuthology.orchestra.run.smithi069.stdout: "in_progress": true, 2024-04-14T14:24:13.185 INFO:teuthology.orchestra.run.smithi069.stdout: "which": "Upgrading daemons of type(s) crash,osd", 2024-04-14T14:24:13.185 INFO:teuthology.orchestra.run.smithi069.stdout: "services_complete": [], 2024-04-14T14:24:13.185 INFO:teuthology.orchestra.run.smithi069.stdout: "progress": "3/8 daemons upgraded", 2024-04-14T14:24:13.185 INFO:teuthology.orchestra.run.smithi069.stdout: "message": "Currently upgrading osd daemons", 2024-04-14T14:24:13.185 INFO:teuthology.orchestra.run.smithi069.stdout: "is_paused": false 2024-04-14T14:24:13.186 INFO:teuthology.orchestra.run.smithi069.stdout:} 2024-04-14T14:24:13.817 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:24:13 smithi069 ceph-mon[88636]: from='client.54343 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T14:24:13.817 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:24:13 smithi069 ceph-mon[88636]: from='client.54349 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T14:24:13.817 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:24:13 smithi069 ceph-mon[88636]: from='client.54355 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T14:24:13.817 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:24:13 smithi069 ceph-mon[88636]: pgmap v130: 161 pgs: 161 active+clean; 585 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:24:13.817 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:24:13 smithi069 ceph-mon[88636]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:24:13.817 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:24:13 smithi069 ceph-mon[88636]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-04-14T14:24:13.817 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:24:13 smithi069 ceph-mon[88636]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T14:24:13.817 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:24:13 smithi069 ceph-mon[88636]: from='client.? 172.21.15.69:0/2343963479' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T14:24:13.817 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:24:13 smithi069 ceph-mon[83680]: from='client.54343 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T14:24:13.818 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:24:13 smithi069 ceph-mon[83680]: from='client.54349 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T14:24:13.818 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:24:13 smithi069 ceph-mon[83680]: from='client.54355 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T14:24:13.818 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:24:13 smithi069 ceph-mon[83680]: pgmap v130: 161 pgs: 161 active+clean; 585 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:24:13.818 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:24:13 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:24:13.818 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:24:13 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-04-14T14:24:13.818 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:24:13 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T14:24:13.818 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:24:13 smithi069 ceph-mon[83680]: from='client.? 172.21.15.69:0/2343963479' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T14:24:14.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:24:13 smithi122 ceph-mon[67998]: from='client.54343 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T14:24:14.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:24:13 smithi122 ceph-mon[67998]: from='client.54349 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T14:24:14.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:24:13 smithi122 ceph-mon[67998]: from='client.54355 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T14:24:14.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:24:13 smithi122 ceph-mon[67998]: pgmap v130: 161 pgs: 161 active+clean; 585 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:24:14.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:24:13 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:24:14.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:24:13 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-04-14T14:24:14.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:24:13 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T14:24:14.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:24:13 smithi122 ceph-mon[67998]: from='client.? 172.21.15.69:0/2343963479' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T14:24:14.816 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:24:14 smithi069 ceph-mon[83680]: Upgrade: Updating osd.3 2024-04-14T14:24:14.816 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:24:14 smithi069 ceph-mon[83680]: Deploying daemon osd.3 on smithi069 2024-04-14T14:24:14.817 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:24:14 smithi069 ceph-mon[83680]: from='client.44430 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T14:24:14.817 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:24:14 smithi069 ceph-mon[88636]: Upgrade: Updating osd.3 2024-04-14T14:24:14.817 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:24:14 smithi069 ceph-mon[88636]: Deploying daemon osd.3 on smithi069 2024-04-14T14:24:14.817 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:24:14 smithi069 ceph-mon[88636]: from='client.44430 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T14:24:15.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:24:14 smithi122 ceph-mon[67998]: Upgrade: Updating osd.3 2024-04-14T14:24:15.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:24:14 smithi122 ceph-mon[67998]: Deploying daemon osd.3 on smithi069 2024-04-14T14:24:15.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:24:14 smithi122 ceph-mon[67998]: from='client.44430 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T14:24:15.884 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:24:15 smithi069 ceph-mon[83680]: pgmap v131: 161 pgs: 161 active+clean; 585 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:24:15.884 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:24:15 smithi069 ceph-mon[88636]: pgmap v131: 161 pgs: 161 active+clean; 585 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:24:16.094 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:24:15 smithi122 ceph-mon[67998]: pgmap v131: 161 pgs: 161 active+clean; 585 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:24:16.317 INFO:journalctl@ceph.osd.3.smithi069.stdout:Apr 14 14:24:15 smithi069 systemd[1]: Stopping Ceph osd.3 for dc326fc2-fa65-11ee-bc8e-c7b262605968... 2024-04-14T14:24:16.317 INFO:journalctl@ceph.osd.3.smithi069.stdout:Apr 14 14:24:16 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-osd-3[41193]: 2024-04-14T14:24:16.116+0000 7fc665af2700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.3 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-14T14:24:16.317 INFO:journalctl@ceph.osd.3.smithi069.stdout:Apr 14 14:24:16 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-osd-3[41193]: 2024-04-14T14:24:16.116+0000 7fc665af2700 -1 osd.3 130 *** Got signal Terminated *** 2024-04-14T14:24:16.317 INFO:journalctl@ceph.osd.3.smithi069.stdout:Apr 14 14:24:16 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-osd-3[41193]: 2024-04-14T14:24:16.116+0000 7fc665af2700 -1 osd.3 130 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-14T14:24:16.816 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:24:16 smithi069 ceph-mon[83680]: osd.3 marked itself down and dead 2024-04-14T14:24:16.817 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 14 14:24:16 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-y[63038]: ::ffff:172.21.15.122 - - [14/Apr/2024:14:24:16] "GET /metrics HTTP/1.1" 200 38183 "" "Prometheus/2.43.0" 2024-04-14T14:24:16.817 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:24:16 smithi069 ceph-mon[88636]: osd.3 marked itself down and dead 2024-04-14T14:24:17.094 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:24:16 smithi122 ceph-mon[67998]: osd.3 marked itself down and dead 2024-04-14T14:24:17.253 INFO:journalctl@ceph.osd.3.smithi069.stdout:Apr 14 14:24:16 smithi069 podman[113742]: 2024-04-14 14:24:16.963589497 +0000 UTC m=+0.939415899 container died 0925ba3da80149e7085de7260a1312f977d04eff65e4b6886d903c4049a90c51 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-osd-3, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, io.buildah.version=1.19.8, ceph=True, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, CEPH_POINT_RELEASE=-17.2.0, release=754, vendor=Red Hat, Inc., architecture=x86_64, build-date=2022-05-03T08:36:31.336870, com.redhat.component=centos-stream-container, version=8, vcs-type=git, GIT_REPO=https://github.com/ceph/ceph-container.git, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, name=centos-stream, maintainer=Guillaume Abrioux , com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, GIT_BRANCH=HEAD, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, GIT_CLEAN=True, io.openshift.expose-services=, io.openshift.tags=base centos centos-stream, io.k8s.display-name=CentOS Stream 8, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, distribution-scope=public, RELEASE=HEAD) 2024-04-14T14:24:17.566 INFO:journalctl@ceph.osd.3.smithi069.stdout:Apr 14 14:24:17 smithi069 podman[113742]: 2024-04-14 14:24:17.450395037 +0000 UTC m=+1.426221490 container cleanup 0925ba3da80149e7085de7260a1312f977d04eff65e4b6886d903c4049a90c51 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-osd-3, io.k8s.display-name=CentOS Stream 8, ceph=True, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_REPO=https://github.com/ceph/ceph-container.git, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, io.openshift.expose-services=, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, RELEASE=HEAD, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, CEPH_POINT_RELEASE=-17.2.0, architecture=x86_64, maintainer=Guillaume Abrioux , release=754, build-date=2022-05-03T08:36:31.336870, distribution-scope=public, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., vendor=Red Hat, Inc., GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, name=centos-stream, com.redhat.component=centos-stream-container, version=8, vcs-type=git, io.buildah.version=1.19.8, io.openshift.tags=base centos centos-stream) 2024-04-14T14:24:17.567 INFO:journalctl@ceph.osd.3.smithi069.stdout:Apr 14 14:24:17 smithi069 bash[113742]: ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-osd-3 2024-04-14T14:24:17.890 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:24:17 smithi069 ceph-mon[83680]: pgmap v132: 161 pgs: 161 active+clean; 585 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:24:17.890 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:24:17 smithi069 ceph-mon[83680]: Health check failed: 1 osds down (OSD_DOWN) 2024-04-14T14:24:17.891 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:24:17 smithi069 ceph-mon[83680]: osdmap e131: 8 total, 7 up, 8 in 2024-04-14T14:24:17.891 INFO:journalctl@ceph.osd.3.smithi069.stdout:Apr 14 14:24:17 smithi069 podman[113759]: 2024-04-14 14:24:17.760659308 +0000 UTC m=+0.798051769 container remove 0925ba3da80149e7085de7260a1312f977d04eff65e4b6886d903c4049a90c51 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-osd-3, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, RELEASE=HEAD, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, ceph=True, io.openshift.expose-services=, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, architecture=x86_64, io.openshift.tags=base centos centos-stream, build-date=2022-05-03T08:36:31.336870, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, CEPH_POINT_RELEASE=-17.2.0, io.k8s.display-name=CentOS Stream 8, distribution-scope=public, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, io.buildah.version=1.19.8, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, version=8, vendor=Red Hat, Inc., GIT_CLEAN=True, name=centos-stream, GIT_REPO=https://github.com/ceph/ceph-container.git, vcs-type=git, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, release=754, com.redhat.component=centos-stream-container) 2024-04-14T14:24:17.891 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:24:17 smithi069 ceph-mon[88636]: pgmap v132: 161 pgs: 161 active+clean; 585 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:24:17.891 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:24:17 smithi069 ceph-mon[88636]: Health check failed: 1 osds down (OSD_DOWN) 2024-04-14T14:24:17.891 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:24:17 smithi069 ceph-mon[88636]: osdmap e131: 8 total, 7 up, 8 in 2024-04-14T14:24:18.094 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:24:17 smithi122 ceph-mon[67998]: pgmap v132: 161 pgs: 161 active+clean; 585 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:24:18.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:24:17 smithi122 ceph-mon[67998]: Health check failed: 1 osds down (OSD_DOWN) 2024-04-14T14:24:18.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:24:17 smithi122 ceph-mon[67998]: osdmap e131: 8 total, 7 up, 8 in 2024-04-14T14:24:18.252 INFO:journalctl@ceph.osd.3.smithi069.stdout:Apr 14 14:24:18 smithi069 podman[113847]: 2024-04-14 14:24:18.024620402 +0000 UTC m=+0.018043758 image pull b96c509eef841460b3271f5fdd331860da87b004524f4ed59414aa3680183b53 quay.ceph.io/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888 2024-04-14T14:24:18.252 INFO:journalctl@ceph.osd.3.smithi069.stdout:Apr 14 14:24:18 smithi069 podman[113847]: 2024-04-14 14:24:18.128147568 +0000 UTC m=+0.121570908 container create bf9000e46e2eb50a3aa60ec32cda4e4808ef8d64d04250cec26aa7b68903ff23 (image=quay.ceph.io/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888, name=ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-osd-3-deactivate, maintainer=Guillaume Abrioux , GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240403, org.label-schema.vendor=CentOS, RELEASE=main-a971fd5) 2024-04-14T14:24:18.566 INFO:journalctl@ceph.osd.3.smithi069.stdout:Apr 14 14:24:18 smithi069 podman[113847]: 2024-04-14 14:24:18.356029105 +0000 UTC m=+0.349452459 container init bf9000e46e2eb50a3aa60ec32cda4e4808ef8d64d04250cec26aa7b68903ff23 (image=quay.ceph.io/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888, name=ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-osd-3-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, RELEASE=main-a971fd5, GIT_BRANCH=HEAD, org.label-schema.build-date=20240403, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, ceph=True, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0) 2024-04-14T14:24:18.567 INFO:journalctl@ceph.osd.3.smithi069.stdout:Apr 14 14:24:18 smithi069 podman[113847]: 2024-04-14 14:24:18.359466435 +0000 UTC m=+0.352889778 container start bf9000e46e2eb50a3aa60ec32cda4e4808ef8d64d04250cec26aa7b68903ff23 (image=quay.ceph.io/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888, name=ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-osd-3-deactivate, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.schema-version=1.0, RELEASE=main-a971fd5, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.build-date=20240403, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.vendor=CentOS, GIT_CLEAN=True) 2024-04-14T14:24:18.567 INFO:journalctl@ceph.osd.3.smithi069.stdout:Apr 14 14:24:18 smithi069 podman[113847]: 2024-04-14 14:24:18.44720298 +0000 UTC m=+0.440626349 container attach bf9000e46e2eb50a3aa60ec32cda4e4808ef8d64d04250cec26aa7b68903ff23 (image=quay.ceph.io/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888, name=ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-osd-3-deactivate, org.label-schema.license=GPLv2, RELEASE=main-a971fd5, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, ceph=True, io.buildah.version=1.33.5, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240403) 2024-04-14T14:24:18.943 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:24:18 smithi069 ceph-mon[83680]: osdmap e132: 8 total, 7 up, 8 in 2024-04-14T14:24:18.943 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:24:18 smithi069 ceph-mon[88636]: osdmap e132: 8 total, 7 up, 8 in 2024-04-14T14:24:18.944 INFO:journalctl@ceph.osd.3.smithi069.stdout:Apr 14 14:24:18 smithi069 podman[113847]: 2024-04-14 14:24:18.654461719 +0000 UTC m=+0.647885059 container died bf9000e46e2eb50a3aa60ec32cda4e4808ef8d64d04250cec26aa7b68903ff23 (image=quay.ceph.io/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888, name=ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-osd-3-deactivate, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, GIT_CLEAN=True, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240403, org.label-schema.license=GPLv2, RELEASE=main-a971fd5, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD) 2024-04-14T14:24:19.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:24:18 smithi122 ceph-mon[67998]: osdmap e132: 8 total, 7 up, 8 in 2024-04-14T14:24:19.567 INFO:journalctl@ceph.osd.3.smithi069.stdout:Apr 14 14:24:19 smithi069 podman[113873]: 2024-04-14 14:24:19.309194612 +0000 UTC m=+0.645465046 container remove bf9000e46e2eb50a3aa60ec32cda4e4808ef8d64d04250cec26aa7b68903ff23 (image=quay.ceph.io/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888, name=ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-osd-3-deactivate, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.build-date=20240403, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=main-a971fd5, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_CLEAN=True, org.label-schema.license=GPLv2) 2024-04-14T14:24:19.567 INFO:journalctl@ceph.osd.3.smithi069.stdout:Apr 14 14:24:19 smithi069 systemd[1]: ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968@osd.3.service: Deactivated successfully. 2024-04-14T14:24:19.567 INFO:journalctl@ceph.osd.3.smithi069.stdout:Apr 14 14:24:19 smithi069 systemd[1]: Stopped Ceph osd.3 for dc326fc2-fa65-11ee-bc8e-c7b262605968. 2024-04-14T14:24:19.567 INFO:journalctl@ceph.osd.3.smithi069.stdout:Apr 14 14:24:19 smithi069 systemd[1]: ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968@osd.3.service: Consumed 15.992s CPU time. 2024-04-14T14:24:19.567 INFO:journalctl@ceph.osd.3.smithi069.stdout:Apr 14 14:24:19 smithi069 systemd[1]: Starting Ceph osd.3 for dc326fc2-fa65-11ee-bc8e-c7b262605968... 2024-04-14T14:24:19.888 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:24:19 smithi069 ceph-mon[83680]: from='client.25417 -' entity='client.iscsi.foo.smithi069.yuumtr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T14:24:19.888 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:24:19 smithi069 ceph-mon[83680]: pgmap v135: 161 pgs: 7 active+undersized+degraded, 41 peering, 14 active+undersized, 99 active+clean; 585 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 511 B/s rd, 0 op/s; 25/627 objects degraded (3.987%) 2024-04-14T14:24:19.888 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:24:19 smithi069 ceph-mon[83680]: Health check failed: Reduced data availability: 10 pgs inactive, 15 pgs peering (PG_AVAILABILITY) 2024-04-14T14:24:19.888 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:24:19 smithi069 ceph-mon[83680]: Health check failed: Degraded data redundancy: 25/627 objects degraded (3.987%), 7 pgs degraded (PG_DEGRADED) 2024-04-14T14:24:19.888 INFO:journalctl@ceph.osd.3.smithi069.stdout:Apr 14 14:24:19 smithi069 podman[113978]: 2024-04-14 14:24:19.660590635 +0000 UTC m=+0.019052330 image pull b96c509eef841460b3271f5fdd331860da87b004524f4ed59414aa3680183b53 quay.ceph.io/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888 2024-04-14T14:24:19.888 INFO:journalctl@ceph.osd.3.smithi069.stdout:Apr 14 14:24:19 smithi069 podman[113978]: 2024-04-14 14:24:19.777468921 +0000 UTC m=+0.135930589 container create ac9984a808ffe6033998448fef8a888a62e28ff10ac210019867910acf3d083c (image=quay.ceph.io/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888, name=ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-osd-3-activate, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=main-a971fd5, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240403, ceph=True, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_CLEAN=True) 2024-04-14T14:24:19.889 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:24:19 smithi069 ceph-mon[88636]: from='client.25417 -' entity='client.iscsi.foo.smithi069.yuumtr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T14:24:19.889 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:24:19 smithi069 ceph-mon[88636]: pgmap v135: 161 pgs: 7 active+undersized+degraded, 41 peering, 14 active+undersized, 99 active+clean; 585 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 511 B/s rd, 0 op/s; 25/627 objects degraded (3.987%) 2024-04-14T14:24:19.889 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:24:19 smithi069 ceph-mon[88636]: Health check failed: Reduced data availability: 10 pgs inactive, 15 pgs peering (PG_AVAILABILITY) 2024-04-14T14:24:19.889 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:24:19 smithi069 ceph-mon[88636]: Health check failed: Degraded data redundancy: 25/627 objects degraded (3.987%), 7 pgs degraded (PG_DEGRADED) 2024-04-14T14:24:20.094 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:24:19 smithi122 ceph-mon[67998]: from='client.25417 -' entity='client.iscsi.foo.smithi069.yuumtr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T14:24:20.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:24:19 smithi122 ceph-mon[67998]: pgmap v135: 161 pgs: 7 active+undersized+degraded, 41 peering, 14 active+undersized, 99 active+clean; 585 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 511 B/s rd, 0 op/s; 25/627 objects degraded (3.987%) 2024-04-14T14:24:20.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:24:19 smithi122 ceph-mon[67998]: Health check failed: Reduced data availability: 10 pgs inactive, 15 pgs peering (PG_AVAILABILITY) 2024-04-14T14:24:20.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:24:19 smithi122 ceph-mon[67998]: Health check failed: Degraded data redundancy: 25/627 objects degraded (3.987%), 7 pgs degraded (PG_DEGRADED) 2024-04-14T14:24:20.316 INFO:journalctl@ceph.osd.3.smithi069.stdout:Apr 14 14:24:19 smithi069 podman[113978]: 2024-04-14 14:24:19.9882492 +0000 UTC m=+0.346710881 container init ac9984a808ffe6033998448fef8a888a62e28ff10ac210019867910acf3d083c (image=quay.ceph.io/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888, name=ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-osd-3-activate, io.buildah.version=1.33.5, RELEASE=main-a971fd5, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.build-date=20240403) 2024-04-14T14:24:20.317 INFO:journalctl@ceph.osd.3.smithi069.stdout:Apr 14 14:24:19 smithi069 podman[113978]: 2024-04-14 14:24:19.991528672 +0000 UTC m=+0.349990345 container start ac9984a808ffe6033998448fef8a888a62e28ff10ac210019867910acf3d083c (image=quay.ceph.io/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888, name=ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-osd-3-activate, RELEASE=main-a971fd5, maintainer=Guillaume Abrioux , GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, ceph=True, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240403, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD) 2024-04-14T14:24:20.317 INFO:journalctl@ceph.osd.3.smithi069.stdout:Apr 14 14:24:20 smithi069 podman[113978]: 2024-04-14 14:24:20.063121244 +0000 UTC m=+0.421582914 container attach ac9984a808ffe6033998448fef8a888a62e28ff10ac210019867910acf3d083c (image=quay.ceph.io/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888, name=ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-osd-3-activate, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240403, ceph=True, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, RELEASE=main-a971fd5, org.label-schema.schema-version=1.0, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux ) 2024-04-14T14:24:20.784 INFO:journalctl@ceph.osd.3.smithi069.stdout:Apr 14 14:24:20 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-osd-3-activate[114001]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2024-04-14T14:24:20.784 INFO:journalctl@ceph.osd.3.smithi069.stdout:Apr 14 14:24:20 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-osd-3-activate[114001]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-3 --no-mon-config --dev /dev/mapper/vg_nvme-lv_1 2024-04-14T14:24:20.784 INFO:journalctl@ceph.osd.3.smithi069.stdout:Apr 14 14:24:20 smithi069 bash[113978]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2024-04-14T14:24:20.784 INFO:journalctl@ceph.osd.3.smithi069.stdout:Apr 14 14:24:20 smithi069 bash[113978]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-3 --no-mon-config --dev /dev/mapper/vg_nvme-lv_1 2024-04-14T14:24:20.785 INFO:journalctl@ceph.osd.3.smithi069.stdout:Apr 14 14:24:20 smithi069 bash[113978]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_1 2024-04-14T14:24:20.785 INFO:journalctl@ceph.osd.3.smithi069.stdout:Apr 14 14:24:20 smithi069 bash[113978]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-0 2024-04-14T14:24:20.785 INFO:journalctl@ceph.osd.3.smithi069.stdout:Apr 14 14:24:20 smithi069 bash[113978]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-3/block 2024-04-14T14:24:20.785 INFO:journalctl@ceph.osd.3.smithi069.stdout:Apr 14 14:24:20 smithi069 bash[113978]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2024-04-14T14:24:20.785 INFO:journalctl@ceph.osd.3.smithi069.stdout:Apr 14 14:24:20 smithi069 bash[113978]: --> ceph-volume raw activate successful for osd ID: 3 2024-04-14T14:24:20.785 INFO:journalctl@ceph.osd.3.smithi069.stdout:Apr 14 14:24:20 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-osd-3-activate[114001]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_1 2024-04-14T14:24:20.785 INFO:journalctl@ceph.osd.3.smithi069.stdout:Apr 14 14:24:20 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-osd-3-activate[114001]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-0 2024-04-14T14:24:20.785 INFO:journalctl@ceph.osd.3.smithi069.stdout:Apr 14 14:24:20 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-osd-3-activate[114001]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-3/block 2024-04-14T14:24:20.785 INFO:journalctl@ceph.osd.3.smithi069.stdout:Apr 14 14:24:20 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-osd-3-activate[114001]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2024-04-14T14:24:20.785 INFO:journalctl@ceph.osd.3.smithi069.stdout:Apr 14 14:24:20 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-osd-3-activate[114001]: --> ceph-volume raw activate successful for osd ID: 3 2024-04-14T14:24:20.785 INFO:journalctl@ceph.osd.3.smithi069.stdout:Apr 14 14:24:20 smithi069 podman[113978]: 2024-04-14 14:24:20.46390232 +0000 UTC m=+0.822363994 container died ac9984a808ffe6033998448fef8a888a62e28ff10ac210019867910acf3d083c (image=quay.ceph.io/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888, name=ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-osd-3-activate, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=main-a971fd5, ceph=True, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, io.buildah.version=1.33.5, org.label-schema.build-date=20240403) 2024-04-14T14:24:21.435 INFO:journalctl@ceph.osd.3.smithi069.stdout:Apr 14 14:24:21 smithi069 podman[114139]: 2024-04-14 14:24:21.125636466 +0000 UTC m=+0.656844530 container remove ac9984a808ffe6033998448fef8a888a62e28ff10ac210019867910acf3d083c (image=quay.ceph.io/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888, name=ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-osd-3-activate, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, ceph=True, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.build-date=20240403, CEPH_POINT_RELEASE=, RELEASE=main-a971fd5) 2024-04-14T14:24:21.436 INFO:journalctl@ceph.osd.3.smithi069.stdout:Apr 14 14:24:21 smithi069 podman[114207]: 2024-04-14 14:24:21.318234836 +0000 UTC m=+0.093082865 container create d13460da0a65c304df961bf789bfab5641d98dd1203ab79193a183d03a7e0f46 (image=quay.ceph.io/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888, name=ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-osd-3, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240403, org.label-schema.license=GPLv2, ceph=True, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, RELEASE=main-a971fd5, org.label-schema.vendor=CentOS) 2024-04-14T14:24:21.436 INFO:journalctl@ceph.osd.3.smithi069.stdout:Apr 14 14:24:21 smithi069 podman[114207]: 2024-04-14 14:24:21.24343369 +0000 UTC m=+0.018281684 image pull b96c509eef841460b3271f5fdd331860da87b004524f4ed59414aa3680183b53 quay.ceph.io/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888 2024-04-14T14:24:21.816 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:24:21 smithi069 ceph-mon[83680]: pgmap v136: 161 pgs: 7 active+undersized+degraded, 41 peering, 14 active+undersized, 99 active+clean; 585 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 25/627 objects degraded (3.987%) 2024-04-14T14:24:21.817 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:24:21 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:24:21.817 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:24:21 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T14:24:21.817 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:24:21 smithi069 ceph-mon[88636]: pgmap v136: 161 pgs: 7 active+undersized+degraded, 41 peering, 14 active+undersized, 99 active+clean; 585 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 25/627 objects degraded (3.987%) 2024-04-14T14:24:21.817 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:24:21 smithi069 ceph-mon[88636]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:24:21.817 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:24:21 smithi069 ceph-mon[88636]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T14:24:21.817 INFO:journalctl@ceph.osd.3.smithi069.stdout:Apr 14 14:24:21 smithi069 podman[114207]: 2024-04-14 14:24:21.545754211 +0000 UTC m=+0.320602198 container init d13460da0a65c304df961bf789bfab5641d98dd1203ab79193a183d03a7e0f46 (image=quay.ceph.io/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888, name=ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-osd-3, org.label-schema.build-date=20240403, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , RELEASE=main-a971fd5, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0) 2024-04-14T14:24:21.817 INFO:journalctl@ceph.osd.3.smithi069.stdout:Apr 14 14:24:21 smithi069 podman[114207]: 2024-04-14 14:24:21.548892209 +0000 UTC m=+0.323740205 container start d13460da0a65c304df961bf789bfab5641d98dd1203ab79193a183d03a7e0f46 (image=quay.ceph.io/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888, name=ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-osd-3, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, RELEASE=main-a971fd5, org.label-schema.build-date=20240403, ceph=True) 2024-04-14T14:24:21.817 INFO:journalctl@ceph.osd.3.smithi069.stdout:Apr 14 14:24:21 smithi069 bash[114207]: d13460da0a65c304df961bf789bfab5641d98dd1203ab79193a183d03a7e0f46 2024-04-14T14:24:21.818 INFO:journalctl@ceph.osd.3.smithi069.stdout:Apr 14 14:24:21 smithi069 systemd[1]: Started Ceph osd.3 for dc326fc2-fa65-11ee-bc8e-c7b262605968. 2024-04-14T14:24:22.094 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:24:21 smithi122 ceph-mon[67998]: pgmap v136: 161 pgs: 7 active+undersized+degraded, 41 peering, 14 active+undersized, 99 active+clean; 585 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 25/627 objects degraded (3.987%) 2024-04-14T14:24:22.094 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:24:21 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:24:22.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:24:21 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T14:24:22.509 INFO:journalctl@ceph.osd.3.smithi069.stdout:Apr 14 14:24:22 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-osd-3[114221]: 2024-04-14T14:24:22.100+0000 7f7fe11277c0 -1 Falling back to public interface 2024-04-14T14:24:22.816 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:24:22 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:24:22.816 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:24:22 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:24:22.817 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:24:22 smithi069 ceph-mon[83680]: pgmap v137: 161 pgs: 23 active+undersized+degraded, 39 active+undersized, 99 active+clean; 585 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 511 B/s rd, 0 op/s; 79/627 objects degraded (12.600%) 2024-04-14T14:24:22.817 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:24:22 smithi069 ceph-mon[88636]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:24:22.817 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:24:22 smithi069 ceph-mon[88636]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:24:22.817 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:24:22 smithi069 ceph-mon[88636]: pgmap v137: 161 pgs: 23 active+undersized+degraded, 39 active+undersized, 99 active+clean; 585 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 511 B/s rd, 0 op/s; 79/627 objects degraded (12.600%) 2024-04-14T14:24:23.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:24:22 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:24:23.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:24:22 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:24:23.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:24:22 smithi122 ceph-mon[67998]: pgmap v137: 161 pgs: 23 active+undersized+degraded, 39 active+undersized, 99 active+clean; 585 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 511 B/s rd, 0 op/s; 79/627 objects degraded (12.600%) 2024-04-14T14:24:23.816 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:24:23 smithi069 ceph-mon[83680]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 10 pgs inactive, 15 pgs peering) 2024-04-14T14:24:23.817 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:24:23 smithi069 ceph-mon[88636]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 10 pgs inactive, 15 pgs peering) 2024-04-14T14:24:24.094 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:24:23 smithi122 ceph-mon[67998]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 10 pgs inactive, 15 pgs peering) 2024-04-14T14:24:25.066 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:24:24 smithi069 ceph-mon[83680]: pgmap v138: 161 pgs: 23 active+undersized+degraded, 39 active+undersized, 99 active+clean; 585 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 511 B/s rd, 0 op/s; 79/627 objects degraded (12.600%) 2024-04-14T14:24:25.067 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:24:24 smithi069 ceph-mon[88636]: pgmap v138: 161 pgs: 23 active+undersized+degraded, 39 active+undersized, 99 active+clean; 585 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 511 B/s rd, 0 op/s; 79/627 objects degraded (12.600%) 2024-04-14T14:24:25.067 INFO:journalctl@ceph.osd.3.smithi069.stdout:Apr 14 14:24:24 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-osd-3[114221]: 2024-04-14T14:24:24.824+0000 7f7fe11277c0 -1 osd.3 0 read_superblock omap replica is missing. 2024-04-14T14:24:25.067 INFO:journalctl@ceph.osd.3.smithi069.stdout:Apr 14 14:24:24 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-osd-3[114221]: 2024-04-14T14:24:24.852+0000 7f7fe11277c0 -1 osd.3 130 log_to_monitors true 2024-04-14T14:24:25.094 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:24:24 smithi122 ceph-mon[67998]: pgmap v138: 161 pgs: 23 active+undersized+degraded, 39 active+undersized, 99 active+clean; 585 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 511 B/s rd, 0 op/s; 79/627 objects degraded (12.600%) 2024-04-14T14:24:26.066 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:24:25 smithi069 ceph-mon[83680]: from='osd.3 [v2:172.21.15.69:6826/354018715,v1:172.21.15.69:6827/354018715]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-04-14T14:24:26.067 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:24:25 smithi069 ceph-mon[88636]: from='osd.3 [v2:172.21.15.69:6826/354018715,v1:172.21.15.69:6827/354018715]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-04-14T14:24:26.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:24:25 smithi122 ceph-mon[67998]: from='osd.3 [v2:172.21.15.69:6826/354018715,v1:172.21.15.69:6827/354018715]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-04-14T14:24:26.502 INFO:journalctl@ceph.osd.3.smithi069.stdout:Apr 14 14:24:26 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-osd-3[114221]: 2024-04-14T14:24:26.177+0000 7f7fce577700 -1 osd.3 130 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-14T14:24:26.816 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 14 14:24:26 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-y[63038]: ::ffff:172.21.15.122 - - [14/Apr/2024:14:24:26] "GET /metrics HTTP/1.1" 200 38258 "" "Prometheus/2.43.0" 2024-04-14T14:24:26.817 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:24:26 smithi069 ceph-mon[83680]: from='osd.3 [v2:172.21.15.69:6826/354018715,v1:172.21.15.69:6827/354018715]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-04-14T14:24:26.817 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:24:26 smithi069 ceph-mon[83680]: osdmap e133: 8 total, 7 up, 8 in 2024-04-14T14:24:26.817 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:24:26 smithi069 ceph-mon[83680]: from='osd.3 [v2:172.21.15.69:6826/354018715,v1:172.21.15.69:6827/354018715]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi069", "root=default"]}]: dispatch 2024-04-14T14:24:26.817 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:24:26 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:24:26.817 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:24:26 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:24:26.817 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:24:26 smithi069 ceph-mon[83680]: from='osd.3 [v2:172.21.15.69:6826/354018715,v1:172.21.15.69:6827/354018715]' entity='osd.3' 2024-04-14T14:24:26.817 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:24:26 smithi069 ceph-mon[83680]: pgmap v140: 161 pgs: 23 active+undersized+degraded, 39 active+undersized, 99 active+clean; 585 KiB data, 174 MiB used, 715 GiB / 715 GiB avail; 343 B/s rd, 0 op/s; 79/627 objects degraded (12.600%) 2024-04-14T14:24:26.817 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:24:26 smithi069 ceph-mon[88636]: from='osd.3 [v2:172.21.15.69:6826/354018715,v1:172.21.15.69:6827/354018715]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-04-14T14:24:26.817 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:24:26 smithi069 ceph-mon[88636]: osdmap e133: 8 total, 7 up, 8 in 2024-04-14T14:24:26.818 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:24:26 smithi069 ceph-mon[88636]: from='osd.3 [v2:172.21.15.69:6826/354018715,v1:172.21.15.69:6827/354018715]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi069", "root=default"]}]: dispatch 2024-04-14T14:24:26.818 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:24:26 smithi069 ceph-mon[88636]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:24:26.818 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:24:26 smithi069 ceph-mon[88636]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:24:26.818 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:24:26 smithi069 ceph-mon[88636]: from='osd.3 [v2:172.21.15.69:6826/354018715,v1:172.21.15.69:6827/354018715]' entity='osd.3' 2024-04-14T14:24:26.818 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:24:26 smithi069 ceph-mon[88636]: pgmap v140: 161 pgs: 23 active+undersized+degraded, 39 active+undersized, 99 active+clean; 585 KiB data, 174 MiB used, 715 GiB / 715 GiB avail; 343 B/s rd, 0 op/s; 79/627 objects degraded (12.600%) 2024-04-14T14:24:27.094 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:24:26 smithi122 ceph-mon[67998]: from='osd.3 [v2:172.21.15.69:6826/354018715,v1:172.21.15.69:6827/354018715]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-04-14T14:24:27.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:24:26 smithi122 ceph-mon[67998]: osdmap e133: 8 total, 7 up, 8 in 2024-04-14T14:24:27.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:24:26 smithi122 ceph-mon[67998]: from='osd.3 [v2:172.21.15.69:6826/354018715,v1:172.21.15.69:6827/354018715]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi069", "root=default"]}]: dispatch 2024-04-14T14:24:27.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:24:26 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:24:27.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:24:26 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:24:27.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:24:26 smithi122 ceph-mon[67998]: from='osd.3 [v2:172.21.15.69:6826/354018715,v1:172.21.15.69:6827/354018715]' entity='osd.3' 2024-04-14T14:24:27.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:24:26 smithi122 ceph-mon[67998]: pgmap v140: 161 pgs: 23 active+undersized+degraded, 39 active+undersized, 99 active+clean; 585 KiB data, 174 MiB used, 715 GiB / 715 GiB avail; 343 B/s rd, 0 op/s; 79/627 objects degraded (12.600%) 2024-04-14T14:24:27.964 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:24:27 smithi069 ceph-mon[88636]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-04-14T14:24:27.964 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:24:27 smithi069 ceph-mon[88636]: osd.3 [v2:172.21.15.69:6826/354018715,v1:172.21.15.69:6827/354018715] boot 2024-04-14T14:24:27.964 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:24:27 smithi069 ceph-mon[88636]: osdmap e134: 8 total, 8 up, 8 in 2024-04-14T14:24:27.964 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:24:27 smithi069 ceph-mon[88636]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-14T14:24:27.964 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:24:27 smithi069 ceph-mon[88636]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:24:27.964 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:24:27 smithi069 ceph-mon[88636]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:24:27.965 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:24:27 smithi069 ceph-mon[83680]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-04-14T14:24:27.965 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:24:27 smithi069 ceph-mon[83680]: osd.3 [v2:172.21.15.69:6826/354018715,v1:172.21.15.69:6827/354018715] boot 2024-04-14T14:24:27.965 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:24:27 smithi069 ceph-mon[83680]: osdmap e134: 8 total, 8 up, 8 in 2024-04-14T14:24:27.965 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:24:27 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-14T14:24:27.965 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:24:27 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:24:27.965 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:24:27 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:24:28.094 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:24:27 smithi122 ceph-mon[67998]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-04-14T14:24:28.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:24:27 smithi122 ceph-mon[67998]: osd.3 [v2:172.21.15.69:6826/354018715,v1:172.21.15.69:6827/354018715] boot 2024-04-14T14:24:28.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:24:27 smithi122 ceph-mon[67998]: osdmap e134: 8 total, 8 up, 8 in 2024-04-14T14:24:28.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:24:27 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-14T14:24:28.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:24:27 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:24:28.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:24:27 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:24:29.433 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:24:29 smithi069 ceph-mon[83680]: from='client.25417 -' entity='client.iscsi.foo.smithi069.yuumtr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T14:24:29.434 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:24:29 smithi069 ceph-mon[83680]: osdmap e135: 8 total, 8 up, 8 in 2024-04-14T14:24:29.434 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:24:29 smithi069 ceph-mon[83680]: Health check update: Degraded data redundancy: 79/627 objects degraded (12.600%), 23 pgs degraded (PG_DEGRADED) 2024-04-14T14:24:29.434 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:24:29 smithi069 ceph-mon[83680]: pgmap v143: 161 pgs: 30 peering, 11 active+undersized+degraded, 21 active+undersized, 99 active+clean; 585 KiB data, 178 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 42/627 objects degraded (6.699%) 2024-04-14T14:24:29.434 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:24:29 smithi069 ceph-mon[88636]: from='client.25417 -' entity='client.iscsi.foo.smithi069.yuumtr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T14:24:29.434 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:24:29 smithi069 ceph-mon[88636]: osdmap e135: 8 total, 8 up, 8 in 2024-04-14T14:24:29.434 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:24:29 smithi069 ceph-mon[88636]: Health check update: Degraded data redundancy: 79/627 objects degraded (12.600%), 23 pgs degraded (PG_DEGRADED) 2024-04-14T14:24:29.434 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:24:29 smithi069 ceph-mon[88636]: pgmap v143: 161 pgs: 30 peering, 11 active+undersized+degraded, 21 active+undersized, 99 active+clean; 585 KiB data, 178 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 42/627 objects degraded (6.699%) 2024-04-14T14:24:29.594 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:24:29 smithi122 ceph-mon[67998]: from='client.25417 -' entity='client.iscsi.foo.smithi069.yuumtr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T14:24:29.595 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:24:29 smithi122 ceph-mon[67998]: osdmap e135: 8 total, 8 up, 8 in 2024-04-14T14:24:29.595 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:24:29 smithi122 ceph-mon[67998]: Health check update: Degraded data redundancy: 79/627 objects degraded (12.600%), 23 pgs degraded (PG_DEGRADED) 2024-04-14T14:24:29.595 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:24:29 smithi122 ceph-mon[67998]: pgmap v143: 161 pgs: 30 peering, 11 active+undersized+degraded, 21 active+undersized, 99 active+clean; 585 KiB data, 178 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 42/627 objects degraded (6.699%) 2024-04-14T14:24:31.066 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 14 14:24:30 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-y[63038]: 2024-04-14T14:24:30.978+0000 7fdd38467700 -1 mgr.server reply reply (16) Device or resource busy unsafe to stop osd(s) at this time (19 PGs are or would become offline) 2024-04-14T14:24:32.067 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:24:31 smithi069 ceph-mon[83680]: pgmap v144: 161 pgs: 30 peering, 11 active+undersized+degraded, 21 active+undersized, 99 active+clean; 585 KiB data, 178 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 42/627 objects degraded (6.699%) 2024-04-14T14:24:32.067 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:24:31 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:24:32.067 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:24:31 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:24:32.067 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:24:31 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T14:24:32.067 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:24:31 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-14T14:24:32.067 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:24:31 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:24:32.067 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:24:31 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-14T14:24:32.067 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:24:31 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T14:24:32.067 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:24:31 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T14:24:32.067 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:24:31 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T14:24:32.067 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:24:31 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["4"], "max": 16}]: dispatch 2024-04-14T14:24:32.068 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:24:31 smithi069 ceph-mon[88636]: pgmap v144: 161 pgs: 30 peering, 11 active+undersized+degraded, 21 active+undersized, 99 active+clean; 585 KiB data, 178 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 42/627 objects degraded (6.699%) 2024-04-14T14:24:32.068 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:24:31 smithi069 ceph-mon[88636]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:24:32.068 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:24:31 smithi069 ceph-mon[88636]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:24:32.068 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:24:31 smithi069 ceph-mon[88636]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T14:24:32.068 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:24:31 smithi069 ceph-mon[88636]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-14T14:24:32.068 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:24:31 smithi069 ceph-mon[88636]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:24:32.068 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:24:31 smithi069 ceph-mon[88636]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-14T14:24:32.068 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:24:31 smithi069 ceph-mon[88636]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T14:24:32.068 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:24:31 smithi069 ceph-mon[88636]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T14:24:32.068 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:24:31 smithi069 ceph-mon[88636]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T14:24:32.068 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:24:31 smithi069 ceph-mon[88636]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["4"], "max": 16}]: dispatch 2024-04-14T14:24:32.094 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:24:31 smithi122 ceph-mon[67998]: pgmap v144: 161 pgs: 30 peering, 11 active+undersized+degraded, 21 active+undersized, 99 active+clean; 585 KiB data, 178 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 42/627 objects degraded (6.699%) 2024-04-14T14:24:32.094 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:24:31 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:24:32.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:24:31 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:24:32.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:24:31 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T14:24:32.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:24:31 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-14T14:24:32.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:24:31 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:24:32.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:24:31 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-14T14:24:32.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:24:31 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T14:24:32.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:24:31 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T14:24:32.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:24:31 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T14:24:32.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:24:31 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["4"], "max": 16}]: dispatch 2024-04-14T14:24:33.066 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:24:32 smithi069 ceph-mon[83680]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["4"], "max": 16}]: dispatch 2024-04-14T14:24:33.066 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:24:32 smithi069 ceph-mon[83680]: Upgrade: unsafe to stop osd(s) at this time (19 PGs are or would become offline) 2024-04-14T14:24:33.067 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:24:32 smithi069 ceph-mon[88636]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["4"], "max": 16}]: dispatch 2024-04-14T14:24:33.067 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:24:32 smithi069 ceph-mon[88636]: Upgrade: unsafe to stop osd(s) at this time (19 PGs are or would become offline) 2024-04-14T14:24:33.094 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:24:32 smithi122 ceph-mon[67998]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["4"], "max": 16}]: dispatch 2024-04-14T14:24:33.094 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:24:32 smithi122 ceph-mon[67998]: Upgrade: unsafe to stop osd(s) at this time (19 PGs are or would become offline) 2024-04-14T14:24:34.066 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:24:33 smithi069 ceph-mon[83680]: pgmap v145: 161 pgs: 13 peering, 148 active+clean; 585 KiB data, 178 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-04-14T14:24:34.067 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:24:33 smithi069 ceph-mon[83680]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 42/627 objects degraded (6.699%), 11 pgs degraded) 2024-04-14T14:24:34.067 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:24:33 smithi069 ceph-mon[83680]: Cluster is now healthy 2024-04-14T14:24:34.067 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:24:33 smithi069 ceph-mon[88636]: pgmap v145: 161 pgs: 13 peering, 148 active+clean; 585 KiB data, 178 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-04-14T14:24:34.067 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:24:33 smithi069 ceph-mon[88636]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 42/627 objects degraded (6.699%), 11 pgs degraded) 2024-04-14T14:24:34.067 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:24:33 smithi069 ceph-mon[88636]: Cluster is now healthy 2024-04-14T14:24:34.094 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:24:33 smithi122 ceph-mon[67998]: pgmap v145: 161 pgs: 13 peering, 148 active+clean; 585 KiB data, 178 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-04-14T14:24:34.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:24:33 smithi122 ceph-mon[67998]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 42/627 objects degraded (6.699%), 11 pgs degraded) 2024-04-14T14:24:34.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:24:33 smithi122 ceph-mon[67998]: Cluster is now healthy 2024-04-14T14:24:36.066 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:24:35 smithi069 ceph-mon[83680]: pgmap v146: 161 pgs: 161 active+clean; 585 KiB data, 178 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:24:36.067 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:24:35 smithi069 ceph-mon[88636]: pgmap v146: 161 pgs: 161 active+clean; 585 KiB data, 178 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:24:36.094 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:24:35 smithi122 ceph-mon[67998]: pgmap v146: 161 pgs: 161 active+clean; 585 KiB data, 178 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:24:36.816 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 14 14:24:36 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-y[63038]: ::ffff:172.21.15.122 - - [14/Apr/2024:14:24:36] "GET /metrics HTTP/1.1" 200 38258 "" "Prometheus/2.43.0" 2024-04-14T14:24:37.844 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:24:37 smithi122 ceph-mon[67998]: pgmap v147: 161 pgs: 161 active+clean; 585 KiB data, 178 MiB used, 715 GiB / 715 GiB avail; 545 B/s rd, 0 op/s 2024-04-14T14:24:37.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:24:37 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:24:37.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:24:37 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T14:24:38.066 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:24:37 smithi069 ceph-mon[83680]: pgmap v147: 161 pgs: 161 active+clean; 585 KiB data, 178 MiB used, 715 GiB / 715 GiB avail; 545 B/s rd, 0 op/s 2024-04-14T14:24:38.067 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:24:37 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:24:38.067 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:24:37 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T14:24:38.067 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:24:37 smithi069 ceph-mon[88636]: pgmap v147: 161 pgs: 161 active+clean; 585 KiB data, 178 MiB used, 715 GiB / 715 GiB avail; 545 B/s rd, 0 op/s 2024-04-14T14:24:38.067 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:24:37 smithi069 ceph-mon[88636]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:24:38.067 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:24:37 smithi069 ceph-mon[88636]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T14:24:39.844 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:24:39 smithi122 ceph-mon[67998]: from='client.25417 -' entity='client.iscsi.foo.smithi069.yuumtr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T14:24:39.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:24:39 smithi122 ceph-mon[67998]: pgmap v148: 161 pgs: 161 active+clean; 585 KiB data, 178 MiB used, 715 GiB / 715 GiB avail; 985 B/s rd, 0 op/s 2024-04-14T14:24:40.066 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:24:39 smithi069 ceph-mon[88636]: from='client.25417 -' entity='client.iscsi.foo.smithi069.yuumtr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T14:24:40.067 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:24:39 smithi069 ceph-mon[88636]: pgmap v148: 161 pgs: 161 active+clean; 585 KiB data, 178 MiB used, 715 GiB / 715 GiB avail; 985 B/s rd, 0 op/s 2024-04-14T14:24:40.067 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:24:39 smithi069 ceph-mon[83680]: from='client.25417 -' entity='client.iscsi.foo.smithi069.yuumtr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T14:24:40.067 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:24:39 smithi069 ceph-mon[83680]: pgmap v148: 161 pgs: 161 active+clean; 585 KiB data, 178 MiB used, 715 GiB / 715 GiB avail; 985 B/s rd, 0 op/s 2024-04-14T14:24:41.844 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:24:41 smithi122 ceph-mon[67998]: pgmap v149: 161 pgs: 161 active+clean; 585 KiB data, 178 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:24:42.067 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:24:41 smithi069 ceph-mon[83680]: pgmap v149: 161 pgs: 161 active+clean; 585 KiB data, 178 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:24:42.067 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:24:41 smithi069 ceph-mon[88636]: pgmap v149: 161 pgs: 161 active+clean; 585 KiB data, 178 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:24:43.066 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:24:42 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:24:43.067 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:24:42 smithi069 ceph-mon[88636]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:24:43.094 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:24:42 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:24:43.550 INFO:teuthology.orchestra.run.smithi069.stdout:true 2024-04-14T14:24:43.816 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:24:43 smithi069 ceph-mon[83680]: pgmap v150: 161 pgs: 161 active+clean; 585 KiB data, 178 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:24:43.817 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:24:43 smithi069 ceph-mon[88636]: pgmap v150: 161 pgs: 161 active+clean; 585 KiB data, 178 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:24:44.094 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:24:43 smithi122 ceph-mon[67998]: pgmap v150: 161 pgs: 161 active+clean; 585 KiB data, 178 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:24:44.202 INFO:teuthology.orchestra.run.smithi069.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-14T14:24:44.203 INFO:teuthology.orchestra.run.smithi069.stdout:alertmanager.a smithi069 *:9093,9094 running (6m) 18s ago 28m 26.5M - 0.25.0 c8568f914cd2 7524f106e7bd 2024-04-14T14:24:44.203 INFO:teuthology.orchestra.run.smithi069.stdout:grafana.a smithi122 *:3000 running (6m) 4m ago 27m 81.0M - 9.4.12 2bacad6d85d8 5685178e3eba 2024-04-14T14:24:44.203 INFO:teuthology.orchestra.run.smithi069.stdout:iscsi.foo.smithi069.yuumtr smithi069 running (7m) 18s ago 27m 62.9M - 3.5 e1d6a67b021e 38cc81ea8cfa 2024-04-14T14:24:44.203 INFO:teuthology.orchestra.run.smithi069.stdout:mgr.x smithi122 *:8443,9283,8765 running (7m) 4m ago 32m 450M - 19.0.0-2778-ga971fd5f b96c509eef84 e29977590b43 2024-04-14T14:24:44.203 INFO:teuthology.orchestra.run.smithi069.stdout:mgr.y smithi069 *:8443,9283,8765 running (17m) 18s ago 34m 507M - 19.0.0-2778-ga971fd5f b96c509eef84 7af3d29d0e5f 2024-04-14T14:24:44.203 INFO:teuthology.orchestra.run.smithi069.stdout:mon.a smithi069 running (4m) 18s ago 34m 54.7M 2048M 19.0.0-2778-ga971fd5f b96c509eef84 8bcd6b3d3846 2024-04-14T14:24:44.203 INFO:teuthology.orchestra.run.smithi069.stdout:mon.b smithi122 running (5m) 4m ago 32m 33.9M 2048M 19.0.0-2778-ga971fd5f b96c509eef84 05f437a1a9a1 2024-04-14T14:24:44.203 INFO:teuthology.orchestra.run.smithi069.stdout:mon.c smithi069 running (3m) 18s ago 33m 39.3M 2048M 19.0.0-2778-ga971fd5f b96c509eef84 84aa95f11d0c 2024-04-14T14:24:44.203 INFO:teuthology.orchestra.run.smithi069.stdout:node-exporter.a smithi069 *:9100 running (7m) 18s ago 28m 15.0M - 1.5.0 0da6a335fe13 0de0eb2171b9 2024-04-14T14:24:44.203 INFO:teuthology.orchestra.run.smithi069.stdout:node-exporter.b smithi122 *:9100 running (7m) 4m ago 28m 16.1M - 1.5.0 0da6a335fe13 67877bf02d8e 2024-04-14T14:24:44.203 INFO:teuthology.orchestra.run.smithi069.stdout:osd.0 smithi069 running (2m) 18s ago 32m 53.9M 2992M 19.0.0-2778-ga971fd5f b96c509eef84 a0e57e3fb494 2024-04-14T14:24:44.203 INFO:teuthology.orchestra.run.smithi069.stdout:osd.1 smithi069 running (2m) 18s ago 31m 52.9M 2992M 19.0.0-2778-ga971fd5f b96c509eef84 cf73d17919bb 2024-04-14T14:24:44.203 INFO:teuthology.orchestra.run.smithi069.stdout:osd.2 smithi069 running (75s) 18s ago 31m 41.9M 2992M 19.0.0-2778-ga971fd5f b96c509eef84 c0efba5ef451 2024-04-14T14:24:44.204 INFO:teuthology.orchestra.run.smithi069.stdout:osd.3 smithi069 running (22s) 18s ago 31m 15.1M 2992M 19.0.0-2778-ga971fd5f b96c509eef84 d13460da0a65 2024-04-14T14:24:44.204 INFO:teuthology.orchestra.run.smithi069.stdout:osd.4 smithi122 running (30m) 4m ago 30m 56.1M 3248M 17.2.0 e1d6a67b021e c00975caf771 2024-04-14T14:24:44.204 INFO:teuthology.orchestra.run.smithi069.stdout:osd.5 smithi122 running (30m) 4m ago 30m 55.2M 3248M 17.2.0 e1d6a67b021e 5f52d217ceb8 2024-04-14T14:24:44.204 INFO:teuthology.orchestra.run.smithi069.stdout:osd.6 smithi122 running (30m) 4m ago 30m 52.4M 3248M 17.2.0 e1d6a67b021e a2acfec61475 2024-04-14T14:24:44.204 INFO:teuthology.orchestra.run.smithi069.stdout:osd.7 smithi122 running (29m) 4m ago 29m 57.1M 3248M 17.2.0 e1d6a67b021e 62411cd06801 2024-04-14T14:24:44.204 INFO:teuthology.orchestra.run.smithi069.stdout:prometheus.a smithi122 *:9095 running (6m) 4m ago 29m 66.4M - 2.43.0 a07b618ecd1d 5c1f41d3bbe6 2024-04-14T14:24:44.204 INFO:teuthology.orchestra.run.smithi069.stdout:rgw.foo.smithi069.wesedz smithi069 *:8000 running (27m) 18s ago 27m 79.7M - 17.2.0 e1d6a67b021e 16de491440f3 2024-04-14T14:24:44.204 INFO:teuthology.orchestra.run.smithi069.stdout:rgw.foo.smithi122.jiobyv smithi122 *:8000 running (27m) 4m ago 27m 75.8M - 17.2.0 e1d6a67b021e 6d16de445dba 2024-04-14T14:24:44.600 INFO:teuthology.orchestra.run.smithi069.stdout:{ 2024-04-14T14:24:44.600 INFO:teuthology.orchestra.run.smithi069.stdout: "mon": { 2024-04-14T14:24:44.600 INFO:teuthology.orchestra.run.smithi069.stdout: "ceph version 19.0.0-2778-ga971fd5f (a971fd5f5719038c70b817bafacd6d5400b1d888) squid (dev)": 3 2024-04-14T14:24:44.600 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-14T14:24:44.600 INFO:teuthology.orchestra.run.smithi069.stdout: "mgr": { 2024-04-14T14:24:44.600 INFO:teuthology.orchestra.run.smithi069.stdout: "ceph version 19.0.0-2778-ga971fd5f (a971fd5f5719038c70b817bafacd6d5400b1d888) squid (dev)": 2 2024-04-14T14:24:44.601 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-14T14:24:44.601 INFO:teuthology.orchestra.run.smithi069.stdout: "osd": { 2024-04-14T14:24:44.601 INFO:teuthology.orchestra.run.smithi069.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 4, 2024-04-14T14:24:44.601 INFO:teuthology.orchestra.run.smithi069.stdout: "ceph version 19.0.0-2778-ga971fd5f (a971fd5f5719038c70b817bafacd6d5400b1d888) squid (dev)": 4 2024-04-14T14:24:44.601 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-14T14:24:44.601 INFO:teuthology.orchestra.run.smithi069.stdout: "rgw": { 2024-04-14T14:24:44.601 INFO:teuthology.orchestra.run.smithi069.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-04-14T14:24:44.601 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-14T14:24:44.601 INFO:teuthology.orchestra.run.smithi069.stdout: "overall": { 2024-04-14T14:24:44.601 INFO:teuthology.orchestra.run.smithi069.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 6, 2024-04-14T14:24:44.601 INFO:teuthology.orchestra.run.smithi069.stdout: "ceph version 19.0.0-2778-ga971fd5f (a971fd5f5719038c70b817bafacd6d5400b1d888) squid (dev)": 9 2024-04-14T14:24:44.601 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-14T14:24:44.601 INFO:teuthology.orchestra.run.smithi069.stdout:} 2024-04-14T14:24:44.935 INFO:teuthology.orchestra.run.smithi069.stdout:{ 2024-04-14T14:24:44.936 INFO:teuthology.orchestra.run.smithi069.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888", 2024-04-14T14:24:44.936 INFO:teuthology.orchestra.run.smithi069.stdout: "in_progress": true, 2024-04-14T14:24:44.936 INFO:teuthology.orchestra.run.smithi069.stdout: "which": "Upgrading daemons of type(s) crash,osd", 2024-04-14T14:24:44.936 INFO:teuthology.orchestra.run.smithi069.stdout: "services_complete": [], 2024-04-14T14:24:44.936 INFO:teuthology.orchestra.run.smithi069.stdout: "progress": "4/8 daemons upgraded", 2024-04-14T14:24:44.936 INFO:teuthology.orchestra.run.smithi069.stdout: "message": "Currently upgrading osd daemons", 2024-04-14T14:24:44.936 INFO:teuthology.orchestra.run.smithi069.stdout: "is_paused": false 2024-04-14T14:24:44.936 INFO:teuthology.orchestra.run.smithi069.stdout:} 2024-04-14T14:24:44.936 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:24:44 smithi069 ceph-mon[83680]: from='client.54379 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T14:24:44.937 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:24:44 smithi069 ceph-mon[83680]: from='client.? 172.21.15.69:0/2286651505' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T14:24:44.937 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:24:44 smithi069 ceph-mon[88636]: from='client.54379 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T14:24:44.937 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:24:44 smithi069 ceph-mon[88636]: from='client.? 172.21.15.69:0/2286651505' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T14:24:45.094 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:24:44 smithi122 ceph-mon[67998]: from='client.54379 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T14:24:45.094 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:24:44 smithi122 ceph-mon[67998]: from='client.? 172.21.15.69:0/2286651505' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T14:24:46.067 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:24:45 smithi069 ceph-mon[88636]: from='client.54385 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T14:24:46.067 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:24:45 smithi069 ceph-mon[88636]: from='client.54391 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T14:24:46.067 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:24:45 smithi069 ceph-mon[88636]: pgmap v151: 161 pgs: 161 active+clean; 585 KiB data, 178 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:24:46.067 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:24:45 smithi069 ceph-mon[83680]: from='client.54385 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T14:24:46.067 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:24:45 smithi069 ceph-mon[83680]: from='client.54391 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T14:24:46.067 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:24:45 smithi069 ceph-mon[83680]: pgmap v151: 161 pgs: 161 active+clean; 585 KiB data, 178 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:24:46.094 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:24:45 smithi122 ceph-mon[67998]: from='client.54385 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T14:24:46.094 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:24:45 smithi122 ceph-mon[67998]: from='client.54391 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T14:24:46.094 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:24:45 smithi122 ceph-mon[67998]: pgmap v151: 161 pgs: 161 active+clean; 585 KiB data, 178 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:24:46.818 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:24:46 smithi069 ceph-mon[88636]: from='client.54403 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T14:24:46.818 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:24:46 smithi069 ceph-mon[88636]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["4"], "max": 16}]: dispatch 2024-04-14T14:24:46.818 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:24:46 smithi069 ceph-mon[83680]: from='client.54403 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T14:24:46.818 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:24:46 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["4"], "max": 16}]: dispatch 2024-04-14T14:24:46.818 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 14 14:24:46 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-y[63038]: ::ffff:172.21.15.122 - - [14/Apr/2024:14:24:46] "GET /metrics HTTP/1.1" 200 38266 "" "Prometheus/2.43.0" 2024-04-14T14:24:46.844 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:24:46 smithi122 ceph-mon[67998]: from='client.54403 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T14:24:46.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:24:46 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["4"], "max": 16}]: dispatch 2024-04-14T14:24:47.844 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:24:47 smithi122 ceph-mon[67998]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["4"], "max": 16}]: dispatch 2024-04-14T14:24:47.844 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:24:47 smithi122 ceph-mon[67998]: Upgrade: osd.4 is safe to restart 2024-04-14T14:24:47.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:24:47 smithi122 ceph-mon[67998]: pgmap v152: 161 pgs: 161 active+clean; 585 KiB data, 178 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:24:47.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:24:47 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:24:47.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:24:47 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-04-14T14:24:47.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:24:47 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T14:24:48.067 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:24:47 smithi069 ceph-mon[83680]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["4"], "max": 16}]: dispatch 2024-04-14T14:24:48.067 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:24:47 smithi069 ceph-mon[83680]: Upgrade: osd.4 is safe to restart 2024-04-14T14:24:48.067 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:24:47 smithi069 ceph-mon[83680]: pgmap v152: 161 pgs: 161 active+clean; 585 KiB data, 178 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:24:48.067 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:24:47 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:24:48.067 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:24:47 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-04-14T14:24:48.067 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:24:47 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T14:24:48.067 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:24:47 smithi069 ceph-mon[88636]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["4"], "max": 16}]: dispatch 2024-04-14T14:24:48.067 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:24:47 smithi069 ceph-mon[88636]: Upgrade: osd.4 is safe to restart 2024-04-14T14:24:48.067 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:24:47 smithi069 ceph-mon[88636]: pgmap v152: 161 pgs: 161 active+clean; 585 KiB data, 178 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:24:48.067 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:24:47 smithi069 ceph-mon[88636]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:24:48.068 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:24:47 smithi069 ceph-mon[88636]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-04-14T14:24:48.068 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:24:47 smithi069 ceph-mon[88636]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T14:24:48.844 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:24:48 smithi122 ceph-mon[67998]: Upgrade: Updating osd.4 2024-04-14T14:24:48.844 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:24:48 smithi122 ceph-mon[67998]: Deploying daemon osd.4 on smithi122 2024-04-14T14:24:49.066 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:24:48 smithi069 ceph-mon[83680]: Upgrade: Updating osd.4 2024-04-14T14:24:49.066 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:24:48 smithi069 ceph-mon[83680]: Deploying daemon osd.4 on smithi122 2024-04-14T14:24:49.067 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:24:48 smithi069 ceph-mon[88636]: Upgrade: Updating osd.4 2024-04-14T14:24:49.067 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:24:48 smithi069 ceph-mon[88636]: Deploying daemon osd.4 on smithi122 2024-04-14T14:24:49.816 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:24:49 smithi069 ceph-mon[83680]: from='client.25417 -' entity='client.iscsi.foo.smithi069.yuumtr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T14:24:49.816 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:24:49 smithi069 ceph-mon[83680]: pgmap v153: 161 pgs: 161 active+clean; 585 KiB data, 178 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:24:49.817 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:24:49 smithi069 ceph-mon[88636]: from='client.25417 -' entity='client.iscsi.foo.smithi069.yuumtr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T14:24:49.817 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:24:49 smithi069 ceph-mon[88636]: pgmap v153: 161 pgs: 161 active+clean; 585 KiB data, 178 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:24:49.844 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:24:49 smithi122 ceph-mon[67998]: from='client.25417 -' entity='client.iscsi.foo.smithi069.yuumtr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T14:24:49.844 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:24:49 smithi122 ceph-mon[67998]: pgmap v153: 161 pgs: 161 active+clean; 585 KiB data, 178 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:24:50.800 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:24:50 smithi122 ceph-mon[67998]: pgmap v154: 161 pgs: 161 active+clean; 585 KiB data, 178 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:24:51.058 INFO:journalctl@ceph.osd.4.smithi122.stdout:Apr 14 14:24:50 smithi122 systemd[1]: Stopping Ceph osd.4 for dc326fc2-fa65-11ee-bc8e-c7b262605968... 2024-04-14T14:24:51.066 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:24:50 smithi069 ceph-mon[83680]: pgmap v154: 161 pgs: 161 active+clean; 585 KiB data, 178 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:24:51.067 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:24:50 smithi069 ceph-mon[88636]: pgmap v154: 161 pgs: 161 active+clean; 585 KiB data, 178 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:24:51.344 INFO:journalctl@ceph.osd.4.smithi122.stdout:Apr 14 14:24:51 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-osd-4[29170]: 2024-04-14T14:24:51.077+0000 7f1bb17ee700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.4 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-14T14:24:51.344 INFO:journalctl@ceph.osd.4.smithi122.stdout:Apr 14 14:24:51 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-osd-4[29170]: 2024-04-14T14:24:51.077+0000 7f1bb17ee700 -1 osd.4 135 *** Got signal Terminated *** 2024-04-14T14:24:51.345 INFO:journalctl@ceph.osd.4.smithi122.stdout:Apr 14 14:24:51 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-osd-4[29170]: 2024-04-14T14:24:51.077+0000 7f1bb17ee700 -1 osd.4 135 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-14T14:24:51.964 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:24:51 smithi122 ceph-mon[67998]: osd.4 marked itself down and dead 2024-04-14T14:24:51.964 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:24:51 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T14:24:52.066 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:24:51 smithi069 ceph-mon[83680]: osd.4 marked itself down and dead 2024-04-14T14:24:52.067 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:24:51 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T14:24:52.067 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:24:51 smithi069 ceph-mon[88636]: osd.4 marked itself down and dead 2024-04-14T14:24:52.067 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:24:51 smithi069 ceph-mon[88636]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T14:24:52.344 INFO:journalctl@ceph.osd.4.smithi122.stdout:Apr 14 14:24:52 smithi122 podman[77704]: 2024-04-14 14:24:52.076159315 +0000 UTC m=+1.100220245 container died c00975caf771f6ed08e6dc7331ae82516422ef4ee1f86e88dfc8314d43e8332f (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-osd-4, architecture=x86_64, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, io.openshift.tags=base centos centos-stream, maintainer=Guillaume Abrioux , distribution-scope=public, CEPH_POINT_RELEASE=-17.2.0, io.openshift.expose-services=, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, vendor=Red Hat, Inc., RELEASE=HEAD, com.redhat.component=centos-stream-container, name=centos-stream, build-date=2022-05-03T08:36:31.336870, io.buildah.version=1.19.8, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, GIT_BRANCH=HEAD, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, io.k8s.display-name=CentOS Stream 8, GIT_REPO=https://github.com/ceph/ceph-container.git, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, release=754, version=8, GIT_CLEAN=True, ceph=True, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, vcs-type=git) 2024-04-14T14:24:52.686 INFO:journalctl@ceph.osd.4.smithi122.stdout:Apr 14 14:24:52 smithi122 podman[77704]: 2024-04-14 14:24:52.435741233 +0000 UTC m=+1.459802167 container cleanup c00975caf771f6ed08e6dc7331ae82516422ef4ee1f86e88dfc8314d43e8332f (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-osd-4, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, RELEASE=HEAD, GIT_CLEAN=True, maintainer=Guillaume Abrioux , summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, vendor=Red Hat, Inc., CEPH_POINT_RELEASE=-17.2.0, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, distribution-scope=public, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, com.redhat.component=centos-stream-container, io.openshift.tags=base centos centos-stream, build-date=2022-05-03T08:36:31.336870, version=8, vcs-type=git, architecture=x86_64, io.openshift.expose-services=, ceph=True, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, GIT_REPO=https://github.com/ceph/ceph-container.git, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., io.k8s.display-name=CentOS Stream 8, name=centos-stream, release=754, io.buildah.version=1.19.8, GIT_BRANCH=HEAD) 2024-04-14T14:24:52.686 INFO:journalctl@ceph.osd.4.smithi122.stdout:Apr 14 14:24:52 smithi122 bash[77704]: ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-osd-4 2024-04-14T14:24:53.051 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:24:52 smithi122 ceph-mon[67998]: Health check failed: 1 osds down (OSD_DOWN) 2024-04-14T14:24:53.051 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:24:52 smithi122 ceph-mon[67998]: osdmap e136: 8 total, 7 up, 8 in 2024-04-14T14:24:53.051 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:24:52 smithi122 ceph-mon[67998]: pgmap v156: 161 pgs: 30 peering, 14 stale+active+clean, 117 active+clean; 585 KiB data, 178 MiB used, 715 GiB / 715 GiB avail; 921 B/s rd, 0 op/s 2024-04-14T14:24:53.052 INFO:journalctl@ceph.osd.4.smithi122.stdout:Apr 14 14:24:52 smithi122 podman[77732]: 2024-04-14 14:24:52.738044366 +0000 UTC m=+0.658760945 container remove c00975caf771f6ed08e6dc7331ae82516422ef4ee1f86e88dfc8314d43e8332f (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-osd-4, maintainer=Guillaume Abrioux , io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, name=centos-stream, io.openshift.tags=base centos centos-stream, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, RELEASE=HEAD, vendor=Red Hat, Inc., CEPH_POINT_RELEASE=-17.2.0, release=754, io.buildah.version=1.19.8, io.k8s.display-name=CentOS Stream 8, com.redhat.component=centos-stream-container, io.openshift.expose-services=, distribution-scope=public, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, GIT_REPO=https://github.com/ceph/ceph-container.git, vcs-type=git, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, ceph=True, architecture=x86_64, version=8, build-date=2022-05-03T08:36:31.336870, GIT_CLEAN=True, GIT_BRANCH=HEAD) 2024-04-14T14:24:53.067 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:24:52 smithi069 ceph-mon[83680]: Health check failed: 1 osds down (OSD_DOWN) 2024-04-14T14:24:53.067 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:24:52 smithi069 ceph-mon[83680]: osdmap e136: 8 total, 7 up, 8 in 2024-04-14T14:24:53.067 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:24:52 smithi069 ceph-mon[83680]: pgmap v156: 161 pgs: 30 peering, 14 stale+active+clean, 117 active+clean; 585 KiB data, 178 MiB used, 715 GiB / 715 GiB avail; 921 B/s rd, 0 op/s 2024-04-14T14:24:53.067 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:24:52 smithi069 ceph-mon[88636]: Health check failed: 1 osds down (OSD_DOWN) 2024-04-14T14:24:53.067 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:24:52 smithi069 ceph-mon[88636]: osdmap e136: 8 total, 7 up, 8 in 2024-04-14T14:24:53.068 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:24:52 smithi069 ceph-mon[88636]: pgmap v156: 161 pgs: 30 peering, 14 stale+active+clean, 117 active+clean; 585 KiB data, 178 MiB used, 715 GiB / 715 GiB avail; 921 B/s rd, 0 op/s 2024-04-14T14:24:53.345 INFO:journalctl@ceph.osd.4.smithi122.stdout:Apr 14 14:24:53 smithi122 podman[77829]: 2024-04-14 14:24:52.953852441 +0000 UTC m=+0.018022981 image pull b96c509eef841460b3271f5fdd331860da87b004524f4ed59414aa3680183b53 quay.ceph.io/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888 2024-04-14T14:24:53.345 INFO:journalctl@ceph.osd.4.smithi122.stdout:Apr 14 14:24:53 smithi122 podman[77829]: 2024-04-14 14:24:53.064031116 +0000 UTC m=+0.128201657 container create 68ea64384b4ab4dc1a7b87da6845e8cea7f4365d5133f20f72e95da831379108 (image=quay.ceph.io/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888, name=ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-osd-4-deactivate, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, maintainer=Guillaume Abrioux , RELEASE=main-a971fd5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240403, GIT_BRANCH=HEAD, ceph=True, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS) 2024-04-14T14:24:53.345 INFO:journalctl@ceph.osd.4.smithi122.stdout:Apr 14 14:24:53 smithi122 podman[77829]: 2024-04-14 14:24:53.275463637 +0000 UTC m=+0.339634177 container init 68ea64384b4ab4dc1a7b87da6845e8cea7f4365d5133f20f72e95da831379108 (image=quay.ceph.io/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888, name=ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-osd-4-deactivate, org.label-schema.vendor=CentOS, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.build-date=20240403, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, RELEASE=main-a971fd5, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True) 2024-04-14T14:24:53.345 INFO:journalctl@ceph.osd.4.smithi122.stdout:Apr 14 14:24:53 smithi122 podman[77829]: 2024-04-14 14:24:53.27850412 +0000 UTC m=+0.342674670 container start 68ea64384b4ab4dc1a7b87da6845e8cea7f4365d5133f20f72e95da831379108 (image=quay.ceph.io/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888, name=ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-osd-4-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.build-date=20240403, GIT_BRANCH=HEAD, RELEASE=main-a971fd5, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, ceph=True) 2024-04-14T14:24:53.686 INFO:journalctl@ceph.osd.4.smithi122.stdout:Apr 14 14:24:53 smithi122 podman[77829]: 2024-04-14 14:24:53.350282322 +0000 UTC m=+0.414452874 container attach 68ea64384b4ab4dc1a7b87da6845e8cea7f4365d5133f20f72e95da831379108 (image=quay.ceph.io/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888, name=ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-osd-4-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240403, CEPH_POINT_RELEASE=, RELEASE=main-a971fd5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS) 2024-04-14T14:24:53.687 INFO:journalctl@ceph.osd.4.smithi122.stdout:Apr 14 14:24:53 smithi122 podman[77829]: 2024-04-14 14:24:53.566326512 +0000 UTC m=+0.630497044 container died 68ea64384b4ab4dc1a7b87da6845e8cea7f4365d5133f20f72e95da831379108 (image=quay.ceph.io/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888, name=ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-osd-4-deactivate, GIT_CLEAN=True, org.label-schema.build-date=20240403, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, RELEASE=main-a971fd5, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5) 2024-04-14T14:24:54.066 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:24:53 smithi069 ceph-mon[83680]: Health check failed: Reduced data availability: 6 pgs inactive, 9 pgs peering (PG_AVAILABILITY) 2024-04-14T14:24:54.067 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:24:53 smithi069 ceph-mon[83680]: osdmap e137: 8 total, 7 up, 8 in 2024-04-14T14:24:54.067 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:24:53 smithi069 ceph-mon[88636]: Health check failed: Reduced data availability: 6 pgs inactive, 9 pgs peering (PG_AVAILABILITY) 2024-04-14T14:24:54.067 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:24:53 smithi069 ceph-mon[88636]: osdmap e137: 8 total, 7 up, 8 in 2024-04-14T14:24:54.089 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:24:53 smithi122 ceph-mon[67998]: Health check failed: Reduced data availability: 6 pgs inactive, 9 pgs peering (PG_AVAILABILITY) 2024-04-14T14:24:54.089 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:24:53 smithi122 ceph-mon[67998]: osdmap e137: 8 total, 7 up, 8 in 2024-04-14T14:24:54.344 INFO:journalctl@ceph.osd.4.smithi122.stdout:Apr 14 14:24:54 smithi122 podman[77854]: 2024-04-14 14:24:54.088289157 +0000 UTC m=+0.517180660 container remove 68ea64384b4ab4dc1a7b87da6845e8cea7f4365d5133f20f72e95da831379108 (image=quay.ceph.io/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888, name=ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-osd-4-deactivate, org.label-schema.build-date=20240403, CEPH_POINT_RELEASE=, ceph=True, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, RELEASE=main-a971fd5) 2024-04-14T14:24:54.345 INFO:journalctl@ceph.osd.4.smithi122.stdout:Apr 14 14:24:54 smithi122 systemd[1]: ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968@osd.4.service: Deactivated successfully. 2024-04-14T14:24:54.345 INFO:journalctl@ceph.osd.4.smithi122.stdout:Apr 14 14:24:54 smithi122 systemd[1]: Stopped Ceph osd.4 for dc326fc2-fa65-11ee-bc8e-c7b262605968. 2024-04-14T14:24:54.345 INFO:journalctl@ceph.osd.4.smithi122.stdout:Apr 14 14:24:54 smithi122 systemd[1]: ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968@osd.4.service: Consumed 17.068s CPU time. 2024-04-14T14:24:54.345 INFO:journalctl@ceph.osd.4.smithi122.stdout:Apr 14 14:24:54 smithi122 systemd[1]: Starting Ceph osd.4 for dc326fc2-fa65-11ee-bc8e-c7b262605968... 2024-04-14T14:24:54.802 INFO:journalctl@ceph.osd.4.smithi122.stdout:Apr 14 14:24:54 smithi122 podman[77962]: 2024-04-14 14:24:54.416990443 +0000 UTC m=+0.018311303 image pull b96c509eef841460b3271f5fdd331860da87b004524f4ed59414aa3680183b53 quay.ceph.io/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888 2024-04-14T14:24:54.802 INFO:journalctl@ceph.osd.4.smithi122.stdout:Apr 14 14:24:54 smithi122 podman[77962]: 2024-04-14 14:24:54.556043607 +0000 UTC m=+0.157364466 container create 631f31c9bb3784bdc12c985920eeb61e14ada3bc3df6ac6205a05f309198b259 (image=quay.ceph.io/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888, name=ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-osd-4-activate, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240403, org.label-schema.license=GPLv2, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , RELEASE=main-a971fd5) 2024-04-14T14:24:54.802 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:24:54 smithi122 ceph-mon[67998]: pgmap v158: 161 pgs: 7 active+undersized+degraded, 37 peering, 9 stale+active+clean, 2 active+undersized, 106 active+clean; 585 KiB data, 178 MiB used, 715 GiB / 715 GiB avail; 28/627 objects degraded (4.466%) 2024-04-14T14:24:55.067 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:24:54 smithi069 ceph-mon[83680]: pgmap v158: 161 pgs: 7 active+undersized+degraded, 37 peering, 9 stale+active+clean, 2 active+undersized, 106 active+clean; 585 KiB data, 178 MiB used, 715 GiB / 715 GiB avail; 28/627 objects degraded (4.466%) 2024-04-14T14:24:55.067 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:24:54 smithi069 ceph-mon[88636]: pgmap v158: 161 pgs: 7 active+undersized+degraded, 37 peering, 9 stale+active+clean, 2 active+undersized, 106 active+clean; 585 KiB data, 178 MiB used, 715 GiB / 715 GiB avail; 28/627 objects degraded (4.466%) 2024-04-14T14:24:55.094 INFO:journalctl@ceph.osd.4.smithi122.stdout:Apr 14 14:24:54 smithi122 podman[77962]: 2024-04-14 14:24:54.800726409 +0000 UTC m=+0.402047266 container init 631f31c9bb3784bdc12c985920eeb61e14ada3bc3df6ac6205a05f309198b259 (image=quay.ceph.io/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888, name=ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-osd-4-activate, io.buildah.version=1.33.5, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, GIT_CLEAN=True, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, RELEASE=main-a971fd5, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240403, GIT_BRANCH=HEAD) 2024-04-14T14:24:55.095 INFO:journalctl@ceph.osd.4.smithi122.stdout:Apr 14 14:24:54 smithi122 podman[77962]: 2024-04-14 14:24:54.804687037 +0000 UTC m=+0.406007898 container start 631f31c9bb3784bdc12c985920eeb61e14ada3bc3df6ac6205a05f309198b259 (image=quay.ceph.io/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888, name=ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-osd-4-activate, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, ceph=True, io.buildah.version=1.33.5, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, GIT_CLEAN=True, org.label-schema.build-date=20240403, RELEASE=main-a971fd5, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD) 2024-04-14T14:24:55.095 INFO:journalctl@ceph.osd.4.smithi122.stdout:Apr 14 14:24:54 smithi122 podman[77962]: 2024-04-14 14:24:54.892431679 +0000 UTC m=+0.493752537 container attach 631f31c9bb3784bdc12c985920eeb61e14ada3bc3df6ac6205a05f309198b259 (image=quay.ceph.io/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888, name=ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-osd-4-activate, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=main-a971fd5, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240403, io.buildah.version=1.33.5, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, GIT_CLEAN=True, GIT_BRANCH=HEAD) 2024-04-14T14:24:55.505 INFO:journalctl@ceph.osd.4.smithi122.stdout:Apr 14 14:24:55 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-osd-4-activate[77976]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2024-04-14T14:24:55.505 INFO:journalctl@ceph.osd.4.smithi122.stdout:Apr 14 14:24:55 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-osd-4-activate[77976]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-4 --no-mon-config --dev /dev/mapper/vg_nvme-lv_4 2024-04-14T14:24:55.505 INFO:journalctl@ceph.osd.4.smithi122.stdout:Apr 14 14:24:55 smithi122 bash[77962]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2024-04-14T14:24:55.505 INFO:journalctl@ceph.osd.4.smithi122.stdout:Apr 14 14:24:55 smithi122 bash[77962]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-4 --no-mon-config --dev /dev/mapper/vg_nvme-lv_4 2024-04-14T14:24:55.505 INFO:journalctl@ceph.osd.4.smithi122.stdout:Apr 14 14:24:55 smithi122 bash[77962]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_4 2024-04-14T14:24:55.505 INFO:journalctl@ceph.osd.4.smithi122.stdout:Apr 14 14:24:55 smithi122 bash[77962]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-3 2024-04-14T14:24:55.505 INFO:journalctl@ceph.osd.4.smithi122.stdout:Apr 14 14:24:55 smithi122 bash[77962]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-4/block 2024-04-14T14:24:55.505 INFO:journalctl@ceph.osd.4.smithi122.stdout:Apr 14 14:24:55 smithi122 bash[77962]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2024-04-14T14:24:55.506 INFO:journalctl@ceph.osd.4.smithi122.stdout:Apr 14 14:24:55 smithi122 bash[77962]: --> ceph-volume raw activate successful for osd ID: 4 2024-04-14T14:24:55.506 INFO:journalctl@ceph.osd.4.smithi122.stdout:Apr 14 14:24:55 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-osd-4-activate[77976]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_4 2024-04-14T14:24:55.506 INFO:journalctl@ceph.osd.4.smithi122.stdout:Apr 14 14:24:55 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-osd-4-activate[77976]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-3 2024-04-14T14:24:55.506 INFO:journalctl@ceph.osd.4.smithi122.stdout:Apr 14 14:24:55 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-osd-4-activate[77976]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-4/block 2024-04-14T14:24:55.506 INFO:journalctl@ceph.osd.4.smithi122.stdout:Apr 14 14:24:55 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-osd-4-activate[77976]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2024-04-14T14:24:55.506 INFO:journalctl@ceph.osd.4.smithi122.stdout:Apr 14 14:24:55 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-osd-4-activate[77976]: --> ceph-volume raw activate successful for osd ID: 4 2024-04-14T14:24:55.506 INFO:journalctl@ceph.osd.4.smithi122.stdout:Apr 14 14:24:55 smithi122 podman[77962]: 2024-04-14 14:24:55.269252459 +0000 UTC m=+0.870573309 container died 631f31c9bb3784bdc12c985920eeb61e14ada3bc3df6ac6205a05f309198b259 (image=quay.ceph.io/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888, name=ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-osd-4-activate, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, ceph=True, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240403, RELEASE=main-a971fd5) 2024-04-14T14:24:55.806 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:24:55 smithi122 ceph-mon[67998]: Health check failed: Degraded data redundancy: 28/627 objects degraded (4.466%), 7 pgs degraded (PG_DEGRADED) 2024-04-14T14:24:56.066 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:24:55 smithi069 ceph-mon[83680]: Health check failed: Degraded data redundancy: 28/627 objects degraded (4.466%), 7 pgs degraded (PG_DEGRADED) 2024-04-14T14:24:56.067 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:24:55 smithi069 ceph-mon[88636]: Health check failed: Degraded data redundancy: 28/627 objects degraded (4.466%), 7 pgs degraded (PG_DEGRADED) 2024-04-14T14:24:56.094 INFO:journalctl@ceph.osd.4.smithi122.stdout:Apr 14 14:24:55 smithi122 podman[78094]: 2024-04-14 14:24:55.805298482 +0000 UTC m=+0.531198832 container remove 631f31c9bb3784bdc12c985920eeb61e14ada3bc3df6ac6205a05f309198b259 (image=quay.ceph.io/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888, name=ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-osd-4-activate, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=main-a971fd5, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240403, org.label-schema.license=GPLv2, ceph=True, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0) 2024-04-14T14:24:56.095 INFO:journalctl@ceph.osd.4.smithi122.stdout:Apr 14 14:24:56 smithi122 podman[78161]: 2024-04-14 14:24:55.920223951 +0000 UTC m=+0.018138334 image pull b96c509eef841460b3271f5fdd331860da87b004524f4ed59414aa3680183b53 quay.ceph.io/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888 2024-04-14T14:24:56.095 INFO:journalctl@ceph.osd.4.smithi122.stdout:Apr 14 14:24:56 smithi122 podman[78161]: 2024-04-14 14:24:56.023068115 +0000 UTC m=+0.120982497 container create 7a87a024f984e181c2e3243a4f3d5f6097efb32288914d71fbafb502229e5d07 (image=quay.ceph.io/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888, name=ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-osd-4, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, RELEASE=main-a971fd5, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240403, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5) 2024-04-14T14:24:56.594 INFO:journalctl@ceph.osd.4.smithi122.stdout:Apr 14 14:24:56 smithi122 podman[78161]: 2024-04-14 14:24:56.242677342 +0000 UTC m=+0.340591733 container init 7a87a024f984e181c2e3243a4f3d5f6097efb32288914d71fbafb502229e5d07 (image=quay.ceph.io/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888, name=ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-osd-4, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240403, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, ceph=True, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, RELEASE=main-a971fd5) 2024-04-14T14:24:56.595 INFO:journalctl@ceph.osd.4.smithi122.stdout:Apr 14 14:24:56 smithi122 podman[78161]: 2024-04-14 14:24:56.245879353 +0000 UTC m=+0.343793735 container start 7a87a024f984e181c2e3243a4f3d5f6097efb32288914d71fbafb502229e5d07 (image=quay.ceph.io/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888, name=ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-osd-4, RELEASE=main-a971fd5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.build-date=20240403, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, ceph=True, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux ) 2024-04-14T14:24:56.595 INFO:journalctl@ceph.osd.4.smithi122.stdout:Apr 14 14:24:56 smithi122 ceph-osd[78179]: -- 172.21.15.122:0/3151209701 <== mon.2 v2:172.21.15.122:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55a2bb8b9680 con 0x55a2bb883400 2024-04-14T14:24:56.595 INFO:journalctl@ceph.osd.4.smithi122.stdout:Apr 14 14:24:56 smithi122 bash[78161]: 7a87a024f984e181c2e3243a4f3d5f6097efb32288914d71fbafb502229e5d07 2024-04-14T14:24:56.595 INFO:journalctl@ceph.osd.4.smithi122.stdout:Apr 14 14:24:56 smithi122 systemd[1]: Started Ceph osd.4 for dc326fc2-fa65-11ee-bc8e-c7b262605968. 2024-04-14T14:24:56.816 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 14 14:24:56 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-y[63038]: ::ffff:172.21.15.122 - - [14/Apr/2024:14:24:56] "GET /metrics HTTP/1.1" 200 38267 "" "Prometheus/2.43.0" 2024-04-14T14:24:57.094 INFO:journalctl@ceph.osd.4.smithi122.stdout:Apr 14 14:24:56 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-osd-4[78175]: 2024-04-14T14:24:56.791+0000 7f1cc9adf7c0 -1 Falling back to public interface 2024-04-14T14:24:57.594 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:24:57 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:24:57.594 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:24:57 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:24:57.595 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:24:57 smithi122 ceph-mon[67998]: pgmap v159: 161 pgs: 9 active+undersized+degraded, 37 peering, 3 stale+active+clean, 10 active+undersized, 102 active+clean; 585 KiB data, 178 MiB used, 715 GiB / 715 GiB avail; 33/627 objects degraded (5.263%) 2024-04-14T14:24:57.727 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:24:57 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:24:57.728 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:24:57 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:24:57.728 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:24:57 smithi069 ceph-mon[83680]: pgmap v159: 161 pgs: 9 active+undersized+degraded, 37 peering, 3 stale+active+clean, 10 active+undersized, 102 active+clean; 585 KiB data, 178 MiB used, 715 GiB / 715 GiB avail; 33/627 objects degraded (5.263%) 2024-04-14T14:24:57.728 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:24:57 smithi069 ceph-mon[88636]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:24:57.728 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:24:57 smithi069 ceph-mon[88636]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:24:57.728 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:24:57 smithi069 ceph-mon[88636]: pgmap v159: 161 pgs: 9 active+undersized+degraded, 37 peering, 3 stale+active+clean, 10 active+undersized, 102 active+clean; 585 KiB data, 178 MiB used, 715 GiB / 715 GiB avail; 33/627 objects degraded (5.263%) 2024-04-14T14:24:58.594 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:24:58 smithi122 ceph-mon[67998]: Health check update: Reduced data availability: 8 pgs inactive, 12 pgs peering (PG_AVAILABILITY) 2024-04-14T14:24:58.816 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:24:58 smithi069 ceph-mon[83680]: Health check update: Reduced data availability: 8 pgs inactive, 12 pgs peering (PG_AVAILABILITY) 2024-04-14T14:24:58.817 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:24:58 smithi069 ceph-mon[88636]: Health check update: Reduced data availability: 8 pgs inactive, 12 pgs peering (PG_AVAILABILITY) 2024-04-14T14:24:59.740 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:24:59 smithi069 ceph-mon[83680]: from='client.25417 -' entity='client.iscsi.foo.smithi069.yuumtr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T14:24:59.741 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:24:59 smithi069 ceph-mon[83680]: pgmap v160: 161 pgs: 25 active+undersized+degraded, 43 active+undersized, 93 active+clean; 585 KiB data, 179 MiB used, 715 GiB / 715 GiB avail; 106/627 objects degraded (16.906%) 2024-04-14T14:24:59.741 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:24:59 smithi069 ceph-mon[88636]: from='client.25417 -' entity='client.iscsi.foo.smithi069.yuumtr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T14:24:59.741 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:24:59 smithi069 ceph-mon[88636]: pgmap v160: 161 pgs: 25 active+undersized+degraded, 43 active+undersized, 93 active+clean; 585 KiB data, 179 MiB used, 715 GiB / 715 GiB avail; 106/627 objects degraded (16.906%) 2024-04-14T14:24:59.750 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:24:59 smithi122 ceph-mon[67998]: from='client.25417 -' entity='client.iscsi.foo.smithi069.yuumtr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T14:24:59.750 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:24:59 smithi122 ceph-mon[67998]: pgmap v160: 161 pgs: 25 active+undersized+degraded, 43 active+undersized, 93 active+clean; 585 KiB data, 179 MiB used, 715 GiB / 715 GiB avail; 106/627 objects degraded (16.906%) 2024-04-14T14:24:59.750 INFO:journalctl@ceph.osd.4.smithi122.stdout:Apr 14 14:24:59 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-osd-4[78175]: 2024-04-14T14:24:59.503+0000 7f1cc9adf7c0 -1 osd.4 0 read_superblock omap replica is missing. 2024-04-14T14:24:59.750 INFO:journalctl@ceph.osd.4.smithi122.stdout:Apr 14 14:24:59 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-osd-4[78175]: 2024-04-14T14:24:59.536+0000 7f1cc9adf7c0 -1 osd.4 135 log_to_monitors true 2024-04-14T14:25:00.566 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:25:00 smithi069 ceph-mon[83680]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 8 pgs inactive, 12 pgs peering) 2024-04-14T14:25:00.567 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:25:00 smithi069 ceph-mon[83680]: from='osd.4 [v2:172.21.15.122:6800/346611071,v1:172.21.15.122:6801/346611071]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-04-14T14:25:00.567 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:25:00 smithi069 ceph-mon[83680]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-04-14T14:25:00.567 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:25:00 smithi069 ceph-mon[88636]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 8 pgs inactive, 12 pgs peering) 2024-04-14T14:25:00.567 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:25:00 smithi069 ceph-mon[88636]: from='osd.4 [v2:172.21.15.122:6800/346611071,v1:172.21.15.122:6801/346611071]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-04-14T14:25:00.567 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:25:00 smithi069 ceph-mon[88636]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-04-14T14:25:00.844 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:25:00 smithi122 ceph-mon[67998]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 8 pgs inactive, 12 pgs peering) 2024-04-14T14:25:00.844 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:25:00 smithi122 ceph-mon[67998]: from='osd.4 [v2:172.21.15.122:6800/346611071,v1:172.21.15.122:6801/346611071]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-04-14T14:25:00.844 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:25:00 smithi122 ceph-mon[67998]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-04-14T14:25:01.344 INFO:journalctl@ceph.osd.4.smithi122.stdout:Apr 14 14:25:00 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-osd-4[78175]: 2024-04-14T14:25:00.888+0000 7f1cb6f2f700 -1 osd.4 135 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-14T14:25:01.817 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:25:01 smithi069 ceph-mon[83680]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-04-14T14:25:01.817 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:25:01 smithi069 ceph-mon[83680]: from='osd.4 [v2:172.21.15.122:6800/346611071,v1:172.21.15.122:6801/346611071]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi122", "root=default"]}]: dispatch 2024-04-14T14:25:01.817 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:25:01 smithi069 ceph-mon[83680]: osdmap e138: 8 total, 7 up, 8 in 2024-04-14T14:25:01.817 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:25:01 smithi069 ceph-mon[83680]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi122", "root=default"]}]: dispatch 2024-04-14T14:25:01.817 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:25:01 smithi069 ceph-mon[83680]: pgmap v162: 161 pgs: 25 active+undersized+degraded, 43 active+undersized, 93 active+clean; 585 KiB data, 190 MiB used, 715 GiB / 715 GiB avail; 106/627 objects degraded (16.906%) 2024-04-14T14:25:01.818 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:25:01 smithi069 ceph-mon[83680]: from='osd.4 ' entity='osd.4' 2024-04-14T14:25:01.818 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:25:01 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:25:01.818 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:25:01 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:25:01.818 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:25:01 smithi069 ceph-mon[88636]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-04-14T14:25:01.818 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:25:01 smithi069 ceph-mon[88636]: from='osd.4 [v2:172.21.15.122:6800/346611071,v1:172.21.15.122:6801/346611071]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi122", "root=default"]}]: dispatch 2024-04-14T14:25:01.818 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:25:01 smithi069 ceph-mon[88636]: osdmap e138: 8 total, 7 up, 8 in 2024-04-14T14:25:01.819 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:25:01 smithi069 ceph-mon[88636]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi122", "root=default"]}]: dispatch 2024-04-14T14:25:01.819 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:25:01 smithi069 ceph-mon[88636]: pgmap v162: 161 pgs: 25 active+undersized+degraded, 43 active+undersized, 93 active+clean; 585 KiB data, 190 MiB used, 715 GiB / 715 GiB avail; 106/627 objects degraded (16.906%) 2024-04-14T14:25:01.819 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:25:01 smithi069 ceph-mon[88636]: from='osd.4 ' entity='osd.4' 2024-04-14T14:25:01.819 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:25:01 smithi069 ceph-mon[88636]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:25:01.819 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:25:01 smithi069 ceph-mon[88636]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:25:01.844 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:25:01 smithi122 ceph-mon[67998]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-04-14T14:25:01.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:25:01 smithi122 ceph-mon[67998]: from='osd.4 [v2:172.21.15.122:6800/346611071,v1:172.21.15.122:6801/346611071]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi122", "root=default"]}]: dispatch 2024-04-14T14:25:01.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:25:01 smithi122 ceph-mon[67998]: osdmap e138: 8 total, 7 up, 8 in 2024-04-14T14:25:01.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:25:01 smithi122 ceph-mon[67998]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi122", "root=default"]}]: dispatch 2024-04-14T14:25:01.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:25:01 smithi122 ceph-mon[67998]: pgmap v162: 161 pgs: 25 active+undersized+degraded, 43 active+undersized, 93 active+clean; 585 KiB data, 190 MiB used, 715 GiB / 715 GiB avail; 106/627 objects degraded (16.906%) 2024-04-14T14:25:01.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:25:01 smithi122 ceph-mon[67998]: from='osd.4 ' entity='osd.4' 2024-04-14T14:25:01.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:25:01 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:25:01.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:25:01 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:25:02.603 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:25:02 smithi122 ceph-mon[67998]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-04-14T14:25:02.603 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:25:02 smithi122 ceph-mon[67998]: osd.4 [v2:172.21.15.122:6800/346611071,v1:172.21.15.122:6801/346611071] boot 2024-04-14T14:25:02.603 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:25:02 smithi122 ceph-mon[67998]: osdmap e139: 8 total, 8 up, 8 in 2024-04-14T14:25:02.603 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:25:02 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-14T14:25:02.603 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:25:02 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:25:02.603 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:25:02 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:25:02.817 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:25:02 smithi069 ceph-mon[83680]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-04-14T14:25:02.817 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:25:02 smithi069 ceph-mon[83680]: osd.4 [v2:172.21.15.122:6800/346611071,v1:172.21.15.122:6801/346611071] boot 2024-04-14T14:25:02.817 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:25:02 smithi069 ceph-mon[83680]: osdmap e139: 8 total, 8 up, 8 in 2024-04-14T14:25:02.817 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:25:02 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-14T14:25:02.817 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:25:02 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:25:02.817 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:25:02 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:25:02.817 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:25:02 smithi069 ceph-mon[88636]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-04-14T14:25:02.817 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:25:02 smithi069 ceph-mon[88636]: osd.4 [v2:172.21.15.122:6800/346611071,v1:172.21.15.122:6801/346611071] boot 2024-04-14T14:25:02.817 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:25:02 smithi069 ceph-mon[88636]: osdmap e139: 8 total, 8 up, 8 in 2024-04-14T14:25:02.817 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:25:02 smithi069 ceph-mon[88636]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-14T14:25:02.818 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:25:02 smithi069 ceph-mon[88636]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:25:02.818 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:25:02 smithi069 ceph-mon[88636]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:25:03.723 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:25:03 smithi122 ceph-mon[67998]: pgmap v164: 161 pgs: 21 peering, 17 active+undersized+degraded, 30 active+undersized, 93 active+clean; 585 KiB data, 190 MiB used, 715 GiB / 715 GiB avail; 383 B/s rd, 0 op/s; 58/627 objects degraded (9.250%) 2024-04-14T14:25:03.723 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:25:03 smithi122 ceph-mon[67998]: osdmap e140: 8 total, 8 up, 8 in 2024-04-14T14:25:03.723 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:25:03 smithi122 ceph-mon[67998]: Health check update: Degraded data redundancy: 106/627 objects degraded (16.906%), 25 pgs degraded (PG_DEGRADED) 2024-04-14T14:25:03.817 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:25:03 smithi069 ceph-mon[83680]: pgmap v164: 161 pgs: 21 peering, 17 active+undersized+degraded, 30 active+undersized, 93 active+clean; 585 KiB data, 190 MiB used, 715 GiB / 715 GiB avail; 383 B/s rd, 0 op/s; 58/627 objects degraded (9.250%) 2024-04-14T14:25:03.817 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:25:03 smithi069 ceph-mon[83680]: osdmap e140: 8 total, 8 up, 8 in 2024-04-14T14:25:03.817 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:25:03 smithi069 ceph-mon[83680]: Health check update: Degraded data redundancy: 106/627 objects degraded (16.906%), 25 pgs degraded (PG_DEGRADED) 2024-04-14T14:25:03.818 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:25:03 smithi069 ceph-mon[88636]: pgmap v164: 161 pgs: 21 peering, 17 active+undersized+degraded, 30 active+undersized, 93 active+clean; 585 KiB data, 190 MiB used, 715 GiB / 715 GiB avail; 383 B/s rd, 0 op/s; 58/627 objects degraded (9.250%) 2024-04-14T14:25:03.818 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:25:03 smithi069 ceph-mon[88636]: osdmap e140: 8 total, 8 up, 8 in 2024-04-14T14:25:03.818 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:25:03 smithi069 ceph-mon[88636]: Health check update: Degraded data redundancy: 106/627 objects degraded (16.906%), 25 pgs degraded (PG_DEGRADED) 2024-04-14T14:25:06.067 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 14 14:25:05 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-y[63038]: 2024-04-14T14:25:05.576+0000 7fdd38467700 -1 mgr.server reply reply (16) Device or resource busy unsafe to stop osd(s) at this time (12 PGs are or would become offline) 2024-04-14T14:25:06.067 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:25:05 smithi069 ceph-mon[83680]: pgmap v166: 161 pgs: 32 peering, 12 active+undersized+degraded, 24 active+undersized, 93 active+clean; 585 KiB data, 190 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s; 40/627 objects degraded (6.380%) 2024-04-14T14:25:06.067 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:25:05 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:25:06.067 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:25:05 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:25:06.067 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:25:05 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T14:25:06.067 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:25:05 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-14T14:25:06.067 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:25:05 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:25:06.068 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:25:05 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-14T14:25:06.068 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:25:05 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T14:25:06.068 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:25:05 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T14:25:06.068 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:25:05 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T14:25:06.068 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:25:05 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["5"], "max": 16}]: dispatch 2024-04-14T14:25:06.068 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:25:05 smithi069 ceph-mon[88636]: pgmap v166: 161 pgs: 32 peering, 12 active+undersized+degraded, 24 active+undersized, 93 active+clean; 585 KiB data, 190 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s; 40/627 objects degraded (6.380%) 2024-04-14T14:25:06.068 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:25:05 smithi069 ceph-mon[88636]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:25:06.068 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:25:05 smithi069 ceph-mon[88636]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:25:06.068 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:25:05 smithi069 ceph-mon[88636]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T14:25:06.068 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:25:05 smithi069 ceph-mon[88636]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-14T14:25:06.068 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:25:05 smithi069 ceph-mon[88636]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:25:06.068 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:25:05 smithi069 ceph-mon[88636]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-14T14:25:06.068 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:25:05 smithi069 ceph-mon[88636]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T14:25:06.068 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:25:05 smithi069 ceph-mon[88636]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T14:25:06.068 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:25:05 smithi069 ceph-mon[88636]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T14:25:06.069 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:25:05 smithi069 ceph-mon[88636]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["5"], "max": 16}]: dispatch 2024-04-14T14:25:06.094 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:25:05 smithi122 ceph-mon[67998]: pgmap v166: 161 pgs: 32 peering, 12 active+undersized+degraded, 24 active+undersized, 93 active+clean; 585 KiB data, 190 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s; 40/627 objects degraded (6.380%) 2024-04-14T14:25:06.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:25:05 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:25:06.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:25:05 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:25:06.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:25:05 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T14:25:06.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:25:05 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-14T14:25:06.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:25:05 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:25:06.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:25:05 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-14T14:25:06.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:25:05 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T14:25:06.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:25:05 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T14:25:06.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:25:05 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T14:25:06.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:25:05 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["5"], "max": 16}]: dispatch 2024-04-14T14:25:06.816 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 14 14:25:06 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-y[63038]: ::ffff:172.21.15.122 - - [14/Apr/2024:14:25:06] "GET /metrics HTTP/1.1" 200 38267 "" "Prometheus/2.43.0" 2024-04-14T14:25:06.817 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:25:06 smithi069 ceph-mon[83680]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["5"], "max": 16}]: dispatch 2024-04-14T14:25:06.817 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:25:06 smithi069 ceph-mon[83680]: Upgrade: unsafe to stop osd(s) at this time (12 PGs are or would become offline) 2024-04-14T14:25:06.817 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:25:06 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:25:06.817 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:25:06 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T14:25:06.817 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:25:06 smithi069 ceph-mon[88636]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["5"], "max": 16}]: dispatch 2024-04-14T14:25:06.817 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:25:06 smithi069 ceph-mon[88636]: Upgrade: unsafe to stop osd(s) at this time (12 PGs are or would become offline) 2024-04-14T14:25:06.817 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:25:06 smithi069 ceph-mon[88636]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:25:06.817 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:25:06 smithi069 ceph-mon[88636]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T14:25:07.094 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:25:06 smithi122 ceph-mon[67998]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["5"], "max": 16}]: dispatch 2024-04-14T14:25:07.094 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:25:06 smithi122 ceph-mon[67998]: Upgrade: unsafe to stop osd(s) at this time (12 PGs are or would become offline) 2024-04-14T14:25:07.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:25:06 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:25:07.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:25:06 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T14:25:08.066 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:25:07 smithi069 ceph-mon[83680]: pgmap v167: 161 pgs: 32 peering, 1 active+undersized+degraded, 8 active+undersized, 120 active+clean; 585 KiB data, 190 MiB used, 715 GiB / 715 GiB avail; 831 B/s rd, 0 op/s; 7/627 objects degraded (1.116%) 2024-04-14T14:25:08.067 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:25:07 smithi069 ceph-mon[88636]: pgmap v167: 161 pgs: 32 peering, 1 active+undersized+degraded, 8 active+undersized, 120 active+clean; 585 KiB data, 190 MiB used, 715 GiB / 715 GiB avail; 831 B/s rd, 0 op/s; 7/627 objects degraded (1.116%) 2024-04-14T14:25:08.094 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:25:07 smithi122 ceph-mon[67998]: pgmap v167: 161 pgs: 32 peering, 1 active+undersized+degraded, 8 active+undersized, 120 active+clean; 585 KiB data, 190 MiB used, 715 GiB / 715 GiB avail; 831 B/s rd, 0 op/s; 7/627 objects degraded (1.116%) 2024-04-14T14:25:09.066 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:25:08 smithi069 ceph-mon[83680]: Health check update: Degraded data redundancy: 7/627 objects degraded (1.116%), 1 pg degraded (PG_DEGRADED) 2024-04-14T14:25:09.067 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:25:08 smithi069 ceph-mon[88636]: Health check update: Degraded data redundancy: 7/627 objects degraded (1.116%), 1 pg degraded (PG_DEGRADED) 2024-04-14T14:25:09.094 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:25:08 smithi122 ceph-mon[67998]: Health check update: Degraded data redundancy: 7/627 objects degraded (1.116%), 1 pg degraded (PG_DEGRADED) 2024-04-14T14:25:10.070 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:25:09 smithi069 ceph-mon[83680]: from='client.25417 -' entity='client.iscsi.foo.smithi069.yuumtr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T14:25:10.070 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:25:09 smithi069 ceph-mon[83680]: pgmap v168: 161 pgs: 161 active+clean; 585 KiB data, 190 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:25:10.070 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:25:09 smithi069 ceph-mon[83680]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 7/627 objects degraded (1.116%), 1 pg degraded) 2024-04-14T14:25:10.071 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:25:09 smithi069 ceph-mon[83680]: Cluster is now healthy 2024-04-14T14:25:10.071 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:25:09 smithi069 ceph-mon[88636]: from='client.25417 -' entity='client.iscsi.foo.smithi069.yuumtr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T14:25:10.071 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:25:09 smithi069 ceph-mon[88636]: pgmap v168: 161 pgs: 161 active+clean; 585 KiB data, 190 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:25:10.071 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:25:09 smithi069 ceph-mon[88636]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 7/627 objects degraded (1.116%), 1 pg degraded) 2024-04-14T14:25:10.071 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:25:09 smithi069 ceph-mon[88636]: Cluster is now healthy 2024-04-14T14:25:10.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:25:09 smithi122 ceph-mon[67998]: from='client.25417 -' entity='client.iscsi.foo.smithi069.yuumtr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T14:25:10.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:25:09 smithi122 ceph-mon[67998]: pgmap v168: 161 pgs: 161 active+clean; 585 KiB data, 190 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:25:10.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:25:09 smithi122 ceph-mon[67998]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 7/627 objects degraded (1.116%), 1 pg degraded) 2024-04-14T14:25:10.096 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:25:09 smithi122 ceph-mon[67998]: Cluster is now healthy 2024-04-14T14:25:12.066 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:25:11 smithi069 ceph-mon[83680]: pgmap v169: 161 pgs: 161 active+clean; 585 KiB data, 190 MiB used, 715 GiB / 715 GiB avail; 589 B/s rd, 0 op/s 2024-04-14T14:25:12.067 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:25:11 smithi069 ceph-mon[88636]: pgmap v169: 161 pgs: 161 active+clean; 585 KiB data, 190 MiB used, 715 GiB / 715 GiB avail; 589 B/s rd, 0 op/s 2024-04-14T14:25:12.094 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:25:11 smithi122 ceph-mon[67998]: pgmap v169: 161 pgs: 161 active+clean; 585 KiB data, 190 MiB used, 715 GiB / 715 GiB avail; 589 B/s rd, 0 op/s 2024-04-14T14:25:14.066 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:25:13 smithi069 ceph-mon[83680]: pgmap v170: 161 pgs: 161 active+clean; 585 KiB data, 190 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-14T14:25:14.067 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:25:13 smithi069 ceph-mon[88636]: pgmap v170: 161 pgs: 161 active+clean; 585 KiB data, 190 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-14T14:25:14.094 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:25:13 smithi122 ceph-mon[67998]: pgmap v170: 161 pgs: 161 active+clean; 585 KiB data, 190 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-14T14:25:15.296 INFO:teuthology.orchestra.run.smithi069.stdout:true 2024-04-14T14:25:15.947 INFO:teuthology.orchestra.run.smithi069.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-14T14:25:15.947 INFO:teuthology.orchestra.run.smithi069.stdout:alertmanager.a smithi069 *:9093,9094 running (7m) 49s ago 28m 26.5M - 0.25.0 c8568f914cd2 7524f106e7bd 2024-04-14T14:25:15.947 INFO:teuthology.orchestra.run.smithi069.stdout:grafana.a smithi122 *:3000 running (6m) 15s ago 28m 82.9M - 9.4.12 2bacad6d85d8 5685178e3eba 2024-04-14T14:25:15.947 INFO:teuthology.orchestra.run.smithi069.stdout:iscsi.foo.smithi069.yuumtr smithi069 running (8m) 49s ago 27m 62.9M - 3.5 e1d6a67b021e 38cc81ea8cfa 2024-04-14T14:25:15.947 INFO:teuthology.orchestra.run.smithi069.stdout:mgr.x smithi122 *:8443,9283,8765 running (7m) 15s ago 33m 454M - 19.0.0-2778-ga971fd5f b96c509eef84 e29977590b43 2024-04-14T14:25:15.947 INFO:teuthology.orchestra.run.smithi069.stdout:mgr.y smithi069 *:8443,9283,8765 running (17m) 49s ago 34m 507M - 19.0.0-2778-ga971fd5f b96c509eef84 7af3d29d0e5f 2024-04-14T14:25:15.947 INFO:teuthology.orchestra.run.smithi069.stdout:mon.a smithi069 running (4m) 49s ago 34m 54.7M 2048M 19.0.0-2778-ga971fd5f b96c509eef84 8bcd6b3d3846 2024-04-14T14:25:15.948 INFO:teuthology.orchestra.run.smithi069.stdout:mon.b smithi122 running (5m) 15s ago 33m 45.3M 2048M 19.0.0-2778-ga971fd5f b96c509eef84 05f437a1a9a1 2024-04-14T14:25:15.948 INFO:teuthology.orchestra.run.smithi069.stdout:mon.c smithi069 running (4m) 49s ago 33m 39.3M 2048M 19.0.0-2778-ga971fd5f b96c509eef84 84aa95f11d0c 2024-04-14T14:25:15.948 INFO:teuthology.orchestra.run.smithi069.stdout:node-exporter.a smithi069 *:9100 running (7m) 49s ago 29m 15.0M - 1.5.0 0da6a335fe13 0de0eb2171b9 2024-04-14T14:25:15.948 INFO:teuthology.orchestra.run.smithi069.stdout:node-exporter.b smithi122 *:9100 running (7m) 15s ago 29m 16.0M - 1.5.0 0da6a335fe13 67877bf02d8e 2024-04-14T14:25:15.948 INFO:teuthology.orchestra.run.smithi069.stdout:osd.0 smithi069 running (3m) 49s ago 32m 53.9M 2992M 19.0.0-2778-ga971fd5f b96c509eef84 a0e57e3fb494 2024-04-14T14:25:15.948 INFO:teuthology.orchestra.run.smithi069.stdout:osd.1 smithi069 running (2m) 49s ago 32m 52.9M 2992M 19.0.0-2778-ga971fd5f b96c509eef84 cf73d17919bb 2024-04-14T14:25:15.948 INFO:teuthology.orchestra.run.smithi069.stdout:osd.2 smithi069 running (106s) 49s ago 32m 41.9M 2992M 19.0.0-2778-ga971fd5f b96c509eef84 c0efba5ef451 2024-04-14T14:25:15.948 INFO:teuthology.orchestra.run.smithi069.stdout:osd.3 smithi069 running (54s) 49s ago 31m 15.1M 2992M 19.0.0-2778-ga971fd5f b96c509eef84 d13460da0a65 2024-04-14T14:25:15.948 INFO:teuthology.orchestra.run.smithi069.stdout:osd.4 smithi122 running (20s) 15s ago 31m 12.0M 3248M 19.0.0-2778-ga971fd5f b96c509eef84 7a87a024f984 2024-04-14T14:25:15.948 INFO:teuthology.orchestra.run.smithi069.stdout:osd.5 smithi122 running (31m) 15s ago 30m 59.2M 3248M 17.2.0 e1d6a67b021e 5f52d217ceb8 2024-04-14T14:25:15.948 INFO:teuthology.orchestra.run.smithi069.stdout:osd.6 smithi122 running (30m) 15s ago 30m 55.6M 3248M 17.2.0 e1d6a67b021e a2acfec61475 2024-04-14T14:25:15.948 INFO:teuthology.orchestra.run.smithi069.stdout:osd.7 smithi122 running (30m) 15s ago 30m 61.5M 3248M 17.2.0 e1d6a67b021e 62411cd06801 2024-04-14T14:25:15.948 INFO:teuthology.orchestra.run.smithi069.stdout:prometheus.a smithi122 *:9095 running (7m) 15s ago 29m 69.0M - 2.43.0 a07b618ecd1d 5c1f41d3bbe6 2024-04-14T14:25:15.948 INFO:teuthology.orchestra.run.smithi069.stdout:rgw.foo.smithi069.wesedz smithi069 *:8000 running (27m) 49s ago 27m 79.7M - 17.2.0 e1d6a67b021e 16de491440f3 2024-04-14T14:25:15.948 INFO:teuthology.orchestra.run.smithi069.stdout:rgw.foo.smithi122.jiobyv smithi122 *:8000 running (27m) 15s ago 27m 77.2M - 17.2.0 e1d6a67b021e 6d16de445dba 2024-04-14T14:25:15.949 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:25:15 smithi069 ceph-mon[83680]: pgmap v171: 161 pgs: 161 active+clean; 585 KiB data, 190 MiB used, 715 GiB / 715 GiB avail; 876 B/s rd, 0 op/s 2024-04-14T14:25:15.949 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:25:15 smithi069 ceph-mon[88636]: pgmap v171: 161 pgs: 161 active+clean; 585 KiB data, 190 MiB used, 715 GiB / 715 GiB avail; 876 B/s rd, 0 op/s 2024-04-14T14:25:16.094 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:25:15 smithi122 ceph-mon[67998]: pgmap v171: 161 pgs: 161 active+clean; 585 KiB data, 190 MiB used, 715 GiB / 715 GiB avail; 876 B/s rd, 0 op/s 2024-04-14T14:25:16.347 INFO:teuthology.orchestra.run.smithi069.stdout:{ 2024-04-14T14:25:16.347 INFO:teuthology.orchestra.run.smithi069.stdout: "mon": { 2024-04-14T14:25:16.347 INFO:teuthology.orchestra.run.smithi069.stdout: "ceph version 19.0.0-2778-ga971fd5f (a971fd5f5719038c70b817bafacd6d5400b1d888) squid (dev)": 3 2024-04-14T14:25:16.347 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-14T14:25:16.347 INFO:teuthology.orchestra.run.smithi069.stdout: "mgr": { 2024-04-14T14:25:16.347 INFO:teuthology.orchestra.run.smithi069.stdout: "ceph version 19.0.0-2778-ga971fd5f (a971fd5f5719038c70b817bafacd6d5400b1d888) squid (dev)": 2 2024-04-14T14:25:16.347 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-14T14:25:16.348 INFO:teuthology.orchestra.run.smithi069.stdout: "osd": { 2024-04-14T14:25:16.348 INFO:teuthology.orchestra.run.smithi069.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 3, 2024-04-14T14:25:16.348 INFO:teuthology.orchestra.run.smithi069.stdout: "ceph version 19.0.0-2778-ga971fd5f (a971fd5f5719038c70b817bafacd6d5400b1d888) squid (dev)": 5 2024-04-14T14:25:16.348 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-14T14:25:16.348 INFO:teuthology.orchestra.run.smithi069.stdout: "rgw": { 2024-04-14T14:25:16.348 INFO:teuthology.orchestra.run.smithi069.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-04-14T14:25:16.348 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-14T14:25:16.348 INFO:teuthology.orchestra.run.smithi069.stdout: "overall": { 2024-04-14T14:25:16.348 INFO:teuthology.orchestra.run.smithi069.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 5, 2024-04-14T14:25:16.348 INFO:teuthology.orchestra.run.smithi069.stdout: "ceph version 19.0.0-2778-ga971fd5f (a971fd5f5719038c70b817bafacd6d5400b1d888) squid (dev)": 10 2024-04-14T14:25:16.348 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-14T14:25:16.348 INFO:teuthology.orchestra.run.smithi069.stdout:} 2024-04-14T14:25:16.658 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 14 14:25:16 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-y[63038]: ::ffff:172.21.15.122 - - [14/Apr/2024:14:25:16] "GET /metrics HTTP/1.1" 200 38273 "" "Prometheus/2.43.0" 2024-04-14T14:25:16.684 INFO:teuthology.orchestra.run.smithi069.stdout:{ 2024-04-14T14:25:16.684 INFO:teuthology.orchestra.run.smithi069.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888", 2024-04-14T14:25:16.684 INFO:teuthology.orchestra.run.smithi069.stdout: "in_progress": true, 2024-04-14T14:25:16.684 INFO:teuthology.orchestra.run.smithi069.stdout: "which": "Upgrading daemons of type(s) crash,osd", 2024-04-14T14:25:16.684 INFO:teuthology.orchestra.run.smithi069.stdout: "services_complete": [], 2024-04-14T14:25:16.684 INFO:teuthology.orchestra.run.smithi069.stdout: "progress": "5/8 daemons upgraded", 2024-04-14T14:25:16.685 INFO:teuthology.orchestra.run.smithi069.stdout: "message": "Currently upgrading osd daemons", 2024-04-14T14:25:16.685 INFO:teuthology.orchestra.run.smithi069.stdout: "is_paused": false 2024-04-14T14:25:16.685 INFO:teuthology.orchestra.run.smithi069.stdout:} 2024-04-14T14:25:17.067 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:25:16 smithi069 ceph-mon[88636]: from='client.44475 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T14:25:17.067 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:25:16 smithi069 ceph-mon[88636]: from='client.44481 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T14:25:17.067 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:25:16 smithi069 ceph-mon[88636]: from='client.? 172.21.15.69:0/835519729' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T14:25:17.067 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:25:16 smithi069 ceph-mon[83680]: from='client.44475 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T14:25:17.067 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:25:16 smithi069 ceph-mon[83680]: from='client.44481 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T14:25:17.068 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:25:16 smithi069 ceph-mon[83680]: from='client.? 172.21.15.69:0/835519729' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T14:25:17.094 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:25:16 smithi122 ceph-mon[67998]: from='client.44475 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T14:25:17.094 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:25:16 smithi122 ceph-mon[67998]: from='client.44481 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T14:25:17.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:25:16 smithi122 ceph-mon[67998]: from='client.? 172.21.15.69:0/835519729' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T14:25:18.066 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:25:17 smithi069 ceph-mon[83680]: from='client.54424 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T14:25:18.067 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:25:17 smithi069 ceph-mon[83680]: pgmap v172: 161 pgs: 161 active+clean; 585 KiB data, 190 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:25:18.067 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:25:17 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:25:18.067 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:25:17 smithi069 ceph-mon[88636]: from='client.54424 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T14:25:18.067 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:25:17 smithi069 ceph-mon[88636]: pgmap v172: 161 pgs: 161 active+clean; 585 KiB data, 190 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:25:18.067 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:25:17 smithi069 ceph-mon[88636]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:25:18.094 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:25:17 smithi122 ceph-mon[67998]: from='client.54424 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T14:25:18.094 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:25:17 smithi122 ceph-mon[67998]: pgmap v172: 161 pgs: 161 active+clean; 585 KiB data, 190 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:25:18.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:25:17 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:25:18.968 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:25:18 smithi069 ceph-mon[83680]: from='client.54436 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T14:25:18.968 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:25:18 smithi069 ceph-mon[88636]: from='client.54436 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T14:25:19.094 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:25:18 smithi122 ceph-mon[67998]: from='client.54436 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T14:25:20.066 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:25:19 smithi069 ceph-mon[88636]: from='client.25417 -' entity='client.iscsi.foo.smithi069.yuumtr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T14:25:20.067 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:25:19 smithi069 ceph-mon[88636]: pgmap v173: 161 pgs: 161 active+clean; 585 KiB data, 190 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:25:20.067 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:25:19 smithi069 ceph-mon[83680]: from='client.25417 -' entity='client.iscsi.foo.smithi069.yuumtr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T14:25:20.067 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:25:19 smithi069 ceph-mon[83680]: pgmap v173: 161 pgs: 161 active+clean; 585 KiB data, 190 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:25:20.094 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:25:19 smithi122 ceph-mon[67998]: from='client.25417 -' entity='client.iscsi.foo.smithi069.yuumtr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T14:25:20.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:25:19 smithi122 ceph-mon[67998]: pgmap v173: 161 pgs: 161 active+clean; 585 KiB data, 190 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:25:21.066 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:25:20 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["5"], "max": 16}]: dispatch 2024-04-14T14:25:21.067 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:25:20 smithi069 ceph-mon[88636]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["5"], "max": 16}]: dispatch 2024-04-14T14:25:21.094 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:25:20 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["5"], "max": 16}]: dispatch 2024-04-14T14:25:21.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:25:21 smithi122 ceph-mon[67998]: pgmap v174: 161 pgs: 161 active+clean; 585 KiB data, 190 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:25:21.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:25:21 smithi122 ceph-mon[67998]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["5"], "max": 16}]: dispatch 2024-04-14T14:25:21.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:25:21 smithi122 ceph-mon[67998]: Upgrade: osd.5 is safe to restart 2024-04-14T14:25:21.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:25:21 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:25:21.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:25:21 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T14:25:22.066 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:25:21 smithi069 ceph-mon[83680]: pgmap v174: 161 pgs: 161 active+clean; 585 KiB data, 190 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:25:22.067 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:25:21 smithi069 ceph-mon[83680]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["5"], "max": 16}]: dispatch 2024-04-14T14:25:22.067 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:25:21 smithi069 ceph-mon[83680]: Upgrade: osd.5 is safe to restart 2024-04-14T14:25:22.067 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:25:21 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:25:22.067 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:25:21 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T14:25:22.067 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:25:21 smithi069 ceph-mon[88636]: pgmap v174: 161 pgs: 161 active+clean; 585 KiB data, 190 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:25:22.067 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:25:21 smithi069 ceph-mon[88636]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["5"], "max": 16}]: dispatch 2024-04-14T14:25:22.067 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:25:21 smithi069 ceph-mon[88636]: Upgrade: osd.5 is safe to restart 2024-04-14T14:25:22.067 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:25:21 smithi069 ceph-mon[88636]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:25:22.067 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:25:21 smithi069 ceph-mon[88636]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T14:25:23.344 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:25:23 smithi122 ceph-mon[67998]: Upgrade: Updating osd.5 2024-04-14T14:25:23.344 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:25:23 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:25:23.344 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:25:23 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-04-14T14:25:23.345 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:25:23 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T14:25:23.345 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:25:23 smithi122 ceph-mon[67998]: Deploying daemon osd.5 on smithi122 2024-04-14T14:25:23.345 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:25:23 smithi122 ceph-mon[67998]: pgmap v175: 161 pgs: 161 active+clean; 585 KiB data, 190 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:25:23.566 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:25:23 smithi069 ceph-mon[88636]: Upgrade: Updating osd.5 2024-04-14T14:25:23.567 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:25:23 smithi069 ceph-mon[88636]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:25:23.567 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:25:23 smithi069 ceph-mon[88636]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-04-14T14:25:23.567 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:25:23 smithi069 ceph-mon[88636]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T14:25:23.567 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:25:23 smithi069 ceph-mon[88636]: Deploying daemon osd.5 on smithi122 2024-04-14T14:25:23.567 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:25:23 smithi069 ceph-mon[88636]: pgmap v175: 161 pgs: 161 active+clean; 585 KiB data, 190 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:25:23.567 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:25:23 smithi069 ceph-mon[83680]: Upgrade: Updating osd.5 2024-04-14T14:25:23.567 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:25:23 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:25:23.567 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:25:23 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-04-14T14:25:23.567 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:25:23 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T14:25:23.567 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:25:23 smithi069 ceph-mon[83680]: Deploying daemon osd.5 on smithi122 2024-04-14T14:25:23.568 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:25:23 smithi069 ceph-mon[83680]: pgmap v175: 161 pgs: 161 active+clean; 585 KiB data, 190 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:25:25.554 INFO:journalctl@ceph.osd.5.smithi122.stdout:Apr 14 14:25:25 smithi122 systemd[1]: Stopping Ceph osd.5 for dc326fc2-fa65-11ee-bc8e-c7b262605968... 2024-04-14T14:25:25.844 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:25:25 smithi122 ceph-mon[67998]: pgmap v176: 161 pgs: 161 active+clean; 585 KiB data, 190 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:25:25.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:25:25 smithi122 ceph-mon[67998]: osd.5 marked itself down and dead 2024-04-14T14:25:25.845 INFO:journalctl@ceph.osd.5.smithi122.stdout:Apr 14 14:25:25 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-osd-5[32056]: 2024-04-14T14:25:25.566+0000 7fa759761700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.5 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-14T14:25:25.845 INFO:journalctl@ceph.osd.5.smithi122.stdout:Apr 14 14:25:25 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-osd-5[32056]: 2024-04-14T14:25:25.566+0000 7fa759761700 -1 osd.5 140 *** Got signal Terminated *** 2024-04-14T14:25:25.845 INFO:journalctl@ceph.osd.5.smithi122.stdout:Apr 14 14:25:25 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-osd-5[32056]: 2024-04-14T14:25:25.566+0000 7fa759761700 -1 osd.5 140 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-14T14:25:26.066 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:25:25 smithi069 ceph-mon[83680]: pgmap v176: 161 pgs: 161 active+clean; 585 KiB data, 190 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:25:26.067 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:25:25 smithi069 ceph-mon[83680]: osd.5 marked itself down and dead 2024-04-14T14:25:26.067 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:25:25 smithi069 ceph-mon[88636]: pgmap v176: 161 pgs: 161 active+clean; 585 KiB data, 190 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:25:26.067 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:25:25 smithi069 ceph-mon[88636]: osd.5 marked itself down and dead 2024-04-14T14:25:26.453 INFO:journalctl@ceph.osd.5.smithi122.stdout:Apr 14 14:25:26 smithi122 podman[82277]: 2024-04-14 14:25:26.169605528 +0000 UTC m=+0.719612665 container died 5f52d217ceb8a29271ba8b1caa93cc5cb08f0bde43c2effed7392c9c5b9ba24b (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-osd-5, CEPH_POINT_RELEASE=-17.2.0, distribution-scope=public, maintainer=Guillaume Abrioux , ceph=True, version=8, release=754, vcs-type=git, RELEASE=HEAD, architecture=x86_64, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, io.buildah.version=1.19.8, vendor=Red Hat, Inc., GIT_CLEAN=True, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, io.openshift.tags=base centos centos-stream, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, io.openshift.expose-services=, GIT_BRANCH=HEAD, com.redhat.component=centos-stream-container, name=centos-stream, build-date=2022-05-03T08:36:31.336870, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, GIT_REPO=https://github.com/ceph/ceph-container.git, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., io.k8s.display-name=CentOS Stream 8) 2024-04-14T14:25:26.817 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:25:26 smithi069 ceph-mon[83680]: Health check failed: 1 osds down (OSD_DOWN) 2024-04-14T14:25:26.817 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:25:26 smithi069 ceph-mon[83680]: osdmap e141: 8 total, 7 up, 8 in 2024-04-14T14:25:26.817 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 14 14:25:26 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-y[63038]: ::ffff:172.21.15.122 - - [14/Apr/2024:14:25:26] "GET /metrics HTTP/1.1" 200 38273 "" "Prometheus/2.43.0" 2024-04-14T14:25:26.817 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:25:26 smithi069 ceph-mon[88636]: Health check failed: 1 osds down (OSD_DOWN) 2024-04-14T14:25:26.817 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:25:26 smithi069 ceph-mon[88636]: osdmap e141: 8 total, 7 up, 8 in 2024-04-14T14:25:26.822 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:25:26 smithi122 ceph-mon[67998]: Health check failed: 1 osds down (OSD_DOWN) 2024-04-14T14:25:26.822 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:25:26 smithi122 ceph-mon[67998]: osdmap e141: 8 total, 7 up, 8 in 2024-04-14T14:25:26.822 INFO:journalctl@ceph.osd.5.smithi122.stdout:Apr 14 14:25:26 smithi122 podman[82277]: 2024-04-14 14:25:26.5509011 +0000 UTC m=+1.100908238 container cleanup 5f52d217ceb8a29271ba8b1caa93cc5cb08f0bde43c2effed7392c9c5b9ba24b (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-osd-5, architecture=x86_64, io.openshift.expose-services=, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, io.buildah.version=1.19.8, GIT_BRANCH=HEAD, name=centos-stream, io.openshift.tags=base centos centos-stream, distribution-scope=public, build-date=2022-05-03T08:36:31.336870, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, RELEASE=HEAD, ceph=True, io.k8s.display-name=CentOS Stream 8, CEPH_POINT_RELEASE=-17.2.0, release=754, vendor=Red Hat, Inc., com.redhat.component=centos-stream-container, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, maintainer=Guillaume Abrioux , com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., GIT_CLEAN=True, vcs-type=git, version=8, GIT_REPO=https://github.com/ceph/ceph-container.git, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image) 2024-04-14T14:25:26.822 INFO:journalctl@ceph.osd.5.smithi122.stdout:Apr 14 14:25:26 smithi122 bash[82277]: ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-osd-5 2024-04-14T14:25:27.094 INFO:journalctl@ceph.osd.5.smithi122.stdout:Apr 14 14:25:26 smithi122 podman[82294]: 2024-04-14 14:25:26.821810112 +0000 UTC m=+0.648422604 container remove 5f52d217ceb8a29271ba8b1caa93cc5cb08f0bde43c2effed7392c9c5b9ba24b (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-osd-5, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, GIT_CLEAN=True, name=centos-stream, ceph=True, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, distribution-scope=public, RELEASE=HEAD, io.openshift.expose-services=, maintainer=Guillaume Abrioux , io.openshift.tags=base centos centos-stream, GIT_BRANCH=HEAD, com.redhat.component=centos-stream-container, GIT_REPO=https://github.com/ceph/ceph-container.git, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, version=8, build-date=2022-05-03T08:36:31.336870, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, io.k8s.display-name=CentOS Stream 8, vendor=Red Hat, Inc., com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, architecture=x86_64, io.buildah.version=1.19.8, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, release=754, CEPH_POINT_RELEASE=-17.2.0, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., vcs-type=git) 2024-04-14T14:25:27.484 INFO:journalctl@ceph.osd.5.smithi122.stdout:Apr 14 14:25:27 smithi122 podman[82386]: 2024-04-14 14:25:27.087616571 +0000 UTC m=+0.018039123 image pull b96c509eef841460b3271f5fdd331860da87b004524f4ed59414aa3680183b53 quay.ceph.io/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888 2024-04-14T14:25:27.485 INFO:journalctl@ceph.osd.5.smithi122.stdout:Apr 14 14:25:27 smithi122 podman[82386]: 2024-04-14 14:25:27.231355109 +0000 UTC m=+0.161777660 container create afa5a02d160f6fc36ab11ca7dab9b3a92412abc50837a9a1f09bd83f3fd7ca8c (image=quay.ceph.io/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888, name=ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-osd-5-deactivate, GIT_CLEAN=True, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, maintainer=Guillaume Abrioux , RELEASE=main-a971fd5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240403, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.license=GPLv2) 2024-04-14T14:25:27.779 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:25:27 smithi122 ceph-mon[67998]: pgmap v178: 161 pgs: 7 peering, 16 stale+active+clean, 138 active+clean; 585 KiB data, 190 MiB used, 715 GiB / 715 GiB avail; 204 B/s rd, 0 op/s 2024-04-14T14:25:27.779 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:25:27 smithi122 ceph-mon[67998]: Health check failed: Reduced data availability: 4 pgs peering (PG_AVAILABILITY) 2024-04-14T14:25:27.779 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:25:27 smithi122 ceph-mon[67998]: osdmap e142: 8 total, 7 up, 8 in 2024-04-14T14:25:27.779 INFO:journalctl@ceph.osd.5.smithi122.stdout:Apr 14 14:25:27 smithi122 podman[82386]: 2024-04-14 14:25:27.484236838 +0000 UTC m=+0.414659390 container init afa5a02d160f6fc36ab11ca7dab9b3a92412abc50837a9a1f09bd83f3fd7ca8c (image=quay.ceph.io/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888, name=ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-osd-5-deactivate, GIT_CLEAN=True, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=main-a971fd5, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, io.buildah.version=1.33.5, org.label-schema.build-date=20240403, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, ceph=True) 2024-04-14T14:25:27.780 INFO:journalctl@ceph.osd.5.smithi122.stdout:Apr 14 14:25:27 smithi122 podman[82386]: 2024-04-14 14:25:27.487427198 +0000 UTC m=+0.417849753 container start afa5a02d160f6fc36ab11ca7dab9b3a92412abc50837a9a1f09bd83f3fd7ca8c (image=quay.ceph.io/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888, name=ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-osd-5-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, RELEASE=main-a971fd5, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240403, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, ceph=True, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0) 2024-04-14T14:25:27.780 INFO:journalctl@ceph.osd.5.smithi122.stdout:Apr 14 14:25:27 smithi122 podman[82386]: 2024-04-14 14:25:27.584360711 +0000 UTC m=+0.514783275 container attach afa5a02d160f6fc36ab11ca7dab9b3a92412abc50837a9a1f09bd83f3fd7ca8c (image=quay.ceph.io/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888, name=ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-osd-5-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240403, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, ceph=True, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, RELEASE=main-a971fd5) 2024-04-14T14:25:28.066 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:25:27 smithi069 ceph-mon[83680]: pgmap v178: 161 pgs: 7 peering, 16 stale+active+clean, 138 active+clean; 585 KiB data, 190 MiB used, 715 GiB / 715 GiB avail; 204 B/s rd, 0 op/s 2024-04-14T14:25:28.067 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:25:27 smithi069 ceph-mon[83680]: Health check failed: Reduced data availability: 4 pgs peering (PG_AVAILABILITY) 2024-04-14T14:25:28.067 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:25:27 smithi069 ceph-mon[83680]: osdmap e142: 8 total, 7 up, 8 in 2024-04-14T14:25:28.067 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:25:27 smithi069 ceph-mon[88636]: pgmap v178: 161 pgs: 7 peering, 16 stale+active+clean, 138 active+clean; 585 KiB data, 190 MiB used, 715 GiB / 715 GiB avail; 204 B/s rd, 0 op/s 2024-04-14T14:25:28.067 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:25:27 smithi069 ceph-mon[88636]: Health check failed: Reduced data availability: 4 pgs peering (PG_AVAILABILITY) 2024-04-14T14:25:28.067 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:25:27 smithi069 ceph-mon[88636]: osdmap e142: 8 total, 7 up, 8 in 2024-04-14T14:25:28.072 INFO:journalctl@ceph.osd.5.smithi122.stdout:Apr 14 14:25:27 smithi122 podman[82386]: 2024-04-14 14:25:27.778410791 +0000 UTC m=+0.708833351 container died afa5a02d160f6fc36ab11ca7dab9b3a92412abc50837a9a1f09bd83f3fd7ca8c (image=quay.ceph.io/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888, name=ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-osd-5-deactivate, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_CLEAN=True, RELEASE=main-a971fd5, ceph=True, io.buildah.version=1.33.5, org.label-schema.build-date=20240403, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0) 2024-04-14T14:25:28.595 INFO:journalctl@ceph.osd.5.smithi122.stdout:Apr 14 14:25:28 smithi122 podman[82412]: 2024-04-14 14:25:28.330672397 +0000 UTC m=+0.542275137 container remove afa5a02d160f6fc36ab11ca7dab9b3a92412abc50837a9a1f09bd83f3fd7ca8c (image=quay.ceph.io/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888, name=ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-osd-5-deactivate, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240403, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, RELEASE=main-a971fd5, GIT_BRANCH=HEAD, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, GIT_CLEAN=True, org.label-schema.vendor=CentOS) 2024-04-14T14:25:28.595 INFO:journalctl@ceph.osd.5.smithi122.stdout:Apr 14 14:25:28 smithi122 systemd[1]: ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968@osd.5.service: Deactivated successfully. 2024-04-14T14:25:28.595 INFO:journalctl@ceph.osd.5.smithi122.stdout:Apr 14 14:25:28 smithi122 systemd[1]: Stopped Ceph osd.5 for dc326fc2-fa65-11ee-bc8e-c7b262605968. 2024-04-14T14:25:28.595 INFO:journalctl@ceph.osd.5.smithi122.stdout:Apr 14 14:25:28 smithi122 systemd[1]: ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968@osd.5.service: Consumed 16.543s CPU time. 2024-04-14T14:25:28.595 INFO:journalctl@ceph.osd.5.smithi122.stdout:Apr 14 14:25:28 smithi122 systemd[1]: Starting Ceph osd.5 for dc326fc2-fa65-11ee-bc8e-c7b262605968... 2024-04-14T14:25:28.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:25:28 smithi122 ceph-mon[67998]: from='client.25417 -' entity='client.iscsi.foo.smithi069.yuumtr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T14:25:28.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:25:28 smithi122 ceph-mon[67998]: pgmap v180: 161 pgs: 13 active+undersized+degraded, 7 peering, 4 stale+active+clean, 29 active+undersized, 108 active+clean; 585 KiB data, 190 MiB used, 715 GiB / 715 GiB avail; 39/627 objects degraded (6.220%) 2024-04-14T14:25:28.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:25:28 smithi122 ceph-mon[67998]: Health check failed: Degraded data redundancy: 39/627 objects degraded (6.220%), 13 pgs degraded (PG_DEGRADED) 2024-04-14T14:25:28.977 INFO:journalctl@ceph.osd.5.smithi122.stdout:Apr 14 14:25:28 smithi122 podman[82513]: 2024-04-14 14:25:28.66704448 +0000 UTC m=+0.018779123 image pull b96c509eef841460b3271f5fdd331860da87b004524f4ed59414aa3680183b53 quay.ceph.io/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888 2024-04-14T14:25:28.977 INFO:journalctl@ceph.osd.5.smithi122.stdout:Apr 14 14:25:28 smithi122 podman[82513]: 2024-04-14 14:25:28.773403815 +0000 UTC m=+0.125138423 container create 7b5ddec5d9067def6561dc11661015b0955024648674efd52d35c7826841b34d (image=quay.ceph.io/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888, name=ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-osd-5-activate, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240403, ceph=True, maintainer=Guillaume Abrioux , GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, RELEASE=main-a971fd5, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-14T14:25:28.977 INFO:journalctl@ceph.osd.5.smithi122.stdout:Apr 14 14:25:28 smithi122 podman[82513]: 2024-04-14 14:25:28.976077299 +0000 UTC m=+0.327811910 container init 7b5ddec5d9067def6561dc11661015b0955024648674efd52d35c7826841b34d (image=quay.ceph.io/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888, name=ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-osd-5-activate, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.build-date=20240403, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.schema-version=1.0, RELEASE=main-a971fd5, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_CLEAN=True) 2024-04-14T14:25:29.066 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:25:28 smithi069 ceph-mon[88636]: from='client.25417 -' entity='client.iscsi.foo.smithi069.yuumtr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T14:25:29.067 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:25:28 smithi069 ceph-mon[88636]: pgmap v180: 161 pgs: 13 active+undersized+degraded, 7 peering, 4 stale+active+clean, 29 active+undersized, 108 active+clean; 585 KiB data, 190 MiB used, 715 GiB / 715 GiB avail; 39/627 objects degraded (6.220%) 2024-04-14T14:25:29.067 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:25:28 smithi069 ceph-mon[88636]: Health check failed: Degraded data redundancy: 39/627 objects degraded (6.220%), 13 pgs degraded (PG_DEGRADED) 2024-04-14T14:25:29.067 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:25:28 smithi069 ceph-mon[83680]: from='client.25417 -' entity='client.iscsi.foo.smithi069.yuumtr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T14:25:29.067 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:25:28 smithi069 ceph-mon[83680]: pgmap v180: 161 pgs: 13 active+undersized+degraded, 7 peering, 4 stale+active+clean, 29 active+undersized, 108 active+clean; 585 KiB data, 190 MiB used, 715 GiB / 715 GiB avail; 39/627 objects degraded (6.220%) 2024-04-14T14:25:29.067 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:25:28 smithi069 ceph-mon[83680]: Health check failed: Degraded data redundancy: 39/627 objects degraded (6.220%), 13 pgs degraded (PG_DEGRADED) 2024-04-14T14:25:29.344 INFO:journalctl@ceph.osd.5.smithi122.stdout:Apr 14 14:25:28 smithi122 podman[82513]: 2024-04-14 14:25:28.979359601 +0000 UTC m=+0.331094215 container start 7b5ddec5d9067def6561dc11661015b0955024648674efd52d35c7826841b34d (image=quay.ceph.io/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888, name=ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-osd-5-activate, org.label-schema.license=GPLv2, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.build-date=20240403, org.label-schema.schema-version=1.0, ceph=True, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, GIT_CLEAN=True, RELEASE=main-a971fd5) 2024-04-14T14:25:29.344 INFO:journalctl@ceph.osd.5.smithi122.stdout:Apr 14 14:25:29 smithi122 podman[82513]: 2024-04-14 14:25:29.059505326 +0000 UTC m=+0.411239949 container attach 7b5ddec5d9067def6561dc11661015b0955024648674efd52d35c7826841b34d (image=quay.ceph.io/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888, name=ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-osd-5-activate, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=main-a971fd5, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, GIT_BRANCH=HEAD, org.label-schema.build-date=20240403, ceph=True) 2024-04-14T14:25:29.755 INFO:journalctl@ceph.osd.5.smithi122.stdout:Apr 14 14:25:29 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-osd-5-activate[82528]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2024-04-14T14:25:29.755 INFO:journalctl@ceph.osd.5.smithi122.stdout:Apr 14 14:25:29 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-osd-5-activate[82528]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-5 --no-mon-config --dev /dev/mapper/vg_nvme-lv_3 2024-04-14T14:25:29.756 INFO:journalctl@ceph.osd.5.smithi122.stdout:Apr 14 14:25:29 smithi122 bash[82513]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2024-04-14T14:25:29.756 INFO:journalctl@ceph.osd.5.smithi122.stdout:Apr 14 14:25:29 smithi122 bash[82513]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-5 --no-mon-config --dev /dev/mapper/vg_nvme-lv_3 2024-04-14T14:25:29.756 INFO:journalctl@ceph.osd.5.smithi122.stdout:Apr 14 14:25:29 smithi122 bash[82513]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_3 2024-04-14T14:25:29.756 INFO:journalctl@ceph.osd.5.smithi122.stdout:Apr 14 14:25:29 smithi122 bash[82513]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-2 2024-04-14T14:25:29.756 INFO:journalctl@ceph.osd.5.smithi122.stdout:Apr 14 14:25:29 smithi122 bash[82513]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-5/block 2024-04-14T14:25:29.756 INFO:journalctl@ceph.osd.5.smithi122.stdout:Apr 14 14:25:29 smithi122 bash[82513]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2024-04-14T14:25:29.756 INFO:journalctl@ceph.osd.5.smithi122.stdout:Apr 14 14:25:29 smithi122 bash[82513]: --> ceph-volume raw activate successful for osd ID: 5 2024-04-14T14:25:29.756 INFO:journalctl@ceph.osd.5.smithi122.stdout:Apr 14 14:25:29 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-osd-5-activate[82528]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_3 2024-04-14T14:25:29.756 INFO:journalctl@ceph.osd.5.smithi122.stdout:Apr 14 14:25:29 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-osd-5-activate[82528]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-2 2024-04-14T14:25:29.756 INFO:journalctl@ceph.osd.5.smithi122.stdout:Apr 14 14:25:29 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-osd-5-activate[82528]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-5/block 2024-04-14T14:25:29.756 INFO:journalctl@ceph.osd.5.smithi122.stdout:Apr 14 14:25:29 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-osd-5-activate[82528]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2024-04-14T14:25:29.756 INFO:journalctl@ceph.osd.5.smithi122.stdout:Apr 14 14:25:29 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-osd-5-activate[82528]: --> ceph-volume raw activate successful for osd ID: 5 2024-04-14T14:25:29.756 INFO:journalctl@ceph.osd.5.smithi122.stdout:Apr 14 14:25:29 smithi122 podman[82513]: 2024-04-14 14:25:29.444762545 +0000 UTC m=+0.796497155 container died 7b5ddec5d9067def6561dc11661015b0955024648674efd52d35c7826841b34d (image=quay.ceph.io/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888, name=ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-osd-5-activate, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.schema-version=1.0, ceph=True, RELEASE=main-a971fd5, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.build-date=20240403, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=) 2024-04-14T14:25:30.344 INFO:journalctl@ceph.osd.5.smithi122.stdout:Apr 14 14:25:30 smithi122 podman[82655]: 2024-04-14 14:25:30.030799971 +0000 UTC m=+0.580387964 container remove 7b5ddec5d9067def6561dc11661015b0955024648674efd52d35c7826841b34d (image=quay.ceph.io/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888, name=ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-osd-5-activate, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.build-date=20240403, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.vendor=CentOS, RELEASE=main-a971fd5, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, ceph=True, io.buildah.version=1.33.5, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0) 2024-04-14T14:25:30.345 INFO:journalctl@ceph.osd.5.smithi122.stdout:Apr 14 14:25:30 smithi122 podman[82719]: 2024-04-14 14:25:30.147058634 +0000 UTC m=+0.018221402 image pull b96c509eef841460b3271f5fdd331860da87b004524f4ed59414aa3680183b53 quay.ceph.io/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888 2024-04-14T14:25:30.345 INFO:journalctl@ceph.osd.5.smithi122.stdout:Apr 14 14:25:30 smithi122 podman[82719]: 2024-04-14 14:25:30.248372632 +0000 UTC m=+0.119535405 container create 02b623d44db3e6fbaec10a4e9c5d4247395195112deaba2950c7f364765104fa (image=quay.ceph.io/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888, name=ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-osd-5, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.build-date=20240403, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, RELEASE=main-a971fd5, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0) 2024-04-14T14:25:30.844 INFO:journalctl@ceph.osd.5.smithi122.stdout:Apr 14 14:25:30 smithi122 podman[82719]: 2024-04-14 14:25:30.450918132 +0000 UTC m=+0.322080898 container init 02b623d44db3e6fbaec10a4e9c5d4247395195112deaba2950c7f364765104fa (image=quay.ceph.io/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888, name=ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-osd-5, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.build-date=20240403, ceph=True, RELEASE=main-a971fd5, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5) 2024-04-14T14:25:30.845 INFO:journalctl@ceph.osd.5.smithi122.stdout:Apr 14 14:25:30 smithi122 podman[82719]: 2024-04-14 14:25:30.453819635 +0000 UTC m=+0.324982403 container start 02b623d44db3e6fbaec10a4e9c5d4247395195112deaba2950c7f364765104fa (image=quay.ceph.io/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888, name=ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-osd-5, org.label-schema.vendor=CentOS, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.build-date=20240403, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=main-a971fd5, ceph=True) 2024-04-14T14:25:30.845 INFO:journalctl@ceph.osd.5.smithi122.stdout:Apr 14 14:25:30 smithi122 ceph-osd[82738]: -- 172.21.15.122:0/3271885326 <== mon.2 v2:172.21.15.122:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x561237441680 con 0x56123740ac00 2024-04-14T14:25:30.845 INFO:journalctl@ceph.osd.5.smithi122.stdout:Apr 14 14:25:30 smithi122 bash[82719]: 02b623d44db3e6fbaec10a4e9c5d4247395195112deaba2950c7f364765104fa 2024-04-14T14:25:30.845 INFO:journalctl@ceph.osd.5.smithi122.stdout:Apr 14 14:25:30 smithi122 systemd[1]: Started Ceph osd.5 for dc326fc2-fa65-11ee-bc8e-c7b262605968. 2024-04-14T14:25:31.344 INFO:journalctl@ceph.osd.5.smithi122.stdout:Apr 14 14:25:30 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-osd-5[82734]: 2024-04-14T14:25:30.992+0000 7eff190f67c0 -1 Falling back to public interface 2024-04-14T14:25:31.895 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:25:31 smithi122 ceph-mon[67998]: pgmap v181: 161 pgs: 16 active+undersized+degraded, 7 peering, 34 active+undersized, 104 active+clean; 585 KiB data, 190 MiB used, 715 GiB / 715 GiB avail; 53/627 objects degraded (8.453%) 2024-04-14T14:25:31.895 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:25:31 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:25:31.895 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:25:31 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:25:32.066 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:25:31 smithi069 ceph-mon[83680]: pgmap v181: 161 pgs: 16 active+undersized+degraded, 7 peering, 34 active+undersized, 104 active+clean; 585 KiB data, 190 MiB used, 715 GiB / 715 GiB avail; 53/627 objects degraded (8.453%) 2024-04-14T14:25:32.067 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:25:31 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:25:32.067 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:25:31 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:25:32.067 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:25:31 smithi069 ceph-mon[88636]: pgmap v181: 161 pgs: 16 active+undersized+degraded, 7 peering, 34 active+undersized, 104 active+clean; 585 KiB data, 190 MiB used, 715 GiB / 715 GiB avail; 53/627 objects degraded (8.453%) 2024-04-14T14:25:32.067 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:25:31 smithi069 ceph-mon[88636]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:25:32.067 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:25:31 smithi069 ceph-mon[88636]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:25:32.740 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:25:32 smithi122 ceph-mon[67998]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive, 4 pgs peering) 2024-04-14T14:25:33.066 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:25:32 smithi069 ceph-mon[83680]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive, 4 pgs peering) 2024-04-14T14:25:33.067 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:25:32 smithi069 ceph-mon[88636]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive, 4 pgs peering) 2024-04-14T14:25:33.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:25:33 smithi122 ceph-mon[67998]: pgmap v182: 161 pgs: 20 active+undersized+degraded, 37 active+undersized, 104 active+clean; 585 KiB data, 191 MiB used, 715 GiB / 715 GiB avail; 71/627 objects degraded (11.324%) 2024-04-14T14:25:33.845 INFO:journalctl@ceph.osd.5.smithi122.stdout:Apr 14 14:25:33 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-osd-5[82734]: 2024-04-14T14:25:33.699+0000 7eff190f67c0 -1 osd.5 0 read_superblock omap replica is missing. 2024-04-14T14:25:33.845 INFO:journalctl@ceph.osd.5.smithi122.stdout:Apr 14 14:25:33 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-osd-5[82734]: 2024-04-14T14:25:33.729+0000 7eff190f67c0 -1 osd.5 140 log_to_monitors true 2024-04-14T14:25:34.066 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:25:33 smithi069 ceph-mon[83680]: pgmap v182: 161 pgs: 20 active+undersized+degraded, 37 active+undersized, 104 active+clean; 585 KiB data, 191 MiB used, 715 GiB / 715 GiB avail; 71/627 objects degraded (11.324%) 2024-04-14T14:25:34.067 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:25:33 smithi069 ceph-mon[88636]: pgmap v182: 161 pgs: 20 active+undersized+degraded, 37 active+undersized, 104 active+clean; 585 KiB data, 191 MiB used, 715 GiB / 715 GiB avail; 71/627 objects degraded (11.324%) 2024-04-14T14:25:34.942 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:25:34 smithi122 ceph-mon[67998]: from='osd.5 [v2:172.21.15.122:6808/581809437,v1:172.21.15.122:6809/581809437]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-04-14T14:25:34.942 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:25:34 smithi122 ceph-mon[67998]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-04-14T14:25:35.066 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:25:34 smithi069 ceph-mon[83680]: from='osd.5 [v2:172.21.15.122:6808/581809437,v1:172.21.15.122:6809/581809437]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-04-14T14:25:35.067 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:25:34 smithi069 ceph-mon[83680]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-04-14T14:25:35.067 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:25:34 smithi069 ceph-mon[88636]: from='osd.5 [v2:172.21.15.122:6808/581809437,v1:172.21.15.122:6809/581809437]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-04-14T14:25:35.067 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:25:34 smithi069 ceph-mon[88636]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-04-14T14:25:35.344 INFO:journalctl@ceph.osd.5.smithi122.stdout:Apr 14 14:25:35 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-osd-5[82734]: 2024-04-14T14:25:35.112+0000 7eff06546700 -1 osd.5 140 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-14T14:25:36.069 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:25:35 smithi069 ceph-mon[83680]: pgmap v183: 161 pgs: 20 active+undersized+degraded, 37 active+undersized, 104 active+clean; 585 KiB data, 191 MiB used, 715 GiB / 715 GiB avail; 71/627 objects degraded (11.324%) 2024-04-14T14:25:36.069 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:25:35 smithi069 ceph-mon[83680]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-04-14T14:25:36.070 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:25:35 smithi069 ceph-mon[83680]: from='osd.5 [v2:172.21.15.122:6808/581809437,v1:172.21.15.122:6809/581809437]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi122", "root=default"]}]: dispatch 2024-04-14T14:25:36.070 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:25:35 smithi069 ceph-mon[83680]: osdmap e143: 8 total, 7 up, 8 in 2024-04-14T14:25:36.070 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:25:35 smithi069 ceph-mon[83680]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi122", "root=default"]}]: dispatch 2024-04-14T14:25:36.070 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:25:35 smithi069 ceph-mon[83680]: from='osd.5 ' entity='osd.5' 2024-04-14T14:25:36.070 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:25:35 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:25:36.070 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:25:35 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:25:36.070 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:25:35 smithi069 ceph-mon[88636]: pgmap v183: 161 pgs: 20 active+undersized+degraded, 37 active+undersized, 104 active+clean; 585 KiB data, 191 MiB used, 715 GiB / 715 GiB avail; 71/627 objects degraded (11.324%) 2024-04-14T14:25:36.070 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:25:35 smithi069 ceph-mon[88636]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-04-14T14:25:36.070 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:25:35 smithi069 ceph-mon[88636]: from='osd.5 [v2:172.21.15.122:6808/581809437,v1:172.21.15.122:6809/581809437]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi122", "root=default"]}]: dispatch 2024-04-14T14:25:36.070 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:25:35 smithi069 ceph-mon[88636]: osdmap e143: 8 total, 7 up, 8 in 2024-04-14T14:25:36.071 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:25:35 smithi069 ceph-mon[88636]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi122", "root=default"]}]: dispatch 2024-04-14T14:25:36.071 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:25:35 smithi069 ceph-mon[88636]: from='osd.5 ' entity='osd.5' 2024-04-14T14:25:36.071 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:25:35 smithi069 ceph-mon[88636]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:25:36.071 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:25:35 smithi069 ceph-mon[88636]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:25:36.094 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:25:35 smithi122 ceph-mon[67998]: pgmap v183: 161 pgs: 20 active+undersized+degraded, 37 active+undersized, 104 active+clean; 585 KiB data, 191 MiB used, 715 GiB / 715 GiB avail; 71/627 objects degraded (11.324%) 2024-04-14T14:25:36.094 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:25:35 smithi122 ceph-mon[67998]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-04-14T14:25:36.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:25:35 smithi122 ceph-mon[67998]: from='osd.5 [v2:172.21.15.122:6808/581809437,v1:172.21.15.122:6809/581809437]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi122", "root=default"]}]: dispatch 2024-04-14T14:25:36.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:25:35 smithi122 ceph-mon[67998]: osdmap e143: 8 total, 7 up, 8 in 2024-04-14T14:25:36.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:25:35 smithi122 ceph-mon[67998]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi122", "root=default"]}]: dispatch 2024-04-14T14:25:36.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:25:35 smithi122 ceph-mon[67998]: from='osd.5 ' entity='osd.5' 2024-04-14T14:25:36.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:25:35 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:25:36.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:25:35 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:25:36.816 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 14 14:25:36 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-y[63038]: ::ffff:172.21.15.122 - - [14/Apr/2024:14:25:36] "GET /metrics HTTP/1.1" 200 38273 "" "Prometheus/2.43.0" 2024-04-14T14:25:36.817 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:25:36 smithi069 ceph-mon[83680]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-04-14T14:25:36.817 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:25:36 smithi069 ceph-mon[83680]: osd.5 [v2:172.21.15.122:6808/581809437,v1:172.21.15.122:6809/581809437] boot 2024-04-14T14:25:36.817 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:25:36 smithi069 ceph-mon[83680]: osdmap e144: 8 total, 8 up, 8 in 2024-04-14T14:25:36.817 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:25:36 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-14T14:25:36.817 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:25:36 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:25:36.817 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:25:36 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:25:36.817 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:25:36 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:25:36.817 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:25:36 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T14:25:36.817 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:25:36 smithi069 ceph-mon[88636]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-04-14T14:25:36.817 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:25:36 smithi069 ceph-mon[88636]: osd.5 [v2:172.21.15.122:6808/581809437,v1:172.21.15.122:6809/581809437] boot 2024-04-14T14:25:36.818 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:25:36 smithi069 ceph-mon[88636]: osdmap e144: 8 total, 8 up, 8 in 2024-04-14T14:25:36.818 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:25:36 smithi069 ceph-mon[88636]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-14T14:25:36.818 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:25:36 smithi069 ceph-mon[88636]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:25:36.818 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:25:36 smithi069 ceph-mon[88636]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:25:36.818 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:25:36 smithi069 ceph-mon[88636]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:25:36.818 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:25:36 smithi069 ceph-mon[88636]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T14:25:36.989 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:25:36 smithi122 ceph-mon[67998]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-04-14T14:25:36.989 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:25:36 smithi122 ceph-mon[67998]: osd.5 [v2:172.21.15.122:6808/581809437,v1:172.21.15.122:6809/581809437] boot 2024-04-14T14:25:36.989 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:25:36 smithi122 ceph-mon[67998]: osdmap e144: 8 total, 8 up, 8 in 2024-04-14T14:25:36.989 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:25:36 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-14T14:25:36.989 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:25:36 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:25:36.989 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:25:36 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:25:36.989 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:25:36 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:25:36.989 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:25:36 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T14:25:38.067 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:25:37 smithi069 ceph-mon[83680]: pgmap v186: 161 pgs: 20 active+undersized+degraded, 37 active+undersized, 104 active+clean; 585 KiB data, 210 MiB used, 715 GiB / 715 GiB avail; 511 B/s rd, 0 op/s; 71/627 objects degraded (11.324%) 2024-04-14T14:25:38.067 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:25:37 smithi069 ceph-mon[83680]: osdmap e145: 8 total, 8 up, 8 in 2024-04-14T14:25:38.067 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:25:37 smithi069 ceph-mon[88636]: pgmap v186: 161 pgs: 20 active+undersized+degraded, 37 active+undersized, 104 active+clean; 585 KiB data, 210 MiB used, 715 GiB / 715 GiB avail; 511 B/s rd, 0 op/s; 71/627 objects degraded (11.324%) 2024-04-14T14:25:38.068 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:25:37 smithi069 ceph-mon[88636]: osdmap e145: 8 total, 8 up, 8 in 2024-04-14T14:25:38.094 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:25:37 smithi122 ceph-mon[67998]: pgmap v186: 161 pgs: 20 active+undersized+degraded, 37 active+undersized, 104 active+clean; 585 KiB data, 210 MiB used, 715 GiB / 715 GiB avail; 511 B/s rd, 0 op/s; 71/627 objects degraded (11.324%) 2024-04-14T14:25:38.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:25:37 smithi122 ceph-mon[67998]: osdmap e145: 8 total, 8 up, 8 in 2024-04-14T14:25:38.844 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:25:38 smithi122 ceph-mon[67998]: Health check update: Degraded data redundancy: 71/627 objects degraded (11.324%), 20 pgs degraded (PG_DEGRADED) 2024-04-14T14:25:39.066 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:25:38 smithi069 ceph-mon[83680]: Health check update: Degraded data redundancy: 71/627 objects degraded (11.324%), 20 pgs degraded (PG_DEGRADED) 2024-04-14T14:25:39.067 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:25:38 smithi069 ceph-mon[88636]: Health check update: Degraded data redundancy: 71/627 objects degraded (11.324%), 20 pgs degraded (PG_DEGRADED) 2024-04-14T14:25:39.914 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:25:39 smithi122 ceph-mon[67998]: from='client.25417 -' entity='client.iscsi.foo.smithi069.yuumtr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T14:25:39.914 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:25:39 smithi122 ceph-mon[67998]: pgmap v188: 161 pgs: 11 peering, 10 active+undersized+degraded, 17 active+undersized, 123 active+clean; 585 KiB data, 210 MiB used, 715 GiB / 715 GiB avail; 340 B/s rd, 0 op/s; 38/627 objects degraded (6.061%) 2024-04-14T14:25:40.066 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:25:39 smithi069 ceph-mon[83680]: from='client.25417 -' entity='client.iscsi.foo.smithi069.yuumtr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T14:25:40.066 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:25:39 smithi069 ceph-mon[83680]: pgmap v188: 161 pgs: 11 peering, 10 active+undersized+degraded, 17 active+undersized, 123 active+clean; 585 KiB data, 210 MiB used, 715 GiB / 715 GiB avail; 340 B/s rd, 0 op/s; 38/627 objects degraded (6.061%) 2024-04-14T14:25:40.067 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:25:39 smithi069 ceph-mon[88636]: from='client.25417 -' entity='client.iscsi.foo.smithi069.yuumtr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T14:25:40.067 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:25:39 smithi069 ceph-mon[88636]: pgmap v188: 161 pgs: 11 peering, 10 active+undersized+degraded, 17 active+undersized, 123 active+clean; 585 KiB data, 210 MiB used, 715 GiB / 715 GiB avail; 340 B/s rd, 0 op/s; 38/627 objects degraded (6.061%) 2024-04-14T14:25:40.566 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 14 14:25:40 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-y[63038]: 2024-04-14T14:25:40.126+0000 7fdd38467700 -1 mgr.server reply reply (16) Device or resource busy unsafe to stop osd(s) at this time (10 PGs are or would become offline) 2024-04-14T14:25:41.317 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:25:40 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:25:41.317 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:25:40 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:25:41.317 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:25:40 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T14:25:41.317 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:25:40 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-14T14:25:41.317 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:25:40 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:25:41.318 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:25:40 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-14T14:25:41.318 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:25:40 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T14:25:41.318 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:25:40 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T14:25:41.318 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:25:40 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T14:25:41.318 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:25:40 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["6"], "max": 16}]: dispatch 2024-04-14T14:25:41.318 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:25:40 smithi069 ceph-mon[83680]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["6"], "max": 16}]: dispatch 2024-04-14T14:25:41.318 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:25:40 smithi069 ceph-mon[83680]: Upgrade: unsafe to stop osd(s) at this time (10 PGs are or would become offline) 2024-04-14T14:25:41.318 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:25:40 smithi069 ceph-mon[83680]: pgmap v189: 161 pgs: 11 peering, 1 active+undersized+degraded, 2 active+undersized, 147 active+clean; 585 KiB data, 214 MiB used, 715 GiB / 715 GiB avail; 340 B/s rd, 0 op/s; 5/627 objects degraded (0.797%) 2024-04-14T14:25:41.318 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:25:40 smithi069 ceph-mon[88636]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:25:41.318 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:25:40 smithi069 ceph-mon[88636]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:25:41.319 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:25:40 smithi069 ceph-mon[88636]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T14:25:41.319 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:25:40 smithi069 ceph-mon[88636]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-14T14:25:41.319 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:25:40 smithi069 ceph-mon[88636]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:25:41.319 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:25:40 smithi069 ceph-mon[88636]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-14T14:25:41.319 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:25:40 smithi069 ceph-mon[88636]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T14:25:41.319 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:25:40 smithi069 ceph-mon[88636]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T14:25:41.319 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:25:40 smithi069 ceph-mon[88636]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T14:25:41.319 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:25:40 smithi069 ceph-mon[88636]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["6"], "max": 16}]: dispatch 2024-04-14T14:25:41.319 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:25:40 smithi069 ceph-mon[88636]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["6"], "max": 16}]: dispatch 2024-04-14T14:25:41.319 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:25:40 smithi069 ceph-mon[88636]: Upgrade: unsafe to stop osd(s) at this time (10 PGs are or would become offline) 2024-04-14T14:25:41.319 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:25:40 smithi069 ceph-mon[88636]: pgmap v189: 161 pgs: 11 peering, 1 active+undersized+degraded, 2 active+undersized, 147 active+clean; 585 KiB data, 214 MiB used, 715 GiB / 715 GiB avail; 340 B/s rd, 0 op/s; 5/627 objects degraded (0.797%) 2024-04-14T14:25:41.344 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:25:40 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:25:41.344 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:25:40 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:25:41.345 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:25:40 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T14:25:41.345 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:25:40 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-14T14:25:41.345 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:25:40 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:25:41.345 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:25:40 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-14T14:25:41.345 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:25:40 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T14:25:41.345 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:25:40 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T14:25:41.345 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:25:40 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T14:25:41.345 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:25:40 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["6"], "max": 16}]: dispatch 2024-04-14T14:25:41.345 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:25:40 smithi122 ceph-mon[67998]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["6"], "max": 16}]: dispatch 2024-04-14T14:25:41.345 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:25:40 smithi122 ceph-mon[67998]: Upgrade: unsafe to stop osd(s) at this time (10 PGs are or would become offline) 2024-04-14T14:25:41.345 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:25:40 smithi122 ceph-mon[67998]: pgmap v189: 161 pgs: 11 peering, 1 active+undersized+degraded, 2 active+undersized, 147 active+clean; 585 KiB data, 214 MiB used, 715 GiB / 715 GiB avail; 340 B/s rd, 0 op/s; 5/627 objects degraded (0.797%) 2024-04-14T14:25:43.067 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:25:42 smithi069 ceph-mon[88636]: pgmap v190: 161 pgs: 161 active+clean; 585 KiB data, 214 MiB used, 715 GiB / 715 GiB avail; 904 B/s rd, 0 op/s 2024-04-14T14:25:43.067 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:25:42 smithi069 ceph-mon[88636]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 5/627 objects degraded (0.797%), 1 pg degraded) 2024-04-14T14:25:43.067 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:25:42 smithi069 ceph-mon[88636]: Cluster is now healthy 2024-04-14T14:25:43.067 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:25:42 smithi069 ceph-mon[83680]: pgmap v190: 161 pgs: 161 active+clean; 585 KiB data, 214 MiB used, 715 GiB / 715 GiB avail; 904 B/s rd, 0 op/s 2024-04-14T14:25:43.068 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:25:42 smithi069 ceph-mon[83680]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 5/627 objects degraded (0.797%), 1 pg degraded) 2024-04-14T14:25:43.068 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:25:42 smithi069 ceph-mon[83680]: Cluster is now healthy 2024-04-14T14:25:43.094 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:25:42 smithi122 ceph-mon[67998]: pgmap v190: 161 pgs: 161 active+clean; 585 KiB data, 214 MiB used, 715 GiB / 715 GiB avail; 904 B/s rd, 0 op/s 2024-04-14T14:25:43.094 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:25:42 smithi122 ceph-mon[67998]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 5/627 objects degraded (0.797%), 1 pg degraded) 2024-04-14T14:25:43.094 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:25:42 smithi122 ceph-mon[67998]: Cluster is now healthy 2024-04-14T14:25:46.066 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:25:45 smithi069 ceph-mon[83680]: pgmap v191: 161 pgs: 161 active+clean; 585 KiB data, 214 MiB used, 715 GiB / 715 GiB avail; 844 B/s rd, 0 op/s 2024-04-14T14:25:46.067 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:25:45 smithi069 ceph-mon[88636]: pgmap v191: 161 pgs: 161 active+clean; 585 KiB data, 214 MiB used, 715 GiB / 715 GiB avail; 844 B/s rd, 0 op/s 2024-04-14T14:25:46.094 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:25:45 smithi122 ceph-mon[67998]: pgmap v191: 161 pgs: 161 active+clean; 585 KiB data, 214 MiB used, 715 GiB / 715 GiB avail; 844 B/s rd, 0 op/s 2024-04-14T14:25:46.816 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 14 14:25:46 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-y[63038]: ::ffff:172.21.15.122 - - [14/Apr/2024:14:25:46] "GET /metrics HTTP/1.1" 200 38145 "" "Prometheus/2.43.0" 2024-04-14T14:25:47.043 INFO:teuthology.orchestra.run.smithi069.stdout:true 2024-04-14T14:25:47.698 INFO:teuthology.orchestra.run.smithi069.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-14T14:25:47.699 INFO:teuthology.orchestra.run.smithi069.stdout:alertmanager.a smithi069 *:9093,9094 running (7m) 81s ago 29m 26.5M - 0.25.0 c8568f914cd2 7524f106e7bd 2024-04-14T14:25:47.699 INFO:teuthology.orchestra.run.smithi069.stdout:grafana.a smithi122 *:3000 running (7m) 12s ago 28m 83.1M - 9.4.12 2bacad6d85d8 5685178e3eba 2024-04-14T14:25:47.699 INFO:teuthology.orchestra.run.smithi069.stdout:iscsi.foo.smithi069.yuumtr smithi069 running (8m) 81s ago 28m 62.9M - 3.5 e1d6a67b021e 38cc81ea8cfa 2024-04-14T14:25:47.699 INFO:teuthology.orchestra.run.smithi069.stdout:mgr.x smithi122 *:8443,9283,8765 running (8m) 12s ago 33m 454M - 19.0.0-2778-ga971fd5f b96c509eef84 e29977590b43 2024-04-14T14:25:47.699 INFO:teuthology.orchestra.run.smithi069.stdout:mgr.y smithi069 *:8443,9283,8765 running (18m) 81s ago 35m 507M - 19.0.0-2778-ga971fd5f b96c509eef84 7af3d29d0e5f 2024-04-14T14:25:47.699 INFO:teuthology.orchestra.run.smithi069.stdout:mon.a smithi069 running (5m) 81s ago 35m 54.7M 2048M 19.0.0-2778-ga971fd5f b96c509eef84 8bcd6b3d3846 2024-04-14T14:25:47.699 INFO:teuthology.orchestra.run.smithi069.stdout:mon.b smithi122 running (6m) 12s ago 33m 46.4M 2048M 19.0.0-2778-ga971fd5f b96c509eef84 05f437a1a9a1 2024-04-14T14:25:47.699 INFO:teuthology.orchestra.run.smithi069.stdout:mon.c smithi069 running (4m) 81s ago 34m 39.3M 2048M 19.0.0-2778-ga971fd5f b96c509eef84 84aa95f11d0c 2024-04-14T14:25:47.699 INFO:teuthology.orchestra.run.smithi069.stdout:node-exporter.a smithi069 *:9100 running (8m) 81s ago 29m 15.0M - 1.5.0 0da6a335fe13 0de0eb2171b9 2024-04-14T14:25:47.699 INFO:teuthology.orchestra.run.smithi069.stdout:node-exporter.b smithi122 *:9100 running (8m) 12s ago 29m 16.3M - 1.5.0 0da6a335fe13 67877bf02d8e 2024-04-14T14:25:47.699 INFO:teuthology.orchestra.run.smithi069.stdout:osd.0 smithi069 running (3m) 81s ago 33m 53.9M 2992M 19.0.0-2778-ga971fd5f b96c509eef84 a0e57e3fb494 2024-04-14T14:25:47.699 INFO:teuthology.orchestra.run.smithi069.stdout:osd.1 smithi069 running (3m) 81s ago 33m 52.9M 2992M 19.0.0-2778-ga971fd5f b96c509eef84 cf73d17919bb 2024-04-14T14:25:47.699 INFO:teuthology.orchestra.run.smithi069.stdout:osd.2 smithi069 running (2m) 81s ago 32m 41.9M 2992M 19.0.0-2778-ga971fd5f b96c509eef84 c0efba5ef451 2024-04-14T14:25:47.699 INFO:teuthology.orchestra.run.smithi069.stdout:osd.3 smithi069 running (86s) 81s ago 32m 15.1M 2992M 19.0.0-2778-ga971fd5f b96c509eef84 d13460da0a65 2024-04-14T14:25:47.700 INFO:teuthology.orchestra.run.smithi069.stdout:osd.4 smithi122 running (51s) 12s ago 31m 50.4M 3248M 19.0.0-2778-ga971fd5f b96c509eef84 7a87a024f984 2024-04-14T14:25:47.700 INFO:teuthology.orchestra.run.smithi069.stdout:osd.5 smithi122 running (17s) 12s ago 31m 12.5M 3248M 19.0.0-2778-ga971fd5f b96c509eef84 02b623d44db3 2024-04-14T14:25:47.700 INFO:teuthology.orchestra.run.smithi069.stdout:osd.6 smithi122 running (31m) 12s ago 31m 57.2M 3248M 17.2.0 e1d6a67b021e a2acfec61475 2024-04-14T14:25:47.700 INFO:teuthology.orchestra.run.smithi069.stdout:osd.7 smithi122 running (30m) 12s ago 30m 62.1M 3248M 17.2.0 e1d6a67b021e 62411cd06801 2024-04-14T14:25:47.700 INFO:teuthology.orchestra.run.smithi069.stdout:prometheus.a smithi122 *:9095 running (7m) 12s ago 30m 69.0M - 2.43.0 a07b618ecd1d 5c1f41d3bbe6 2024-04-14T14:25:47.700 INFO:teuthology.orchestra.run.smithi069.stdout:rgw.foo.smithi069.wesedz smithi069 *:8000 running (28m) 81s ago 28m 79.7M - 17.2.0 e1d6a67b021e 16de491440f3 2024-04-14T14:25:47.700 INFO:teuthology.orchestra.run.smithi069.stdout:rgw.foo.smithi122.jiobyv smithi122 *:8000 running (28m) 12s ago 28m 77.4M - 17.2.0 e1d6a67b021e 6d16de445dba 2024-04-14T14:25:48.066 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:25:47 smithi069 ceph-mon[83680]: pgmap v192: 161 pgs: 161 active+clean; 585 KiB data, 214 MiB used, 715 GiB / 715 GiB avail; 716 B/s rd, 0 op/s 2024-04-14T14:25:48.066 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:25:47 smithi069 ceph-mon[88636]: pgmap v192: 161 pgs: 161 active+clean; 585 KiB data, 214 MiB used, 715 GiB / 715 GiB avail; 716 B/s rd, 0 op/s 2024-04-14T14:25:48.094 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:25:47 smithi122 ceph-mon[67998]: pgmap v192: 161 pgs: 161 active+clean; 585 KiB data, 214 MiB used, 715 GiB / 715 GiB avail; 716 B/s rd, 0 op/s 2024-04-14T14:25:48.094 INFO:teuthology.orchestra.run.smithi069.stdout:{ 2024-04-14T14:25:48.095 INFO:teuthology.orchestra.run.smithi069.stdout: "mon": { 2024-04-14T14:25:48.095 INFO:teuthology.orchestra.run.smithi069.stdout: "ceph version 19.0.0-2778-ga971fd5f (a971fd5f5719038c70b817bafacd6d5400b1d888) squid (dev)": 3 2024-04-14T14:25:48.095 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-14T14:25:48.095 INFO:teuthology.orchestra.run.smithi069.stdout: "mgr": { 2024-04-14T14:25:48.095 INFO:teuthology.orchestra.run.smithi069.stdout: "ceph version 19.0.0-2778-ga971fd5f (a971fd5f5719038c70b817bafacd6d5400b1d888) squid (dev)": 2 2024-04-14T14:25:48.095 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-14T14:25:48.095 INFO:teuthology.orchestra.run.smithi069.stdout: "osd": { 2024-04-14T14:25:48.095 INFO:teuthology.orchestra.run.smithi069.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2, 2024-04-14T14:25:48.095 INFO:teuthology.orchestra.run.smithi069.stdout: "ceph version 19.0.0-2778-ga971fd5f (a971fd5f5719038c70b817bafacd6d5400b1d888) squid (dev)": 6 2024-04-14T14:25:48.095 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-14T14:25:48.095 INFO:teuthology.orchestra.run.smithi069.stdout: "rgw": { 2024-04-14T14:25:48.095 INFO:teuthology.orchestra.run.smithi069.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-04-14T14:25:48.095 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-14T14:25:48.095 INFO:teuthology.orchestra.run.smithi069.stdout: "overall": { 2024-04-14T14:25:48.096 INFO:teuthology.orchestra.run.smithi069.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 4, 2024-04-14T14:25:48.096 INFO:teuthology.orchestra.run.smithi069.stdout: "ceph version 19.0.0-2778-ga971fd5f (a971fd5f5719038c70b817bafacd6d5400b1d888) squid (dev)": 11 2024-04-14T14:25:48.096 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-14T14:25:48.096 INFO:teuthology.orchestra.run.smithi069.stdout:} 2024-04-14T14:25:48.430 INFO:teuthology.orchestra.run.smithi069.stdout:{ 2024-04-14T14:25:48.430 INFO:teuthology.orchestra.run.smithi069.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888", 2024-04-14T14:25:48.430 INFO:teuthology.orchestra.run.smithi069.stdout: "in_progress": true, 2024-04-14T14:25:48.430 INFO:teuthology.orchestra.run.smithi069.stdout: "which": "Upgrading daemons of type(s) crash,osd", 2024-04-14T14:25:48.430 INFO:teuthology.orchestra.run.smithi069.stdout: "services_complete": [], 2024-04-14T14:25:48.430 INFO:teuthology.orchestra.run.smithi069.stdout: "progress": "6/8 daemons upgraded", 2024-04-14T14:25:48.430 INFO:teuthology.orchestra.run.smithi069.stdout: "message": "Currently upgrading osd daemons", 2024-04-14T14:25:48.430 INFO:teuthology.orchestra.run.smithi069.stdout: "is_paused": false 2024-04-14T14:25:48.430 INFO:teuthology.orchestra.run.smithi069.stdout:} 2024-04-14T14:25:48.816 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:25:48 smithi069 ceph-mon[83680]: from='client.44508 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T14:25:48.816 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:25:48 smithi069 ceph-mon[83680]: from='client.44514 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T14:25:48.817 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:25:48 smithi069 ceph-mon[83680]: from='client.? 172.21.15.69:0/4032962362' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T14:25:48.817 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:25:48 smithi069 ceph-mon[88636]: from='client.44508 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T14:25:48.817 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:25:48 smithi069 ceph-mon[88636]: from='client.44514 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T14:25:48.817 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:25:48 smithi069 ceph-mon[88636]: from='client.? 172.21.15.69:0/4032962362' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T14:25:49.094 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:25:48 smithi122 ceph-mon[67998]: from='client.44508 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T14:25:49.094 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:25:48 smithi122 ceph-mon[67998]: from='client.44514 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T14:25:49.094 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:25:48 smithi122 ceph-mon[67998]: from='client.? 172.21.15.69:0/4032962362' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T14:25:50.067 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:25:49 smithi069 ceph-mon[83680]: from='client.54457 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T14:25:50.067 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:25:49 smithi069 ceph-mon[83680]: from='client.25417 -' entity='client.iscsi.foo.smithi069.yuumtr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T14:25:50.067 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:25:49 smithi069 ceph-mon[83680]: from='client.54469 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T14:25:50.067 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:25:49 smithi069 ceph-mon[83680]: pgmap v193: 161 pgs: 161 active+clean; 585 KiB data, 214 MiB used, 715 GiB / 715 GiB avail; 891 B/s rd, 0 op/s 2024-04-14T14:25:50.067 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:25:49 smithi069 ceph-mon[88636]: from='client.54457 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T14:25:50.068 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:25:49 smithi069 ceph-mon[88636]: from='client.25417 -' entity='client.iscsi.foo.smithi069.yuumtr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T14:25:50.068 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:25:49 smithi069 ceph-mon[88636]: from='client.54469 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T14:25:50.068 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:25:49 smithi069 ceph-mon[88636]: pgmap v193: 161 pgs: 161 active+clean; 585 KiB data, 214 MiB used, 715 GiB / 715 GiB avail; 891 B/s rd, 0 op/s 2024-04-14T14:25:50.094 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:25:49 smithi122 ceph-mon[67998]: from='client.54457 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T14:25:50.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:25:49 smithi122 ceph-mon[67998]: from='client.25417 -' entity='client.iscsi.foo.smithi069.yuumtr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T14:25:50.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:25:49 smithi122 ceph-mon[67998]: from='client.54469 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T14:25:50.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:25:49 smithi122 ceph-mon[67998]: pgmap v193: 161 pgs: 161 active+clean; 585 KiB data, 214 MiB used, 715 GiB / 715 GiB avail; 891 B/s rd, 0 op/s 2024-04-14T14:25:52.066 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:25:51 smithi069 ceph-mon[83680]: pgmap v194: 161 pgs: 161 active+clean; 585 KiB data, 214 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:25:52.067 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:25:51 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:25:52.067 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:25:51 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T14:25:52.067 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:25:51 smithi069 ceph-mon[88636]: pgmap v194: 161 pgs: 161 active+clean; 585 KiB data, 214 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:25:52.067 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:25:51 smithi069 ceph-mon[88636]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:25:52.067 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:25:51 smithi069 ceph-mon[88636]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T14:25:52.094 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:25:51 smithi122 ceph-mon[67998]: pgmap v194: 161 pgs: 161 active+clean; 585 KiB data, 214 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:25:52.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:25:51 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:25:52.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:25:51 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T14:25:53.316 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:25:52 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:25:53.316 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:25:52 smithi069 ceph-mon[83680]: pgmap v195: 161 pgs: 161 active+clean; 585 KiB data, 214 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:25:53.317 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:25:52 smithi069 ceph-mon[88636]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:25:53.317 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:25:52 smithi069 ceph-mon[88636]: pgmap v195: 161 pgs: 161 active+clean; 585 KiB data, 214 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:25:53.344 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:25:52 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:25:53.344 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:25:52 smithi122 ceph-mon[67998]: pgmap v195: 161 pgs: 161 active+clean; 585 KiB data, 214 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:25:55.992 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:25:55 smithi122 ceph-mon[67998]: pgmap v196: 161 pgs: 161 active+clean; 585 KiB data, 214 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:25:55.992 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:25:55 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["6"], "max": 16}]: dispatch 2024-04-14T14:25:56.066 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:25:55 smithi069 ceph-mon[83680]: pgmap v196: 161 pgs: 161 active+clean; 585 KiB data, 214 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:25:56.066 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:25:55 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["6"], "max": 16}]: dispatch 2024-04-14T14:25:56.067 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:25:55 smithi069 ceph-mon[88636]: pgmap v196: 161 pgs: 161 active+clean; 585 KiB data, 214 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:25:56.067 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:25:55 smithi069 ceph-mon[88636]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["6"], "max": 16}]: dispatch 2024-04-14T14:25:56.816 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:25:56 smithi069 ceph-mon[83680]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["6"], "max": 16}]: dispatch 2024-04-14T14:25:56.816 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:25:56 smithi069 ceph-mon[83680]: Upgrade: osd.6 is safe to restart 2024-04-14T14:25:56.817 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 14 14:25:56 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-y[63038]: ::ffff:172.21.15.122 - - [14/Apr/2024:14:25:56] "GET /metrics HTTP/1.1" 200 38282 "" "Prometheus/2.43.0" 2024-04-14T14:25:56.817 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:25:56 smithi069 ceph-mon[88636]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["6"], "max": 16}]: dispatch 2024-04-14T14:25:56.817 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:25:56 smithi069 ceph-mon[88636]: Upgrade: osd.6 is safe to restart 2024-04-14T14:25:57.094 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:25:56 smithi122 ceph-mon[67998]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["6"], "max": 16}]: dispatch 2024-04-14T14:25:57.094 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:25:56 smithi122 ceph-mon[67998]: Upgrade: osd.6 is safe to restart 2024-04-14T14:25:58.066 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:25:57 smithi069 ceph-mon[88636]: pgmap v197: 161 pgs: 161 active+clean; 585 KiB data, 214 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:25:58.067 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:25:57 smithi069 ceph-mon[88636]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:25:58.067 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:25:57 smithi069 ceph-mon[88636]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-04-14T14:25:58.067 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:25:57 smithi069 ceph-mon[88636]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T14:25:58.067 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:25:57 smithi069 ceph-mon[83680]: pgmap v197: 161 pgs: 161 active+clean; 585 KiB data, 214 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:25:58.067 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:25:57 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:25:58.067 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:25:57 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-04-14T14:25:58.067 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:25:57 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T14:25:58.094 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:25:57 smithi122 ceph-mon[67998]: pgmap v197: 161 pgs: 161 active+clean; 585 KiB data, 214 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:25:58.094 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:25:57 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:25:58.094 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:25:57 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-04-14T14:25:58.094 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:25:57 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T14:25:58.943 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:25:58 smithi122 ceph-mon[67998]: Upgrade: Updating osd.6 2024-04-14T14:25:58.943 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:25:58 smithi122 ceph-mon[67998]: Deploying daemon osd.6 on smithi122 2024-04-14T14:25:59.066 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:25:58 smithi069 ceph-mon[83680]: Upgrade: Updating osd.6 2024-04-14T14:25:59.067 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:25:58 smithi069 ceph-mon[83680]: Deploying daemon osd.6 on smithi122 2024-04-14T14:25:59.067 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:25:58 smithi069 ceph-mon[88636]: Upgrade: Updating osd.6 2024-04-14T14:25:59.067 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:25:58 smithi069 ceph-mon[88636]: Deploying daemon osd.6 on smithi122 2024-04-14T14:26:00.066 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:25:59 smithi069 ceph-mon[83680]: from='client.25417 -' entity='client.iscsi.foo.smithi069.yuumtr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T14:26:00.067 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:25:59 smithi069 ceph-mon[83680]: pgmap v198: 161 pgs: 161 active+clean; 585 KiB data, 214 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:26:00.067 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:25:59 smithi069 ceph-mon[88636]: from='client.25417 -' entity='client.iscsi.foo.smithi069.yuumtr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T14:26:00.067 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:25:59 smithi069 ceph-mon[88636]: pgmap v198: 161 pgs: 161 active+clean; 585 KiB data, 214 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:26:00.068 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:25:59 smithi122 ceph-mon[67998]: from='client.25417 -' entity='client.iscsi.foo.smithi069.yuumtr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T14:26:00.068 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:25:59 smithi122 ceph-mon[67998]: pgmap v198: 161 pgs: 161 active+clean; 585 KiB data, 214 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:26:00.321 INFO:journalctl@ceph.osd.6.smithi122.stdout:Apr 14 14:26:00 smithi122 systemd[1]: Stopping Ceph osd.6 for dc326fc2-fa65-11ee-bc8e-c7b262605968... 2024-04-14T14:26:00.594 INFO:journalctl@ceph.osd.6.smithi122.stdout:Apr 14 14:26:00 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-osd-6[34955]: 2024-04-14T14:26:00.359+0000 7f6e61f2e700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.6 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-14T14:26:00.594 INFO:journalctl@ceph.osd.6.smithi122.stdout:Apr 14 14:26:00 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-osd-6[34955]: 2024-04-14T14:26:00.359+0000 7f6e61f2e700 -1 osd.6 145 *** Got signal Terminated *** 2024-04-14T14:26:00.594 INFO:journalctl@ceph.osd.6.smithi122.stdout:Apr 14 14:26:00 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-osd-6[34955]: 2024-04-14T14:26:00.359+0000 7f6e61f2e700 -1 osd.6 145 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-14T14:26:01.035 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:26:00 smithi122 ceph-mon[67998]: osd.6 marked itself down and dead 2024-04-14T14:26:01.066 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:26:00 smithi069 ceph-mon[83680]: osd.6 marked itself down and dead 2024-04-14T14:26:01.067 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:26:00 smithi069 ceph-mon[88636]: osd.6 marked itself down and dead 2024-04-14T14:26:01.320 INFO:journalctl@ceph.osd.6.smithi122.stdout:Apr 14 14:26:01 smithi122 podman[86997]: 2024-04-14 14:26:01.034196381 +0000 UTC m=+0.791674410 container died a2acfec614757cf2447eaabfd3a80b312b8c98a12961a69272fc0ca5537669bc (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-osd-6, io.k8s.display-name=CentOS Stream 8, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, io.openshift.expose-services=, io.buildah.version=1.19.8, RELEASE=HEAD, io.openshift.tags=base centos centos-stream, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, CEPH_POINT_RELEASE=-17.2.0, distribution-scope=public, release=754, com.redhat.component=centos-stream-container, vendor=Red Hat, Inc., maintainer=Guillaume Abrioux , version=8, GIT_REPO=https://github.com/ceph/ceph-container.git, architecture=x86_64, name=centos-stream, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, ceph=True, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, vcs-type=git, GIT_BRANCH=HEAD, GIT_CLEAN=True, build-date=2022-05-03T08:36:31.336870, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., com.redhat.license_terms=https://centos.org/legal/licensing-policy/, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac) 2024-04-14T14:26:01.594 INFO:journalctl@ceph.osd.6.smithi122.stdout:Apr 14 14:26:01 smithi122 podman[86997]: 2024-04-14 14:26:01.418800154 +0000 UTC m=+1.176278178 container cleanup a2acfec614757cf2447eaabfd3a80b312b8c98a12961a69272fc0ca5537669bc (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-osd-6, io.k8s.display-name=CentOS Stream 8, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, io.buildah.version=1.19.8, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, io.openshift.expose-services=, maintainer=Guillaume Abrioux , version=8, release=754, com.redhat.component=centos-stream-container, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, name=centos-stream, GIT_BRANCH=HEAD, RELEASE=HEAD, vendor=Red Hat, Inc., description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, distribution-scope=public, CEPH_POINT_RELEASE=-17.2.0, architecture=x86_64, build-date=2022-05-03T08:36:31.336870, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, io.openshift.tags=base centos centos-stream, GIT_CLEAN=True, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, GIT_REPO=https://github.com/ceph/ceph-container.git, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., ceph=True, vcs-type=git) 2024-04-14T14:26:01.594 INFO:journalctl@ceph.osd.6.smithi122.stdout:Apr 14 14:26:01 smithi122 bash[86997]: ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-osd-6 2024-04-14T14:26:02.026 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:26:01 smithi122 ceph-mon[67998]: pgmap v199: 161 pgs: 161 active+clean; 585 KiB data, 214 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-04-14T14:26:02.026 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:26:01 smithi122 ceph-mon[67998]: Health check failed: 1 osds down (OSD_DOWN) 2024-04-14T14:26:02.026 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:26:01 smithi122 ceph-mon[67998]: osdmap e146: 8 total, 7 up, 8 in 2024-04-14T14:26:02.026 INFO:journalctl@ceph.osd.6.smithi122.stdout:Apr 14 14:26:01 smithi122 podman[87014]: 2024-04-14 14:26:01.703572825 +0000 UTC m=+0.668511719 container remove a2acfec614757cf2447eaabfd3a80b312b8c98a12961a69272fc0ca5537669bc (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-osd-6, io.openshift.tags=base centos centos-stream, RELEASE=HEAD, io.buildah.version=1.19.8, ceph=True, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, io.k8s.display-name=CentOS Stream 8, GIT_CLEAN=True, CEPH_POINT_RELEASE=-17.2.0, vendor=Red Hat, Inc., release=754, build-date=2022-05-03T08:36:31.336870, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, vcs-type=git, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, io.openshift.expose-services=, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, architecture=x86_64, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., GIT_REPO=https://github.com/ceph/ceph-container.git, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, com.redhat.component=centos-stream-container, name=centos-stream, distribution-scope=public, version=8, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD) 2024-04-14T14:26:02.069 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:26:01 smithi069 ceph-mon[83680]: pgmap v199: 161 pgs: 161 active+clean; 585 KiB data, 214 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-04-14T14:26:02.070 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:26:01 smithi069 ceph-mon[83680]: Health check failed: 1 osds down (OSD_DOWN) 2024-04-14T14:26:02.070 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:26:01 smithi069 ceph-mon[83680]: osdmap e146: 8 total, 7 up, 8 in 2024-04-14T14:26:02.070 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:26:01 smithi069 ceph-mon[88636]: pgmap v199: 161 pgs: 161 active+clean; 585 KiB data, 214 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-04-14T14:26:02.070 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:26:01 smithi069 ceph-mon[88636]: Health check failed: 1 osds down (OSD_DOWN) 2024-04-14T14:26:02.070 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:26:01 smithi069 ceph-mon[88636]: osdmap e146: 8 total, 7 up, 8 in 2024-04-14T14:26:02.333 INFO:journalctl@ceph.osd.6.smithi122.stdout:Apr 14 14:26:02 smithi122 podman[87107]: 2024-04-14 14:26:01.928107192 +0000 UTC m=+0.017731147 image pull b96c509eef841460b3271f5fdd331860da87b004524f4ed59414aa3680183b53 quay.ceph.io/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888 2024-04-14T14:26:02.333 INFO:journalctl@ceph.osd.6.smithi122.stdout:Apr 14 14:26:02 smithi122 podman[87107]: 2024-04-14 14:26:02.054637725 +0000 UTC m=+0.144261662 container create 373344abfc5ee5440a587fefe78cf0c77ea85fdd1a198c1cc9de2426dd0676eb (image=quay.ceph.io/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888, name=ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-osd-6-deactivate, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, ceph=True, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=main-a971fd5, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.build-date=20240403) 2024-04-14T14:26:02.333 INFO:journalctl@ceph.osd.6.smithi122.stdout:Apr 14 14:26:02 smithi122 podman[87107]: 2024-04-14 14:26:02.249719658 +0000 UTC m=+0.339343604 container init 373344abfc5ee5440a587fefe78cf0c77ea85fdd1a198c1cc9de2426dd0676eb (image=quay.ceph.io/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888, name=ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-osd-6-deactivate, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240403, ceph=True, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=main-a971fd5) 2024-04-14T14:26:02.333 INFO:journalctl@ceph.osd.6.smithi122.stdout:Apr 14 14:26:02 smithi122 podman[87107]: 2024-04-14 14:26:02.253059687 +0000 UTC m=+0.342683631 container start 373344abfc5ee5440a587fefe78cf0c77ea85fdd1a198c1cc9de2426dd0676eb (image=quay.ceph.io/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888, name=ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-osd-6-deactivate, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.build-date=20240403, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, RELEASE=main-a971fd5, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-14T14:26:02.333 INFO:journalctl@ceph.osd.6.smithi122.stdout:Apr 14 14:26:02 smithi122 podman[87107]: 2024-04-14 14:26:02.33223801 +0000 UTC m=+0.421861956 container attach 373344abfc5ee5440a587fefe78cf0c77ea85fdd1a198c1cc9de2426dd0676eb (image=quay.ceph.io/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888, name=ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-osd-6-deactivate, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, GIT_BRANCH=HEAD, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240403, RELEASE=main-a971fd5) 2024-04-14T14:26:02.594 INFO:journalctl@ceph.osd.6.smithi122.stdout:Apr 14 14:26:02 smithi122 podman[87107]: 2024-04-14 14:26:02.542129917 +0000 UTC m=+0.631753860 container died 373344abfc5ee5440a587fefe78cf0c77ea85fdd1a198c1cc9de2426dd0676eb (image=quay.ceph.io/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888, name=ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-osd-6-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=main-a971fd5, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.build-date=20240403, org.label-schema.license=GPLv2, ceph=True) 2024-04-14T14:26:03.066 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:26:02 smithi069 ceph-mon[83680]: osdmap e147: 8 total, 7 up, 8 in 2024-04-14T14:26:03.067 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:26:02 smithi069 ceph-mon[88636]: osdmap e147: 8 total, 7 up, 8 in 2024-04-14T14:26:03.094 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:26:02 smithi122 ceph-mon[67998]: osdmap e147: 8 total, 7 up, 8 in 2024-04-14T14:26:03.594 INFO:journalctl@ceph.osd.6.smithi122.stdout:Apr 14 14:26:03 smithi122 podman[87134]: 2024-04-14 14:26:03.17882445 +0000 UTC m=+0.628067533 container remove 373344abfc5ee5440a587fefe78cf0c77ea85fdd1a198c1cc9de2426dd0676eb (image=quay.ceph.io/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888, name=ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-osd-6-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240403, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, RELEASE=main-a971fd5, GIT_CLEAN=True, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=) 2024-04-14T14:26:03.595 INFO:journalctl@ceph.osd.6.smithi122.stdout:Apr 14 14:26:03 smithi122 systemd[1]: ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968@osd.6.service: Deactivated successfully. 2024-04-14T14:26:03.595 INFO:journalctl@ceph.osd.6.smithi122.stdout:Apr 14 14:26:03 smithi122 systemd[1]: Stopped Ceph osd.6 for dc326fc2-fa65-11ee-bc8e-c7b262605968. 2024-04-14T14:26:03.595 INFO:journalctl@ceph.osd.6.smithi122.stdout:Apr 14 14:26:03 smithi122 systemd[1]: ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968@osd.6.service: Consumed 14.429s CPU time. 2024-04-14T14:26:03.595 INFO:journalctl@ceph.osd.6.smithi122.stdout:Apr 14 14:26:03 smithi122 systemd[1]: Starting Ceph osd.6 for dc326fc2-fa65-11ee-bc8e-c7b262605968... 2024-04-14T14:26:03.876 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:26:03 smithi122 ceph-mon[67998]: pgmap v202: 161 pgs: 6 active+undersized+degraded, 8 peering, 10 stale+active+clean, 13 active+undersized, 124 active+clean; 585 KiB data, 214 MiB used, 715 GiB / 715 GiB avail; 255 B/s rd, 0 op/s; 26/627 objects degraded (4.147%) 2024-04-14T14:26:03.876 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:26:03 smithi122 ceph-mon[67998]: Health check failed: Degraded data redundancy: 26/627 objects degraded (4.147%), 6 pgs degraded (PG_DEGRADED) 2024-04-14T14:26:03.876 INFO:journalctl@ceph.osd.6.smithi122.stdout:Apr 14 14:26:03 smithi122 podman[87239]: 2024-04-14 14:26:03.515774376 +0000 UTC m=+0.018641290 image pull b96c509eef841460b3271f5fdd331860da87b004524f4ed59414aa3680183b53 quay.ceph.io/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888 2024-04-14T14:26:03.876 INFO:journalctl@ceph.osd.6.smithi122.stdout:Apr 14 14:26:03 smithi122 podman[87239]: 2024-04-14 14:26:03.631634733 +0000 UTC m=+0.134501634 container create e185e12f0ad9ccf52f97670e84f77cc4de627f4e101c957664b95652dd087bd7 (image=quay.ceph.io/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888, name=ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-osd-6-activate, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=main-a971fd5, org.label-schema.build-date=20240403, GIT_CLEAN=True, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD) 2024-04-14T14:26:04.066 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:26:03 smithi069 ceph-mon[83680]: pgmap v202: 161 pgs: 6 active+undersized+degraded, 8 peering, 10 stale+active+clean, 13 active+undersized, 124 active+clean; 585 KiB data, 214 MiB used, 715 GiB / 715 GiB avail; 255 B/s rd, 0 op/s; 26/627 objects degraded (4.147%) 2024-04-14T14:26:04.067 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:26:03 smithi069 ceph-mon[83680]: Health check failed: Degraded data redundancy: 26/627 objects degraded (4.147%), 6 pgs degraded (PG_DEGRADED) 2024-04-14T14:26:04.067 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:26:03 smithi069 ceph-mon[88636]: pgmap v202: 161 pgs: 6 active+undersized+degraded, 8 peering, 10 stale+active+clean, 13 active+undersized, 124 active+clean; 585 KiB data, 214 MiB used, 715 GiB / 715 GiB avail; 255 B/s rd, 0 op/s; 26/627 objects degraded (4.147%) 2024-04-14T14:26:04.067 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:26:03 smithi069 ceph-mon[88636]: Health check failed: Degraded data redundancy: 26/627 objects degraded (4.147%), 6 pgs degraded (PG_DEGRADED) 2024-04-14T14:26:04.327 INFO:journalctl@ceph.osd.6.smithi122.stdout:Apr 14 14:26:03 smithi122 podman[87239]: 2024-04-14 14:26:03.875277237 +0000 UTC m=+0.378144148 container init e185e12f0ad9ccf52f97670e84f77cc4de627f4e101c957664b95652dd087bd7 (image=quay.ceph.io/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888, name=ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-osd-6-activate, ceph=True, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, RELEASE=main-a971fd5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.build-date=20240403, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0) 2024-04-14T14:26:04.327 INFO:journalctl@ceph.osd.6.smithi122.stdout:Apr 14 14:26:03 smithi122 podman[87239]: 2024-04-14 14:26:03.879046431 +0000 UTC m=+0.381913334 container start e185e12f0ad9ccf52f97670e84f77cc4de627f4e101c957664b95652dd087bd7 (image=quay.ceph.io/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888, name=ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-osd-6-activate, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, ceph=True, io.buildah.version=1.33.5, org.label-schema.build-date=20240403, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=main-a971fd5, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-14T14:26:04.327 INFO:journalctl@ceph.osd.6.smithi122.stdout:Apr 14 14:26:03 smithi122 podman[87239]: 2024-04-14 14:26:03.975167254 +0000 UTC m=+0.478034156 container attach e185e12f0ad9ccf52f97670e84f77cc4de627f4e101c957664b95652dd087bd7 (image=quay.ceph.io/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888, name=ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-osd-6-activate, org.label-schema.vendor=CentOS, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, RELEASE=main-a971fd5, org.label-schema.build-date=20240403, CEPH_POINT_RELEASE=, GIT_CLEAN=True, maintainer=Guillaume Abrioux ) 2024-04-14T14:26:04.595 INFO:journalctl@ceph.osd.6.smithi122.stdout:Apr 14 14:26:04 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-osd-6-activate[87254]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-6 2024-04-14T14:26:04.595 INFO:journalctl@ceph.osd.6.smithi122.stdout:Apr 14 14:26:04 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-osd-6-activate[87254]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-6 --no-mon-config --dev /dev/mapper/vg_nvme-lv_2 2024-04-14T14:26:04.595 INFO:journalctl@ceph.osd.6.smithi122.stdout:Apr 14 14:26:04 smithi122 bash[87239]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-6 2024-04-14T14:26:04.595 INFO:journalctl@ceph.osd.6.smithi122.stdout:Apr 14 14:26:04 smithi122 bash[87239]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-6 --no-mon-config --dev /dev/mapper/vg_nvme-lv_2 2024-04-14T14:26:04.595 INFO:journalctl@ceph.osd.6.smithi122.stdout:Apr 14 14:26:04 smithi122 bash[87239]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_2 2024-04-14T14:26:04.595 INFO:journalctl@ceph.osd.6.smithi122.stdout:Apr 14 14:26:04 smithi122 bash[87239]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-1 2024-04-14T14:26:04.595 INFO:journalctl@ceph.osd.6.smithi122.stdout:Apr 14 14:26:04 smithi122 bash[87239]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-6/block 2024-04-14T14:26:04.596 INFO:journalctl@ceph.osd.6.smithi122.stdout:Apr 14 14:26:04 smithi122 bash[87239]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-6 2024-04-14T14:26:04.596 INFO:journalctl@ceph.osd.6.smithi122.stdout:Apr 14 14:26:04 smithi122 bash[87239]: --> ceph-volume raw activate successful for osd ID: 6 2024-04-14T14:26:04.596 INFO:journalctl@ceph.osd.6.smithi122.stdout:Apr 14 14:26:04 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-osd-6-activate[87254]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_2 2024-04-14T14:26:04.596 INFO:journalctl@ceph.osd.6.smithi122.stdout:Apr 14 14:26:04 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-osd-6-activate[87254]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-1 2024-04-14T14:26:04.596 INFO:journalctl@ceph.osd.6.smithi122.stdout:Apr 14 14:26:04 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-osd-6-activate[87254]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-6/block 2024-04-14T14:26:04.596 INFO:journalctl@ceph.osd.6.smithi122.stdout:Apr 14 14:26:04 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-osd-6-activate[87254]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-6 2024-04-14T14:26:04.596 INFO:journalctl@ceph.osd.6.smithi122.stdout:Apr 14 14:26:04 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-osd-6-activate[87254]: --> ceph-volume raw activate successful for osd ID: 6 2024-04-14T14:26:04.596 INFO:journalctl@ceph.osd.6.smithi122.stdout:Apr 14 14:26:04 smithi122 podman[87239]: 2024-04-14 14:26:04.350825362 +0000 UTC m=+0.853692266 container died e185e12f0ad9ccf52f97670e84f77cc4de627f4e101c957664b95652dd087bd7 (image=quay.ceph.io/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888, name=ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-osd-6-activate, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, ceph=True, GIT_CLEAN=True, GIT_BRANCH=HEAD, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, RELEASE=main-a971fd5, org.label-schema.build-date=20240403, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-14T14:26:05.270 INFO:journalctl@ceph.osd.6.smithi122.stdout:Apr 14 14:26:04 smithi122 podman[87371]: 2024-04-14 14:26:04.946291829 +0000 UTC m=+0.590674902 container remove e185e12f0ad9ccf52f97670e84f77cc4de627f4e101c957664b95652dd087bd7 (image=quay.ceph.io/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888, name=ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-osd-6-activate, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_CLEAN=True, io.buildah.version=1.33.5, ceph=True, org.label-schema.vendor=CentOS, RELEASE=main-a971fd5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240403) 2024-04-14T14:26:05.271 INFO:journalctl@ceph.osd.6.smithi122.stdout:Apr 14 14:26:05 smithi122 podman[87436]: 2024-04-14 14:26:05.070162276 +0000 UTC m=+0.019611520 image pull b96c509eef841460b3271f5fdd331860da87b004524f4ed59414aa3680183b53 quay.ceph.io/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888 2024-04-14T14:26:05.271 INFO:journalctl@ceph.osd.6.smithi122.stdout:Apr 14 14:26:05 smithi122 podman[87436]: 2024-04-14 14:26:05.180447853 +0000 UTC m=+0.129897078 container create 72f85ece3e75f338fd86fb1d159cf293ad4896b0c47290c9634e9324ea9ae4d9 (image=quay.ceph.io/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888, name=ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-osd-6, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_CLEAN=True, io.buildah.version=1.33.5, RELEASE=main-a971fd5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, ceph=True, org.label-schema.build-date=20240403, maintainer=Guillaume Abrioux , GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-14T14:26:05.594 INFO:journalctl@ceph.osd.6.smithi122.stdout:Apr 14 14:26:05 smithi122 podman[87436]: 2024-04-14 14:26:05.375019475 +0000 UTC m=+0.324468701 container init 72f85ece3e75f338fd86fb1d159cf293ad4896b0c47290c9634e9324ea9ae4d9 (image=quay.ceph.io/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888, name=ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-osd-6, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240403, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, RELEASE=main-a971fd5, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD) 2024-04-14T14:26:05.595 INFO:journalctl@ceph.osd.6.smithi122.stdout:Apr 14 14:26:05 smithi122 podman[87436]: 2024-04-14 14:26:05.378047225 +0000 UTC m=+0.327496451 container start 72f85ece3e75f338fd86fb1d159cf293ad4896b0c47290c9634e9324ea9ae4d9 (image=quay.ceph.io/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888, name=ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-osd-6, RELEASE=main-a971fd5, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, maintainer=Guillaume Abrioux , GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240403) 2024-04-14T14:26:05.595 INFO:journalctl@ceph.osd.6.smithi122.stdout:Apr 14 14:26:05 smithi122 ceph-osd[87455]: -- 172.21.15.122:0/643812542 <== mon.2 v2:172.21.15.122:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55558d091680 con 0x55558d05b400 2024-04-14T14:26:05.595 INFO:journalctl@ceph.osd.6.smithi122.stdout:Apr 14 14:26:05 smithi122 bash[87436]: 72f85ece3e75f338fd86fb1d159cf293ad4896b0c47290c9634e9324ea9ae4d9 2024-04-14T14:26:05.595 INFO:journalctl@ceph.osd.6.smithi122.stdout:Apr 14 14:26:05 smithi122 systemd[1]: Started Ceph osd.6 for dc326fc2-fa65-11ee-bc8e-c7b262605968. 2024-04-14T14:26:06.066 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:26:05 smithi069 ceph-mon[83680]: pgmap v203: 161 pgs: 7 active+undersized+degraded, 8 peering, 8 stale+active+clean, 15 active+undersized, 123 active+clean; 585 KiB data, 215 MiB used, 715 GiB / 715 GiB avail; 127 B/s rd, 0 op/s; 28/627 objects degraded (4.466%) 2024-04-14T14:26:06.066 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:26:05 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:26:06.067 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:26:05 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:26:06.067 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:26:05 smithi069 ceph-mon[88636]: pgmap v203: 161 pgs: 7 active+undersized+degraded, 8 peering, 8 stale+active+clean, 15 active+undersized, 123 active+clean; 585 KiB data, 215 MiB used, 715 GiB / 715 GiB avail; 127 B/s rd, 0 op/s; 28/627 objects degraded (4.466%) 2024-04-14T14:26:06.067 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:26:05 smithi069 ceph-mon[88636]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:26:06.067 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:26:05 smithi069 ceph-mon[88636]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:26:06.094 INFO:journalctl@ceph.osd.6.smithi122.stdout:Apr 14 14:26:05 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-osd-6[87451]: 2024-04-14T14:26:05.921+0000 7ff6593207c0 -1 Falling back to public interface 2024-04-14T14:26:06.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:26:05 smithi122 ceph-mon[67998]: pgmap v203: 161 pgs: 7 active+undersized+degraded, 8 peering, 8 stale+active+clean, 15 active+undersized, 123 active+clean; 585 KiB data, 215 MiB used, 715 GiB / 715 GiB avail; 127 B/s rd, 0 op/s; 28/627 objects degraded (4.466%) 2024-04-14T14:26:06.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:26:05 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:26:06.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:26:05 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:26:06.816 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 14 14:26:06 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-y[63038]: ::ffff:172.21.15.122 - - [14/Apr/2024:14:26:06] "GET /metrics HTTP/1.1" 200 38282 "" "Prometheus/2.43.0" 2024-04-14T14:26:07.844 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:26:07 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:26:07.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:26:07 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T14:26:07.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:26:07 smithi122 ceph-mon[67998]: pgmap v204: 161 pgs: 17 active+undersized+degraded, 36 active+undersized, 108 active+clean; 585 KiB data, 215 MiB used, 715 GiB / 715 GiB avail; 73/627 objects degraded (11.643%) 2024-04-14T14:26:08.067 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:26:07 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:26:08.067 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:26:07 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T14:26:08.067 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:26:07 smithi069 ceph-mon[83680]: pgmap v204: 161 pgs: 17 active+undersized+degraded, 36 active+undersized, 108 active+clean; 585 KiB data, 215 MiB used, 715 GiB / 715 GiB avail; 73/627 objects degraded (11.643%) 2024-04-14T14:26:08.068 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:26:07 smithi069 ceph-mon[88636]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:26:08.069 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:26:07 smithi069 ceph-mon[88636]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T14:26:08.069 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:26:07 smithi069 ceph-mon[88636]: pgmap v204: 161 pgs: 17 active+undersized+degraded, 36 active+undersized, 108 active+clean; 585 KiB data, 215 MiB used, 715 GiB / 715 GiB avail; 73/627 objects degraded (11.643%) 2024-04-14T14:26:08.844 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:26:08 smithi122 ceph-mon[67998]: Health check update: Degraded data redundancy: 73/627 objects degraded (11.643%), 17 pgs degraded (PG_DEGRADED) 2024-04-14T14:26:08.844 INFO:journalctl@ceph.osd.6.smithi122.stdout:Apr 14 14:26:08 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-osd-6[87451]: 2024-04-14T14:26:08.644+0000 7ff6593207c0 -1 osd.6 0 read_superblock omap replica is missing. 2024-04-14T14:26:08.845 INFO:journalctl@ceph.osd.6.smithi122.stdout:Apr 14 14:26:08 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-osd-6[87451]: 2024-04-14T14:26:08.669+0000 7ff6593207c0 -1 osd.6 145 log_to_monitors true 2024-04-14T14:26:09.066 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:26:08 smithi069 ceph-mon[83680]: Health check update: Degraded data redundancy: 73/627 objects degraded (11.643%), 17 pgs degraded (PG_DEGRADED) 2024-04-14T14:26:09.124 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:26:08 smithi069 ceph-mon[88636]: Health check update: Degraded data redundancy: 73/627 objects degraded (11.643%), 17 pgs degraded (PG_DEGRADED) 2024-04-14T14:26:09.844 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:26:09 smithi122 ceph-mon[67998]: from='client.25417 -' entity='client.iscsi.foo.smithi069.yuumtr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T14:26:09.844 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:26:09 smithi122 ceph-mon[67998]: pgmap v205: 161 pgs: 17 active+undersized+degraded, 36 active+undersized, 108 active+clean; 585 KiB data, 215 MiB used, 715 GiB / 715 GiB avail; 73/627 objects degraded (11.643%) 2024-04-14T14:26:09.844 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:26:09 smithi122 ceph-mon[67998]: from='osd.6 [v2:172.21.15.122:6816/1449680613,v1:172.21.15.122:6817/1449680613]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-04-14T14:26:09.844 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:26:09 smithi122 ceph-mon[67998]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-04-14T14:26:10.066 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:26:09 smithi069 ceph-mon[83680]: from='client.25417 -' entity='client.iscsi.foo.smithi069.yuumtr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T14:26:10.066 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:26:09 smithi069 ceph-mon[83680]: pgmap v205: 161 pgs: 17 active+undersized+degraded, 36 active+undersized, 108 active+clean; 585 KiB data, 215 MiB used, 715 GiB / 715 GiB avail; 73/627 objects degraded (11.643%) 2024-04-14T14:26:10.066 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:26:09 smithi069 ceph-mon[83680]: from='osd.6 [v2:172.21.15.122:6816/1449680613,v1:172.21.15.122:6817/1449680613]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-04-14T14:26:10.067 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:26:09 smithi069 ceph-mon[83680]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-04-14T14:26:10.067 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:26:09 smithi069 ceph-mon[88636]: from='client.25417 -' entity='client.iscsi.foo.smithi069.yuumtr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T14:26:10.067 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:26:09 smithi069 ceph-mon[88636]: pgmap v205: 161 pgs: 17 active+undersized+degraded, 36 active+undersized, 108 active+clean; 585 KiB data, 215 MiB used, 715 GiB / 715 GiB avail; 73/627 objects degraded (11.643%) 2024-04-14T14:26:10.067 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:26:09 smithi069 ceph-mon[88636]: from='osd.6 [v2:172.21.15.122:6816/1449680613,v1:172.21.15.122:6817/1449680613]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-04-14T14:26:10.067 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:26:09 smithi069 ceph-mon[88636]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-04-14T14:26:10.344 INFO:journalctl@ceph.osd.6.smithi122.stdout:Apr 14 14:26:10 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-osd-6[87451]: 2024-04-14T14:26:10.028+0000 7ff646770700 -1 osd.6 145 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-14T14:26:10.844 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:26:10 smithi122 ceph-mon[67998]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-04-14T14:26:10.844 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:26:10 smithi122 ceph-mon[67998]: from='osd.6 [v2:172.21.15.122:6816/1449680613,v1:172.21.15.122:6817/1449680613]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi122", "root=default"]}]: dispatch 2024-04-14T14:26:10.844 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:26:10 smithi122 ceph-mon[67998]: osdmap e148: 8 total, 7 up, 8 in 2024-04-14T14:26:10.844 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:26:10 smithi122 ceph-mon[67998]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi122", "root=default"]}]: dispatch 2024-04-14T14:26:10.844 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:26:10 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:26:10.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:26:10 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:26:10.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:26:10 smithi122 ceph-mon[67998]: from='osd.6 ' entity='osd.6' 2024-04-14T14:26:11.066 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:26:10 smithi069 ceph-mon[83680]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-04-14T14:26:11.066 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:26:10 smithi069 ceph-mon[83680]: from='osd.6 [v2:172.21.15.122:6816/1449680613,v1:172.21.15.122:6817/1449680613]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi122", "root=default"]}]: dispatch 2024-04-14T14:26:11.066 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:26:10 smithi069 ceph-mon[83680]: osdmap e148: 8 total, 7 up, 8 in 2024-04-14T14:26:11.067 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:26:10 smithi069 ceph-mon[83680]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi122", "root=default"]}]: dispatch 2024-04-14T14:26:11.067 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:26:10 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:26:11.067 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:26:10 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:26:11.067 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:26:10 smithi069 ceph-mon[83680]: from='osd.6 ' entity='osd.6' 2024-04-14T14:26:11.067 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:26:10 smithi069 ceph-mon[88636]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-04-14T14:26:11.067 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:26:10 smithi069 ceph-mon[88636]: from='osd.6 [v2:172.21.15.122:6816/1449680613,v1:172.21.15.122:6817/1449680613]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi122", "root=default"]}]: dispatch 2024-04-14T14:26:11.067 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:26:10 smithi069 ceph-mon[88636]: osdmap e148: 8 total, 7 up, 8 in 2024-04-14T14:26:11.067 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:26:10 smithi069 ceph-mon[88636]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi122", "root=default"]}]: dispatch 2024-04-14T14:26:11.067 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:26:10 smithi069 ceph-mon[88636]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:26:11.067 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:26:10 smithi069 ceph-mon[88636]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:26:11.067 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:26:10 smithi069 ceph-mon[88636]: from='osd.6 ' entity='osd.6' 2024-04-14T14:26:11.844 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:26:11 smithi122 ceph-mon[67998]: pgmap v207: 161 pgs: 17 active+undersized+degraded, 36 active+undersized, 108 active+clean; 585 KiB data, 298 MiB used, 715 GiB / 715 GiB avail; 73/627 objects degraded (11.643%) 2024-04-14T14:26:11.844 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:26:11 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:26:11.844 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:26:11 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:26:11.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:26:11 smithi122 ceph-mon[67998]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-04-14T14:26:11.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:26:11 smithi122 ceph-mon[67998]: osd.6 [v2:172.21.15.122:6816/1449680613,v1:172.21.15.122:6817/1449680613] boot 2024-04-14T14:26:11.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:26:11 smithi122 ceph-mon[67998]: osdmap e149: 8 total, 8 up, 8 in 2024-04-14T14:26:11.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:26:11 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-14T14:26:12.066 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:26:11 smithi069 ceph-mon[88636]: pgmap v207: 161 pgs: 17 active+undersized+degraded, 36 active+undersized, 108 active+clean; 585 KiB data, 298 MiB used, 715 GiB / 715 GiB avail; 73/627 objects degraded (11.643%) 2024-04-14T14:26:12.067 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:26:11 smithi069 ceph-mon[88636]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:26:12.067 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:26:11 smithi069 ceph-mon[88636]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:26:12.067 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:26:11 smithi069 ceph-mon[88636]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-04-14T14:26:12.067 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:26:11 smithi069 ceph-mon[88636]: osd.6 [v2:172.21.15.122:6816/1449680613,v1:172.21.15.122:6817/1449680613] boot 2024-04-14T14:26:12.067 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:26:11 smithi069 ceph-mon[88636]: osdmap e149: 8 total, 8 up, 8 in 2024-04-14T14:26:12.067 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:26:11 smithi069 ceph-mon[88636]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-14T14:26:12.067 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:26:11 smithi069 ceph-mon[83680]: pgmap v207: 161 pgs: 17 active+undersized+degraded, 36 active+undersized, 108 active+clean; 585 KiB data, 298 MiB used, 715 GiB / 715 GiB avail; 73/627 objects degraded (11.643%) 2024-04-14T14:26:12.067 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:26:11 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:26:12.067 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:26:11 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:26:12.067 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:26:11 smithi069 ceph-mon[83680]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-04-14T14:26:12.067 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:26:11 smithi069 ceph-mon[83680]: osd.6 [v2:172.21.15.122:6816/1449680613,v1:172.21.15.122:6817/1449680613] boot 2024-04-14T14:26:12.068 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:26:11 smithi069 ceph-mon[83680]: osdmap e149: 8 total, 8 up, 8 in 2024-04-14T14:26:12.068 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:26:11 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-14T14:26:13.244 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:26:13 smithi122 ceph-mon[67998]: osdmap e150: 8 total, 8 up, 8 in 2024-04-14T14:26:13.244 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:26:13 smithi122 ceph-mon[67998]: pgmap v210: 161 pgs: 9 peering, 12 active+undersized+degraded, 24 active+undersized, 116 active+clean; 585 KiB data, 302 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s; 48/627 objects degraded (7.656%) 2024-04-14T14:26:13.316 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:26:13 smithi069 ceph-mon[83680]: osdmap e150: 8 total, 8 up, 8 in 2024-04-14T14:26:13.317 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:26:13 smithi069 ceph-mon[83680]: pgmap v210: 161 pgs: 9 peering, 12 active+undersized+degraded, 24 active+undersized, 116 active+clean; 585 KiB data, 302 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s; 48/627 objects degraded (7.656%) 2024-04-14T14:26:13.317 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:26:13 smithi069 ceph-mon[88636]: osdmap e150: 8 total, 8 up, 8 in 2024-04-14T14:26:13.317 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:26:13 smithi069 ceph-mon[88636]: pgmap v210: 161 pgs: 9 peering, 12 active+undersized+degraded, 24 active+undersized, 116 active+clean; 585 KiB data, 302 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s; 48/627 objects degraded (7.656%) 2024-04-14T14:26:14.222 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:26:14 smithi122 ceph-mon[67998]: Health check update: Degraded data redundancy: 48/627 objects degraded (7.656%), 12 pgs degraded (PG_DEGRADED) 2024-04-14T14:26:14.316 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:26:14 smithi069 ceph-mon[83680]: Health check update: Degraded data redundancy: 48/627 objects degraded (7.656%), 12 pgs degraded (PG_DEGRADED) 2024-04-14T14:26:14.316 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:26:14 smithi069 ceph-mon[88636]: Health check update: Degraded data redundancy: 48/627 objects degraded (7.656%), 12 pgs degraded (PG_DEGRADED) 2024-04-14T14:26:15.066 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 14 14:26:14 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-y[63038]: 2024-04-14T14:26:14.779+0000 7fdd38467700 -1 mgr.server reply reply (16) Device or resource busy unsafe to stop osd(s) at this time (10 PGs are or would become offline) 2024-04-14T14:26:16.066 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:26:15 smithi069 ceph-mon[83680]: pgmap v211: 161 pgs: 9 peering, 12 active+undersized+degraded, 23 active+undersized, 117 active+clean; 585 KiB data, 302 MiB used, 715 GiB / 715 GiB avail; 511 B/s rd, 0 op/s; 48/627 objects degraded (7.656%) 2024-04-14T14:26:16.067 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:26:15 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:26:16.067 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:26:15 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:26:16.067 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:26:15 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T14:26:16.067 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:26:15 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-14T14:26:16.067 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:26:15 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:26:16.067 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:26:15 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-14T14:26:16.067 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:26:15 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T14:26:16.067 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:26:15 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T14:26:16.067 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:26:15 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T14:26:16.067 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:26:15 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["7"], "max": 16}]: dispatch 2024-04-14T14:26:16.068 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:26:15 smithi069 ceph-mon[88636]: pgmap v211: 161 pgs: 9 peering, 12 active+undersized+degraded, 23 active+undersized, 117 active+clean; 585 KiB data, 302 MiB used, 715 GiB / 715 GiB avail; 511 B/s rd, 0 op/s; 48/627 objects degraded (7.656%) 2024-04-14T14:26:16.068 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:26:15 smithi069 ceph-mon[88636]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:26:16.068 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:26:15 smithi069 ceph-mon[88636]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:26:16.068 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:26:15 smithi069 ceph-mon[88636]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T14:26:16.068 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:26:15 smithi069 ceph-mon[88636]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-14T14:26:16.068 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:26:15 smithi069 ceph-mon[88636]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:26:16.068 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:26:15 smithi069 ceph-mon[88636]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-14T14:26:16.068 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:26:15 smithi069 ceph-mon[88636]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T14:26:16.068 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:26:15 smithi069 ceph-mon[88636]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T14:26:16.068 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:26:15 smithi069 ceph-mon[88636]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T14:26:16.068 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:26:15 smithi069 ceph-mon[88636]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["7"], "max": 16}]: dispatch 2024-04-14T14:26:16.094 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:26:15 smithi122 ceph-mon[67998]: pgmap v211: 161 pgs: 9 peering, 12 active+undersized+degraded, 23 active+undersized, 117 active+clean; 585 KiB data, 302 MiB used, 715 GiB / 715 GiB avail; 511 B/s rd, 0 op/s; 48/627 objects degraded (7.656%) 2024-04-14T14:26:16.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:26:15 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:26:16.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:26:15 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:26:16.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:26:15 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T14:26:16.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:26:15 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-14T14:26:16.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:26:15 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:26:16.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:26:15 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-14T14:26:16.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:26:15 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T14:26:16.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:26:15 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T14:26:16.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:26:15 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T14:26:16.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:26:15 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["7"], "max": 16}]: dispatch 2024-04-14T14:26:16.816 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:26:16 smithi069 ceph-mon[88636]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["7"], "max": 16}]: dispatch 2024-04-14T14:26:16.816 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:26:16 smithi069 ceph-mon[88636]: Upgrade: unsafe to stop osd(s) at this time (10 PGs are or would become offline) 2024-04-14T14:26:16.817 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:26:16 smithi069 ceph-mon[83680]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["7"], "max": 16}]: dispatch 2024-04-14T14:26:16.817 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:26:16 smithi069 ceph-mon[83680]: Upgrade: unsafe to stop osd(s) at this time (10 PGs are or would become offline) 2024-04-14T14:26:16.817 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 14 14:26:16 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-y[63038]: ::ffff:172.21.15.122 - - [14/Apr/2024:14:26:16] "GET /metrics HTTP/1.1" 200 38282 "" "Prometheus/2.43.0" 2024-04-14T14:26:17.094 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:26:16 smithi122 ceph-mon[67998]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["7"], "max": 16}]: dispatch 2024-04-14T14:26:17.094 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:26:16 smithi122 ceph-mon[67998]: Upgrade: unsafe to stop osd(s) at this time (10 PGs are or would become offline) 2024-04-14T14:26:18.066 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:26:17 smithi069 ceph-mon[83680]: pgmap v212: 161 pgs: 9 peering, 152 active+clean; 585 KiB data, 238 MiB used, 715 GiB / 715 GiB avail; 437 B/s rd, 0 op/s 2024-04-14T14:26:18.066 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:26:17 smithi069 ceph-mon[83680]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 48/627 objects degraded (7.656%), 12 pgs degraded) 2024-04-14T14:26:18.067 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:26:17 smithi069 ceph-mon[83680]: Cluster is now healthy 2024-04-14T14:26:18.067 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:26:17 smithi069 ceph-mon[88636]: pgmap v212: 161 pgs: 9 peering, 152 active+clean; 585 KiB data, 238 MiB used, 715 GiB / 715 GiB avail; 437 B/s rd, 0 op/s 2024-04-14T14:26:18.067 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:26:17 smithi069 ceph-mon[88636]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 48/627 objects degraded (7.656%), 12 pgs degraded) 2024-04-14T14:26:18.067 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:26:17 smithi069 ceph-mon[88636]: Cluster is now healthy 2024-04-14T14:26:18.094 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:26:17 smithi122 ceph-mon[67998]: pgmap v212: 161 pgs: 9 peering, 152 active+clean; 585 KiB data, 238 MiB used, 715 GiB / 715 GiB avail; 437 B/s rd, 0 op/s 2024-04-14T14:26:18.094 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:26:17 smithi122 ceph-mon[67998]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 48/627 objects degraded (7.656%), 12 pgs degraded) 2024-04-14T14:26:18.094 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:26:17 smithi122 ceph-mon[67998]: Cluster is now healthy 2024-04-14T14:26:18.790 INFO:teuthology.orchestra.run.smithi069.stdout:true 2024-04-14T14:26:19.437 INFO:teuthology.orchestra.run.smithi069.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-14T14:26:19.438 INFO:teuthology.orchestra.run.smithi069.stdout:alertmanager.a smithi069 *:9093,9094 running (8m) 113s ago 29m 26.5M - 0.25.0 c8568f914cd2 7524f106e7bd 2024-04-14T14:26:19.438 INFO:teuthology.orchestra.run.smithi069.stdout:grafana.a smithi122 *:3000 running (7m) 9s ago 29m 80.2M - 9.4.12 2bacad6d85d8 5685178e3eba 2024-04-14T14:26:19.438 INFO:teuthology.orchestra.run.smithi069.stdout:iscsi.foo.smithi069.yuumtr smithi069 running (9m) 113s ago 28m 62.9M - 3.5 e1d6a67b021e 38cc81ea8cfa 2024-04-14T14:26:19.438 INFO:teuthology.orchestra.run.smithi069.stdout:mgr.x smithi122 *:8443,9283,8765 running (8m) 9s ago 34m 454M - 19.0.0-2778-ga971fd5f b96c509eef84 e29977590b43 2024-04-14T14:26:19.438 INFO:teuthology.orchestra.run.smithi069.stdout:mgr.y smithi069 *:8443,9283,8765 running (18m) 113s ago 35m 507M - 19.0.0-2778-ga971fd5f b96c509eef84 7af3d29d0e5f 2024-04-14T14:26:19.438 INFO:teuthology.orchestra.run.smithi069.stdout:mon.a smithi069 running (5m) 113s ago 35m 54.7M 2048M 19.0.0-2778-ga971fd5f b96c509eef84 8bcd6b3d3846 2024-04-14T14:26:19.438 INFO:teuthology.orchestra.run.smithi069.stdout:mon.b smithi122 running (6m) 9s ago 34m 47.7M 2048M 19.0.0-2778-ga971fd5f b96c509eef84 05f437a1a9a1 2024-04-14T14:26:19.438 INFO:teuthology.orchestra.run.smithi069.stdout:mon.c smithi069 running (5m) 113s ago 34m 39.3M 2048M 19.0.0-2778-ga971fd5f b96c509eef84 84aa95f11d0c 2024-04-14T14:26:19.438 INFO:teuthology.orchestra.run.smithi069.stdout:node-exporter.a smithi069 *:9100 running (8m) 113s ago 30m 15.0M - 1.5.0 0da6a335fe13 0de0eb2171b9 2024-04-14T14:26:19.438 INFO:teuthology.orchestra.run.smithi069.stdout:node-exporter.b smithi122 *:9100 running (8m) 9s ago 30m 16.1M - 1.5.0 0da6a335fe13 67877bf02d8e 2024-04-14T14:26:19.438 INFO:teuthology.orchestra.run.smithi069.stdout:osd.0 smithi069 running (4m) 113s ago 33m 53.9M 2992M 19.0.0-2778-ga971fd5f b96c509eef84 a0e57e3fb494 2024-04-14T14:26:19.438 INFO:teuthology.orchestra.run.smithi069.stdout:osd.1 smithi069 running (3m) 113s ago 33m 52.9M 2992M 19.0.0-2778-ga971fd5f b96c509eef84 cf73d17919bb 2024-04-14T14:26:19.438 INFO:teuthology.orchestra.run.smithi069.stdout:osd.2 smithi069 running (2m) 113s ago 33m 41.9M 2992M 19.0.0-2778-ga971fd5f b96c509eef84 c0efba5ef451 2024-04-14T14:26:19.439 INFO:teuthology.orchestra.run.smithi069.stdout:osd.3 smithi069 running (118s) 113s ago 32m 15.1M 2992M 19.0.0-2778-ga971fd5f b96c509eef84 d13460da0a65 2024-04-14T14:26:19.439 INFO:teuthology.orchestra.run.smithi069.stdout:osd.4 smithi122 running (83s) 9s ago 32m 52.0M 3248M 19.0.0-2778-ga971fd5f b96c509eef84 7a87a024f984 2024-04-14T14:26:19.439 INFO:teuthology.orchestra.run.smithi069.stdout:osd.5 smithi122 running (49s) 9s ago 32m 52.5M 3248M 19.0.0-2778-ga971fd5f b96c509eef84 02b623d44db3 2024-04-14T14:26:19.439 INFO:teuthology.orchestra.run.smithi069.stdout:osd.6 smithi122 running (14s) 9s ago 31m 14.3M 3248M 19.0.0-2778-ga971fd5f b96c509eef84 72f85ece3e75 2024-04-14T14:26:19.439 INFO:teuthology.orchestra.run.smithi069.stdout:osd.7 smithi122 running (31m) 9s ago 31m 63.3M 3248M 17.2.0 e1d6a67b021e 62411cd06801 2024-04-14T14:26:19.439 INFO:teuthology.orchestra.run.smithi069.stdout:prometheus.a smithi122 *:9095 running (8m) 9s ago 30m 71.1M - 2.43.0 a07b618ecd1d 5c1f41d3bbe6 2024-04-14T14:26:19.439 INFO:teuthology.orchestra.run.smithi069.stdout:rgw.foo.smithi069.wesedz smithi069 *:8000 running (29m) 113s ago 29m 79.7M - 17.2.0 e1d6a67b021e 16de491440f3 2024-04-14T14:26:19.439 INFO:teuthology.orchestra.run.smithi069.stdout:rgw.foo.smithi122.jiobyv smithi122 *:8000 running (28m) 9s ago 28m 77.5M - 17.2.0 e1d6a67b021e 6d16de445dba 2024-04-14T14:26:19.816 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:26:19 smithi069 ceph-mon[83680]: from='client.25417 -' entity='client.iscsi.foo.smithi069.yuumtr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T14:26:19.816 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:26:19 smithi069 ceph-mon[83680]: pgmap v213: 161 pgs: 161 active+clean; 585 KiB data, 238 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-14T14:26:19.817 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:26:19 smithi069 ceph-mon[88636]: from='client.25417 -' entity='client.iscsi.foo.smithi069.yuumtr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T14:26:19.817 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:26:19 smithi069 ceph-mon[88636]: pgmap v213: 161 pgs: 161 active+clean; 585 KiB data, 238 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-14T14:26:19.837 INFO:teuthology.orchestra.run.smithi069.stdout:{ 2024-04-14T14:26:19.837 INFO:teuthology.orchestra.run.smithi069.stdout: "mon": { 2024-04-14T14:26:19.837 INFO:teuthology.orchestra.run.smithi069.stdout: "ceph version 19.0.0-2778-ga971fd5f (a971fd5f5719038c70b817bafacd6d5400b1d888) squid (dev)": 3 2024-04-14T14:26:19.837 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-14T14:26:19.837 INFO:teuthology.orchestra.run.smithi069.stdout: "mgr": { 2024-04-14T14:26:19.837 INFO:teuthology.orchestra.run.smithi069.stdout: "ceph version 19.0.0-2778-ga971fd5f (a971fd5f5719038c70b817bafacd6d5400b1d888) squid (dev)": 2 2024-04-14T14:26:19.837 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-14T14:26:19.837 INFO:teuthology.orchestra.run.smithi069.stdout: "osd": { 2024-04-14T14:26:19.837 INFO:teuthology.orchestra.run.smithi069.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 1, 2024-04-14T14:26:19.837 INFO:teuthology.orchestra.run.smithi069.stdout: "ceph version 19.0.0-2778-ga971fd5f (a971fd5f5719038c70b817bafacd6d5400b1d888) squid (dev)": 7 2024-04-14T14:26:19.837 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-14T14:26:19.837 INFO:teuthology.orchestra.run.smithi069.stdout: "rgw": { 2024-04-14T14:26:19.837 INFO:teuthology.orchestra.run.smithi069.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-04-14T14:26:19.838 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-14T14:26:19.838 INFO:teuthology.orchestra.run.smithi069.stdout: "overall": { 2024-04-14T14:26:19.838 INFO:teuthology.orchestra.run.smithi069.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 3, 2024-04-14T14:26:19.838 INFO:teuthology.orchestra.run.smithi069.stdout: "ceph version 19.0.0-2778-ga971fd5f (a971fd5f5719038c70b817bafacd6d5400b1d888) squid (dev)": 12 2024-04-14T14:26:19.838 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-14T14:26:19.838 INFO:teuthology.orchestra.run.smithi069.stdout:} 2024-04-14T14:26:20.094 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:26:19 smithi122 ceph-mon[67998]: from='client.25417 -' entity='client.iscsi.foo.smithi069.yuumtr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T14:26:20.094 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:26:19 smithi122 ceph-mon[67998]: pgmap v213: 161 pgs: 161 active+clean; 585 KiB data, 238 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-14T14:26:20.169 INFO:teuthology.orchestra.run.smithi069.stdout:{ 2024-04-14T14:26:20.169 INFO:teuthology.orchestra.run.smithi069.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888", 2024-04-14T14:26:20.169 INFO:teuthology.orchestra.run.smithi069.stdout: "in_progress": true, 2024-04-14T14:26:20.169 INFO:teuthology.orchestra.run.smithi069.stdout: "which": "Upgrading daemons of type(s) crash,osd", 2024-04-14T14:26:20.169 INFO:teuthology.orchestra.run.smithi069.stdout: "services_complete": [], 2024-04-14T14:26:20.169 INFO:teuthology.orchestra.run.smithi069.stdout: "progress": "7/8 daemons upgraded", 2024-04-14T14:26:20.169 INFO:teuthology.orchestra.run.smithi069.stdout: "message": "Currently upgrading osd daemons", 2024-04-14T14:26:20.169 INFO:teuthology.orchestra.run.smithi069.stdout: "is_paused": false 2024-04-14T14:26:20.169 INFO:teuthology.orchestra.run.smithi069.stdout:} 2024-04-14T14:26:21.066 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:26:20 smithi069 ceph-mon[83680]: from='client.44541 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T14:26:21.067 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:26:20 smithi069 ceph-mon[83680]: from='client.44547 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T14:26:21.067 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:26:20 smithi069 ceph-mon[83680]: from='client.44553 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T14:26:21.067 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:26:20 smithi069 ceph-mon[83680]: from='client.? 172.21.15.69:0/801639465' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T14:26:21.067 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:26:20 smithi069 ceph-mon[88636]: from='client.44541 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T14:26:21.067 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:26:20 smithi069 ceph-mon[88636]: from='client.44547 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T14:26:21.067 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:26:20 smithi069 ceph-mon[88636]: from='client.44553 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T14:26:21.067 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:26:20 smithi069 ceph-mon[88636]: from='client.? 172.21.15.69:0/801639465' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T14:26:21.094 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:26:20 smithi122 ceph-mon[67998]: from='client.44541 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T14:26:21.094 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:26:20 smithi122 ceph-mon[67998]: from='client.44547 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T14:26:21.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:26:20 smithi122 ceph-mon[67998]: from='client.44553 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T14:26:21.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:26:20 smithi122 ceph-mon[67998]: from='client.? 172.21.15.69:0/801639465' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T14:26:22.066 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:26:21 smithi069 ceph-mon[83680]: from='client.54502 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T14:26:22.067 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:26:21 smithi069 ceph-mon[83680]: pgmap v214: 161 pgs: 161 active+clean; 585 KiB data, 238 MiB used, 715 GiB / 715 GiB avail; 854 B/s rd, 0 op/s 2024-04-14T14:26:22.067 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:26:21 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:26:22.067 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:26:21 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T14:26:22.067 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:26:21 smithi069 ceph-mon[88636]: from='client.54502 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T14:26:22.067 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:26:21 smithi069 ceph-mon[88636]: pgmap v214: 161 pgs: 161 active+clean; 585 KiB data, 238 MiB used, 715 GiB / 715 GiB avail; 854 B/s rd, 0 op/s 2024-04-14T14:26:22.067 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:26:21 smithi069 ceph-mon[88636]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:26:22.067 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:26:21 smithi069 ceph-mon[88636]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T14:26:22.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:26:21 smithi122 ceph-mon[67998]: from='client.54502 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T14:26:22.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:26:21 smithi122 ceph-mon[67998]: pgmap v214: 161 pgs: 161 active+clean; 585 KiB data, 238 MiB used, 715 GiB / 715 GiB avail; 854 B/s rd, 0 op/s 2024-04-14T14:26:22.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:26:21 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:26:22.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:26:21 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T14:26:24.066 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:26:23 smithi069 ceph-mon[83680]: pgmap v215: 161 pgs: 161 active+clean; 585 KiB data, 238 MiB used, 715 GiB / 715 GiB avail; 967 B/s rd, 0 op/s 2024-04-14T14:26:24.067 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:26:23 smithi069 ceph-mon[88636]: pgmap v215: 161 pgs: 161 active+clean; 585 KiB data, 238 MiB used, 715 GiB / 715 GiB avail; 967 B/s rd, 0 op/s 2024-04-14T14:26:24.094 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:26:23 smithi122 ceph-mon[67998]: pgmap v215: 161 pgs: 161 active+clean; 585 KiB data, 238 MiB used, 715 GiB / 715 GiB avail; 967 B/s rd, 0 op/s 2024-04-14T14:26:26.066 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:26:25 smithi069 ceph-mon[83680]: pgmap v216: 161 pgs: 161 active+clean; 585 KiB data, 238 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:26:26.067 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:26:25 smithi069 ceph-mon[88636]: pgmap v216: 161 pgs: 161 active+clean; 585 KiB data, 238 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:26:26.094 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:26:25 smithi122 ceph-mon[67998]: pgmap v216: 161 pgs: 161 active+clean; 585 KiB data, 238 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:26:26.816 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 14 14:26:26 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-y[63038]: ::ffff:172.21.15.122 - - [14/Apr/2024:14:26:26] "GET /metrics HTTP/1.1" 200 38287 "" "Prometheus/2.43.0" 2024-04-14T14:26:28.066 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:26:27 smithi069 ceph-mon[83680]: pgmap v217: 161 pgs: 161 active+clean; 585 KiB data, 238 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:26:28.067 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:26:27 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:26:28.067 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:26:27 smithi069 ceph-mon[88636]: pgmap v217: 161 pgs: 161 active+clean; 585 KiB data, 238 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:26:28.067 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:26:27 smithi069 ceph-mon[88636]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:26:28.094 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:26:27 smithi122 ceph-mon[67998]: pgmap v217: 161 pgs: 161 active+clean; 585 KiB data, 238 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:26:28.094 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:26:27 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:26:29.566 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:26:29 smithi069 ceph-mon[83680]: from='client.25417 -' entity='client.iscsi.foo.smithi069.yuumtr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T14:26:29.566 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:26:29 smithi069 ceph-mon[83680]: pgmap v218: 161 pgs: 161 active+clean; 585 KiB data, 238 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:26:29.567 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:26:29 smithi069 ceph-mon[88636]: from='client.25417 -' entity='client.iscsi.foo.smithi069.yuumtr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T14:26:29.567 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:26:29 smithi069 ceph-mon[88636]: pgmap v218: 161 pgs: 161 active+clean; 585 KiB data, 238 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:26:29.594 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:26:29 smithi122 ceph-mon[67998]: from='client.25417 -' entity='client.iscsi.foo.smithi069.yuumtr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T14:26:29.594 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:26:29 smithi122 ceph-mon[67998]: pgmap v218: 161 pgs: 161 active+clean; 585 KiB data, 238 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:26:30.506 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:26:30 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["7"], "max": 16}]: dispatch 2024-04-14T14:26:30.566 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:26:30 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["7"], "max": 16}]: dispatch 2024-04-14T14:26:30.566 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:26:30 smithi069 ceph-mon[88636]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["7"], "max": 16}]: dispatch 2024-04-14T14:26:31.566 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:26:31 smithi069 ceph-mon[88636]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["7"], "max": 16}]: dispatch 2024-04-14T14:26:31.566 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:26:31 smithi069 ceph-mon[88636]: Upgrade: osd.7 is safe to restart 2024-04-14T14:26:31.567 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:26:31 smithi069 ceph-mon[88636]: pgmap v219: 161 pgs: 161 active+clean; 585 KiB data, 238 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:26:31.567 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:26:31 smithi069 ceph-mon[83680]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["7"], "max": 16}]: dispatch 2024-04-14T14:26:31.567 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:26:31 smithi069 ceph-mon[83680]: Upgrade: osd.7 is safe to restart 2024-04-14T14:26:31.567 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:26:31 smithi069 ceph-mon[83680]: pgmap v219: 161 pgs: 161 active+clean; 585 KiB data, 238 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:26:31.594 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:26:31 smithi122 ceph-mon[67998]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["7"], "max": 16}]: dispatch 2024-04-14T14:26:31.594 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:26:31 smithi122 ceph-mon[67998]: Upgrade: osd.7 is safe to restart 2024-04-14T14:26:31.594 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:26:31 smithi122 ceph-mon[67998]: pgmap v219: 161 pgs: 161 active+clean; 585 KiB data, 238 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:26:32.594 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:26:32 smithi122 ceph-mon[67998]: Upgrade: Updating osd.7 2024-04-14T14:26:32.594 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:26:32 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:26:32.594 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:26:32 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-04-14T14:26:32.594 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:26:32 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T14:26:32.595 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:26:32 smithi122 ceph-mon[67998]: Deploying daemon osd.7 on smithi122 2024-04-14T14:26:32.816 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:26:32 smithi069 ceph-mon[83680]: Upgrade: Updating osd.7 2024-04-14T14:26:32.817 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:26:32 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:26:32.817 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:26:32 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-04-14T14:26:32.817 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:26:32 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T14:26:32.817 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:26:32 smithi069 ceph-mon[83680]: Deploying daemon osd.7 on smithi122 2024-04-14T14:26:32.817 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:26:32 smithi069 ceph-mon[88636]: Upgrade: Updating osd.7 2024-04-14T14:26:32.817 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:26:32 smithi069 ceph-mon[88636]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:26:32.817 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:26:32 smithi069 ceph-mon[88636]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-04-14T14:26:32.817 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:26:32 smithi069 ceph-mon[88636]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T14:26:32.817 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:26:32 smithi069 ceph-mon[88636]: Deploying daemon osd.7 on smithi122 2024-04-14T14:26:33.594 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:26:33 smithi122 ceph-mon[67998]: pgmap v220: 161 pgs: 161 active+clean; 585 KiB data, 238 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:26:33.816 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:26:33 smithi069 ceph-mon[83680]: pgmap v220: 161 pgs: 161 active+clean; 585 KiB data, 238 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:26:33.816 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:26:33 smithi069 ceph-mon[88636]: pgmap v220: 161 pgs: 161 active+clean; 585 KiB data, 238 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:26:34.844 INFO:journalctl@ceph.osd.7.smithi122.stdout:Apr 14 14:26:34 smithi122 systemd[1]: Stopping Ceph osd.7 for dc326fc2-fa65-11ee-bc8e-c7b262605968... 2024-04-14T14:26:35.344 INFO:journalctl@ceph.osd.7.smithi122.stdout:Apr 14 14:26:34 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-osd-7[37950]: 2024-04-14T14:26:34.865+0000 7f6198b91700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.7 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-14T14:26:35.344 INFO:journalctl@ceph.osd.7.smithi122.stdout:Apr 14 14:26:34 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-osd-7[37950]: 2024-04-14T14:26:34.865+0000 7f6198b91700 -1 osd.7 150 *** Got signal Terminated *** 2024-04-14T14:26:35.344 INFO:journalctl@ceph.osd.7.smithi122.stdout:Apr 14 14:26:34 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-osd-7[37950]: 2024-04-14T14:26:34.865+0000 7f6198b91700 -1 osd.7 150 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-14T14:26:36.066 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:26:35 smithi069 ceph-mon[83680]: pgmap v221: 161 pgs: 161 active+clean; 585 KiB data, 238 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:26:36.066 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:26:35 smithi069 ceph-mon[83680]: osd.7 marked itself down and dead 2024-04-14T14:26:36.067 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:26:35 smithi069 ceph-mon[88636]: pgmap v221: 161 pgs: 161 active+clean; 585 KiB data, 238 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:26:36.067 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:26:35 smithi069 ceph-mon[88636]: osd.7 marked itself down and dead 2024-04-14T14:26:36.094 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:26:35 smithi122 ceph-mon[67998]: pgmap v221: 161 pgs: 161 active+clean; 585 KiB data, 238 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:26:36.094 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:26:35 smithi122 ceph-mon[67998]: osd.7 marked itself down and dead 2024-04-14T14:26:36.387 INFO:journalctl@ceph.osd.7.smithi122.stdout:Apr 14 14:26:36 smithi122 podman[91974]: 2024-04-14 14:26:36.097934337 +0000 UTC m=+1.416408056 container died 62411cd06801aca05a5fb45000d2f206b708e645d86ebfacc76cf8547451b265 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-osd-7, vcs-type=git, maintainer=Guillaume Abrioux , io.openshift.tags=base centos centos-stream, architecture=x86_64, io.buildah.version=1.19.8, vendor=Red Hat, Inc., vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, com.redhat.component=centos-stream-container, io.openshift.expose-services=, RELEASE=HEAD, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, distribution-scope=public, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, CEPH_POINT_RELEASE=-17.2.0, build-date=2022-05-03T08:36:31.336870, release=754, GIT_CLEAN=True, GIT_BRANCH=HEAD, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, io.k8s.display-name=CentOS Stream 8, version=8, ceph=True, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., GIT_REPO=https://github.com/ceph/ceph-container.git, name=centos-stream, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com) 2024-04-14T14:26:36.689 INFO:journalctl@ceph.osd.7.smithi122.stdout:Apr 14 14:26:36 smithi122 podman[91974]: 2024-04-14 14:26:36.47632598 +0000 UTC m=+1.794799694 container cleanup 62411cd06801aca05a5fb45000d2f206b708e645d86ebfacc76cf8547451b265 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-osd-7, build-date=2022-05-03T08:36:31.336870, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, ceph=True, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, io.k8s.display-name=CentOS Stream 8, GIT_CLEAN=True, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, distribution-scope=public, architecture=x86_64, vendor=Red Hat, Inc., CEPH_POINT_RELEASE=-17.2.0, com.redhat.component=centos-stream-container, name=centos-stream, version=8, GIT_BRANCH=HEAD, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., com.redhat.license_terms=https://centos.org/legal/licensing-policy/, io.openshift.tags=base centos centos-stream, release=754, maintainer=Guillaume Abrioux , io.openshift.expose-services=, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, RELEASE=HEAD, GIT_REPO=https://github.com/ceph/ceph-container.git, io.buildah.version=1.19.8, vcs-type=git, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream) 2024-04-14T14:26:36.689 INFO:journalctl@ceph.osd.7.smithi122.stdout:Apr 14 14:26:36 smithi122 bash[91974]: ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-osd-7 2024-04-14T14:26:36.816 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 14 14:26:36 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-y[63038]: ::ffff:172.21.15.122 - - [14/Apr/2024:14:26:36] "GET /metrics HTTP/1.1" 200 38287 "" "Prometheus/2.43.0" 2024-04-14T14:26:36.817 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:26:36 smithi069 ceph-mon[83680]: Health check failed: 1 osds down (OSD_DOWN) 2024-04-14T14:26:36.817 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:26:36 smithi069 ceph-mon[83680]: Health check failed: all OSDs are running squid or later but require_osd_release < squid (OSD_UPGRADE_FINISHED) 2024-04-14T14:26:36.817 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:26:36 smithi069 ceph-mon[83680]: osdmap e151: 8 total, 7 up, 8 in 2024-04-14T14:26:36.818 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:26:36 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:26:36.818 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:26:36 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T14:26:36.818 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:26:36 smithi069 ceph-mon[83680]: osdmap e152: 8 total, 7 up, 8 in 2024-04-14T14:26:36.818 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:26:36 smithi069 ceph-mon[88636]: Health check failed: 1 osds down (OSD_DOWN) 2024-04-14T14:26:36.819 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:26:36 smithi069 ceph-mon[88636]: Health check failed: all OSDs are running squid or later but require_osd_release < squid (OSD_UPGRADE_FINISHED) 2024-04-14T14:26:36.819 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:26:36 smithi069 ceph-mon[88636]: osdmap e151: 8 total, 7 up, 8 in 2024-04-14T14:26:36.819 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:26:36 smithi069 ceph-mon[88636]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:26:36.819 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:26:36 smithi069 ceph-mon[88636]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T14:26:36.819 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:26:36 smithi069 ceph-mon[88636]: osdmap e152: 8 total, 7 up, 8 in 2024-04-14T14:26:37.049 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:26:36 smithi122 ceph-mon[67998]: Health check failed: 1 osds down (OSD_DOWN) 2024-04-14T14:26:37.050 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:26:36 smithi122 ceph-mon[67998]: Health check failed: all OSDs are running squid or later but require_osd_release < squid (OSD_UPGRADE_FINISHED) 2024-04-14T14:26:37.050 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:26:36 smithi122 ceph-mon[67998]: osdmap e151: 8 total, 7 up, 8 in 2024-04-14T14:26:37.050 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:26:36 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:26:37.050 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:26:36 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T14:26:37.050 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:26:36 smithi122 ceph-mon[67998]: osdmap e152: 8 total, 7 up, 8 in 2024-04-14T14:26:37.050 INFO:journalctl@ceph.osd.7.smithi122.stdout:Apr 14 14:26:36 smithi122 podman[92002]: 2024-04-14 14:26:36.745167159 +0000 UTC m=+0.642429081 container remove 62411cd06801aca05a5fb45000d2f206b708e645d86ebfacc76cf8547451b265 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-osd-7, architecture=x86_64, vendor=Red Hat, Inc., CEPH_POINT_RELEASE=-17.2.0, distribution-scope=public, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, vcs-type=git, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., maintainer=Guillaume Abrioux , name=centos-stream, com.redhat.component=centos-stream-container, io.openshift.tags=base centos centos-stream, GIT_BRANCH=HEAD, ceph=True, GIT_REPO=https://github.com/ceph/ceph-container.git, io.k8s.display-name=CentOS Stream 8, release=754, build-date=2022-05-03T08:36:31.336870, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, version=8, RELEASE=HEAD, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, io.openshift.expose-services=, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, io.buildah.version=1.19.8, GIT_CLEAN=True) 2024-04-14T14:26:37.344 INFO:journalctl@ceph.osd.7.smithi122.stdout:Apr 14 14:26:37 smithi122 podman[92096]: 2024-04-14 14:26:36.952017999 +0000 UTC m=+0.017767567 image pull b96c509eef841460b3271f5fdd331860da87b004524f4ed59414aa3680183b53 quay.ceph.io/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888 2024-04-14T14:26:37.345 INFO:journalctl@ceph.osd.7.smithi122.stdout:Apr 14 14:26:37 smithi122 podman[92096]: 2024-04-14 14:26:37.070136147 +0000 UTC m=+0.135885689 container create 22166f7ed71940835a5f21777452289fed0fd02e895757ae357d546c09d14937 (image=quay.ceph.io/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888, name=ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-osd-7-deactivate, org.label-schema.schema-version=1.0, GIT_CLEAN=True, RELEASE=main-a971fd5, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.build-date=20240403, ceph=True, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2) 2024-04-14T14:26:37.345 INFO:journalctl@ceph.osd.7.smithi122.stdout:Apr 14 14:26:37 smithi122 podman[92096]: 2024-04-14 14:26:37.28296351 +0000 UTC m=+0.348713060 container init 22166f7ed71940835a5f21777452289fed0fd02e895757ae357d546c09d14937 (image=quay.ceph.io/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888, name=ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-osd-7-deactivate, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, RELEASE=main-a971fd5, org.label-schema.schema-version=1.0, GIT_CLEAN=True, CEPH_POINT_RELEASE=, ceph=True, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240403, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2) 2024-04-14T14:26:37.345 INFO:journalctl@ceph.osd.7.smithi122.stdout:Apr 14 14:26:37 smithi122 podman[92096]: 2024-04-14 14:26:37.286560041 +0000 UTC m=+0.352309583 container start 22166f7ed71940835a5f21777452289fed0fd02e895757ae357d546c09d14937 (image=quay.ceph.io/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888, name=ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-osd-7-deactivate, RELEASE=main-a971fd5, ceph=True, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.build-date=20240403, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS) 2024-04-14T14:26:37.690 INFO:journalctl@ceph.osd.7.smithi122.stdout:Apr 14 14:26:37 smithi122 podman[92096]: 2024-04-14 14:26:37.374170514 +0000 UTC m=+0.439920069 container attach 22166f7ed71940835a5f21777452289fed0fd02e895757ae357d546c09d14937 (image=quay.ceph.io/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888, name=ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-osd-7-deactivate, io.buildah.version=1.33.5, GIT_CLEAN=True, RELEASE=main-a971fd5, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.build-date=20240403, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux ) 2024-04-14T14:26:37.691 INFO:journalctl@ceph.osd.7.smithi122.stdout:Apr 14 14:26:37 smithi122 podman[92096]: 2024-04-14 14:26:37.574770696 +0000 UTC m=+0.640520248 container died 22166f7ed71940835a5f21777452289fed0fd02e895757ae357d546c09d14937 (image=quay.ceph.io/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888, name=ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-osd-7-deactivate, org.label-schema.schema-version=1.0, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240403, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, RELEASE=main-a971fd5, ceph=True, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True) 2024-04-14T14:26:38.066 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:26:37 smithi069 ceph-mon[83680]: pgmap v223: 161 pgs: 12 peering, 19 stale+active+clean, 130 active+clean; 585 KiB data, 238 MiB used, 715 GiB / 715 GiB avail; 921 B/s rd, 0 op/s 2024-04-14T14:26:38.066 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:26:37 smithi069 ceph-mon[83680]: Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY) 2024-04-14T14:26:38.067 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:26:37 smithi069 ceph-mon[88636]: pgmap v223: 161 pgs: 12 peering, 19 stale+active+clean, 130 active+clean; 585 KiB data, 238 MiB used, 715 GiB / 715 GiB avail; 921 B/s rd, 0 op/s 2024-04-14T14:26:38.067 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:26:37 smithi069 ceph-mon[88636]: Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY) 2024-04-14T14:26:38.094 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:26:37 smithi122 ceph-mon[67998]: pgmap v223: 161 pgs: 12 peering, 19 stale+active+clean, 130 active+clean; 585 KiB data, 238 MiB used, 715 GiB / 715 GiB avail; 921 B/s rd, 0 op/s 2024-04-14T14:26:38.094 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:26:37 smithi122 ceph-mon[67998]: Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY) 2024-04-14T14:26:38.595 INFO:journalctl@ceph.osd.7.smithi122.stdout:Apr 14 14:26:38 smithi122 podman[92122]: 2024-04-14 14:26:38.17871471 +0000 UTC m=+0.598288350 container remove 22166f7ed71940835a5f21777452289fed0fd02e895757ae357d546c09d14937 (image=quay.ceph.io/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888, name=ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-osd-7-deactivate, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.build-date=20240403, org.label-schema.vendor=CentOS, GIT_CLEAN=True, RELEASE=main-a971fd5, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux ) 2024-04-14T14:26:38.595 INFO:journalctl@ceph.osd.7.smithi122.stdout:Apr 14 14:26:38 smithi122 systemd[1]: ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968@osd.7.service: Deactivated successfully. 2024-04-14T14:26:38.595 INFO:journalctl@ceph.osd.7.smithi122.stdout:Apr 14 14:26:38 smithi122 systemd[1]: Stopped Ceph osd.7 for dc326fc2-fa65-11ee-bc8e-c7b262605968. 2024-04-14T14:26:38.595 INFO:journalctl@ceph.osd.7.smithi122.stdout:Apr 14 14:26:38 smithi122 systemd[1]: ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968@osd.7.service: Consumed 16.115s CPU time. 2024-04-14T14:26:38.595 INFO:journalctl@ceph.osd.7.smithi122.stdout:Apr 14 14:26:38 smithi122 systemd[1]: Starting Ceph osd.7 for dc326fc2-fa65-11ee-bc8e-c7b262605968... 2024-04-14T14:26:38.862 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:26:38 smithi122 ceph-mon[67998]: from='client.25417 -' entity='client.iscsi.foo.smithi069.yuumtr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T14:26:38.863 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:26:38 smithi122 ceph-mon[67998]: pgmap v225: 161 pgs: 8 active+undersized+degraded, 12 peering, 13 stale+active+clean, 16 active+undersized, 112 active+clean; 585 KiB data, 238 MiB used, 715 GiB / 715 GiB avail; 13/627 objects degraded (2.073%) 2024-04-14T14:26:38.863 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:26:38 smithi122 ceph-mon[67998]: Health check failed: Degraded data redundancy: 13/627 objects degraded (2.073%), 8 pgs degraded (PG_DEGRADED) 2024-04-14T14:26:38.863 INFO:journalctl@ceph.osd.7.smithi122.stdout:Apr 14 14:26:38 smithi122 podman[92228]: 2024-04-14 14:26:38.515087504 +0000 UTC m=+0.018369845 image pull b96c509eef841460b3271f5fdd331860da87b004524f4ed59414aa3680183b53 quay.ceph.io/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888 2024-04-14T14:26:38.863 INFO:journalctl@ceph.osd.7.smithi122.stdout:Apr 14 14:26:38 smithi122 podman[92228]: 2024-04-14 14:26:38.620067639 +0000 UTC m=+0.123349991 container create 9ce6d835e8001d9cbeb6efc165be20f595fb48af60ae2b6cd6710bc65c7837f8 (image=quay.ceph.io/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888, name=ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-osd-7-activate, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=main-a971fd5, io.buildah.version=1.33.5, ceph=True, org.label-schema.license=GPLv2, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240403, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=) 2024-04-14T14:26:38.863 INFO:journalctl@ceph.osd.7.smithi122.stdout:Apr 14 14:26:38 smithi122 podman[92228]: 2024-04-14 14:26:38.859138189 +0000 UTC m=+0.362420496 container init 9ce6d835e8001d9cbeb6efc165be20f595fb48af60ae2b6cd6710bc65c7837f8 (image=quay.ceph.io/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888, name=ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-osd-7-activate, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, ceph=True, org.label-schema.build-date=20240403, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, RELEASE=main-a971fd5, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0) 2024-04-14T14:26:38.863 INFO:journalctl@ceph.osd.7.smithi122.stdout:Apr 14 14:26:38 smithi122 podman[92228]: 2024-04-14 14:26:38.862355836 +0000 UTC m=+0.365638162 container start 9ce6d835e8001d9cbeb6efc165be20f595fb48af60ae2b6cd6710bc65c7837f8 (image=quay.ceph.io/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888, name=ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-osd-7-activate, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.build-date=20240403, GIT_CLEAN=True, org.label-schema.vendor=CentOS, RELEASE=main-a971fd5, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux ) 2024-04-14T14:26:39.066 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:26:38 smithi069 ceph-mon[83680]: from='client.25417 -' entity='client.iscsi.foo.smithi069.yuumtr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T14:26:39.066 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:26:38 smithi069 ceph-mon[83680]: pgmap v225: 161 pgs: 8 active+undersized+degraded, 12 peering, 13 stale+active+clean, 16 active+undersized, 112 active+clean; 585 KiB data, 238 MiB used, 715 GiB / 715 GiB avail; 13/627 objects degraded (2.073%) 2024-04-14T14:26:39.066 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:26:38 smithi069 ceph-mon[83680]: Health check failed: Degraded data redundancy: 13/627 objects degraded (2.073%), 8 pgs degraded (PG_DEGRADED) 2024-04-14T14:26:39.067 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:26:38 smithi069 ceph-mon[88636]: from='client.25417 -' entity='client.iscsi.foo.smithi069.yuumtr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T14:26:39.067 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:26:38 smithi069 ceph-mon[88636]: pgmap v225: 161 pgs: 8 active+undersized+degraded, 12 peering, 13 stale+active+clean, 16 active+undersized, 112 active+clean; 585 KiB data, 238 MiB used, 715 GiB / 715 GiB avail; 13/627 objects degraded (2.073%) 2024-04-14T14:26:39.067 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:26:38 smithi069 ceph-mon[88636]: Health check failed: Degraded data redundancy: 13/627 objects degraded (2.073%), 8 pgs degraded (PG_DEGRADED) 2024-04-14T14:26:39.302 INFO:journalctl@ceph.osd.7.smithi122.stdout:Apr 14 14:26:38 smithi122 podman[92228]: 2024-04-14 14:26:38.984152274 +0000 UTC m=+0.487434589 container attach 9ce6d835e8001d9cbeb6efc165be20f595fb48af60ae2b6cd6710bc65c7837f8 (image=quay.ceph.io/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888, name=ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-osd-7-activate, ceph=True, org.label-schema.license=GPLv2, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, RELEASE=main-a971fd5, org.label-schema.build-date=20240403, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.schema-version=1.0) 2024-04-14T14:26:39.594 INFO:journalctl@ceph.osd.7.smithi122.stdout:Apr 14 14:26:39 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-osd-7-activate[92243]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-7 2024-04-14T14:26:39.595 INFO:journalctl@ceph.osd.7.smithi122.stdout:Apr 14 14:26:39 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-osd-7-activate[92243]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-7 --no-mon-config --dev /dev/mapper/vg_nvme-lv_1 2024-04-14T14:26:39.595 INFO:journalctl@ceph.osd.7.smithi122.stdout:Apr 14 14:26:39 smithi122 bash[92228]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-7 2024-04-14T14:26:39.595 INFO:journalctl@ceph.osd.7.smithi122.stdout:Apr 14 14:26:39 smithi122 bash[92228]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-7 --no-mon-config --dev /dev/mapper/vg_nvme-lv_1 2024-04-14T14:26:39.595 INFO:journalctl@ceph.osd.7.smithi122.stdout:Apr 14 14:26:39 smithi122 bash[92228]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_1 2024-04-14T14:26:39.595 INFO:journalctl@ceph.osd.7.smithi122.stdout:Apr 14 14:26:39 smithi122 bash[92228]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-0 2024-04-14T14:26:39.595 INFO:journalctl@ceph.osd.7.smithi122.stdout:Apr 14 14:26:39 smithi122 bash[92228]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-7/block 2024-04-14T14:26:39.595 INFO:journalctl@ceph.osd.7.smithi122.stdout:Apr 14 14:26:39 smithi122 bash[92228]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-7 2024-04-14T14:26:39.595 INFO:journalctl@ceph.osd.7.smithi122.stdout:Apr 14 14:26:39 smithi122 bash[92228]: --> ceph-volume raw activate successful for osd ID: 7 2024-04-14T14:26:39.595 INFO:journalctl@ceph.osd.7.smithi122.stdout:Apr 14 14:26:39 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-osd-7-activate[92243]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_1 2024-04-14T14:26:39.595 INFO:journalctl@ceph.osd.7.smithi122.stdout:Apr 14 14:26:39 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-osd-7-activate[92243]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-0 2024-04-14T14:26:39.595 INFO:journalctl@ceph.osd.7.smithi122.stdout:Apr 14 14:26:39 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-osd-7-activate[92243]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-7/block 2024-04-14T14:26:39.595 INFO:journalctl@ceph.osd.7.smithi122.stdout:Apr 14 14:26:39 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-osd-7-activate[92243]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-7 2024-04-14T14:26:39.595 INFO:journalctl@ceph.osd.7.smithi122.stdout:Apr 14 14:26:39 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-osd-7-activate[92243]: --> ceph-volume raw activate successful for osd ID: 7 2024-04-14T14:26:39.596 INFO:journalctl@ceph.osd.7.smithi122.stdout:Apr 14 14:26:39 smithi122 podman[92228]: 2024-04-14 14:26:39.325886539 +0000 UTC m=+0.829168884 container died 9ce6d835e8001d9cbeb6efc165be20f595fb48af60ae2b6cd6710bc65c7837f8 (image=quay.ceph.io/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888, name=ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-osd-7-activate, org.label-schema.schema-version=1.0, ceph=True, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, RELEASE=main-a971fd5, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.build-date=20240403, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD) 2024-04-14T14:26:40.276 INFO:journalctl@ceph.osd.7.smithi122.stdout:Apr 14 14:26:39 smithi122 podman[92364]: 2024-04-14 14:26:39.91341618 +0000 UTC m=+0.578087974 container remove 9ce6d835e8001d9cbeb6efc165be20f595fb48af60ae2b6cd6710bc65c7837f8 (image=quay.ceph.io/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888, name=ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-osd-7-activate, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.build-date=20240403, RELEASE=main-a971fd5, org.label-schema.license=GPLv2, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS) 2024-04-14T14:26:40.276 INFO:journalctl@ceph.osd.7.smithi122.stdout:Apr 14 14:26:40 smithi122 podman[92431]: 2024-04-14 14:26:40.029278807 +0000 UTC m=+0.018655917 image pull b96c509eef841460b3271f5fdd331860da87b004524f4ed59414aa3680183b53 quay.ceph.io/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888 2024-04-14T14:26:40.276 INFO:journalctl@ceph.osd.7.smithi122.stdout:Apr 14 14:26:40 smithi122 podman[92431]: 2024-04-14 14:26:40.156258569 +0000 UTC m=+0.145635678 container create 0c53c3cd2bc038aa31f168ce055728fa3b0174f20bda562f0065ca029adc0cb2 (image=quay.ceph.io/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888, name=ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-osd-7, org.label-schema.license=GPLv2, org.label-schema.build-date=20240403, RELEASE=main-a971fd5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.vendor=CentOS, ceph=True, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0) 2024-04-14T14:26:40.844 INFO:journalctl@ceph.osd.7.smithi122.stdout:Apr 14 14:26:40 smithi122 podman[92431]: 2024-04-14 14:26:40.409050163 +0000 UTC m=+0.398427274 container init 0c53c3cd2bc038aa31f168ce055728fa3b0174f20bda562f0065ca029adc0cb2 (image=quay.ceph.io/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888, name=ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-osd-7, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, ceph=True, GIT_CLEAN=True, RELEASE=main-a971fd5, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240403, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS) 2024-04-14T14:26:40.844 INFO:journalctl@ceph.osd.7.smithi122.stdout:Apr 14 14:26:40 smithi122 podman[92431]: 2024-04-14 14:26:40.412019727 +0000 UTC m=+0.401396838 container start 0c53c3cd2bc038aa31f168ce055728fa3b0174f20bda562f0065ca029adc0cb2 (image=quay.ceph.io/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888, name=ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-osd-7, io.buildah.version=1.33.5, RELEASE=main-a971fd5, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240403, ceph=True, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=) 2024-04-14T14:26:40.844 INFO:journalctl@ceph.osd.7.smithi122.stdout:Apr 14 14:26:40 smithi122 ceph-osd[92451]: -- 172.21.15.122:0/3064604062 <== mon.2 v2:172.21.15.122:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x563785b7f680 con 0x563785b48c00 2024-04-14T14:26:40.845 INFO:journalctl@ceph.osd.7.smithi122.stdout:Apr 14 14:26:40 smithi122 bash[92431]: 0c53c3cd2bc038aa31f168ce055728fa3b0174f20bda562f0065ca029adc0cb2 2024-04-14T14:26:40.845 INFO:journalctl@ceph.osd.7.smithi122.stdout:Apr 14 14:26:40 smithi122 systemd[1]: Started Ceph osd.7 for dc326fc2-fa65-11ee-bc8e-c7b262605968. 2024-04-14T14:26:41.344 INFO:journalctl@ceph.osd.7.smithi122.stdout:Apr 14 14:26:40 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-osd-7[92447]: 2024-04-14T14:26:40.951+0000 7f97d36227c0 -1 Falling back to public interface 2024-04-14T14:26:41.717 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:26:41 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:26:41.718 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:26:41 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:26:41.718 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:26:41 smithi122 ceph-mon[67998]: pgmap v226: 161 pgs: 20 active+undersized+degraded, 12 peering, 33 active+undersized, 96 active+clean; 585 KiB data, 238 MiB used, 715 GiB / 715 GiB avail; 62/627 objects degraded (9.888%) 2024-04-14T14:26:41.816 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:26:41 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:26:41.816 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:26:41 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:26:41.817 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:26:41 smithi069 ceph-mon[83680]: pgmap v226: 161 pgs: 20 active+undersized+degraded, 12 peering, 33 active+undersized, 96 active+clean; 585 KiB data, 238 MiB used, 715 GiB / 715 GiB avail; 62/627 objects degraded (9.888%) 2024-04-14T14:26:41.817 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:26:41 smithi069 ceph-mon[88636]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:26:41.817 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:26:41 smithi069 ceph-mon[88636]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:26:41.817 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:26:41 smithi069 ceph-mon[88636]: pgmap v226: 161 pgs: 20 active+undersized+degraded, 12 peering, 33 active+undersized, 96 active+clean; 585 KiB data, 238 MiB used, 715 GiB / 715 GiB avail; 62/627 objects degraded (9.888%) 2024-04-14T14:26:42.844 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:26:42 smithi122 ceph-mon[67998]: pgmap v227: 161 pgs: 25 active+undersized+degraded, 40 active+undersized, 96 active+clean; 585 KiB data, 238 MiB used, 715 GiB / 715 GiB avail; 80/627 objects degraded (12.759%) 2024-04-14T14:26:42.844 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:26:42 smithi122 ceph-mon[67998]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg peering) 2024-04-14T14:26:43.066 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:26:42 smithi069 ceph-mon[88636]: pgmap v227: 161 pgs: 25 active+undersized+degraded, 40 active+undersized, 96 active+clean; 585 KiB data, 238 MiB used, 715 GiB / 715 GiB avail; 80/627 objects degraded (12.759%) 2024-04-14T14:26:43.066 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:26:42 smithi069 ceph-mon[88636]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg peering) 2024-04-14T14:26:43.067 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:26:42 smithi069 ceph-mon[83680]: pgmap v227: 161 pgs: 25 active+undersized+degraded, 40 active+undersized, 96 active+clean; 585 KiB data, 238 MiB used, 715 GiB / 715 GiB avail; 80/627 objects degraded (12.759%) 2024-04-14T14:26:43.067 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:26:42 smithi069 ceph-mon[83680]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg peering) 2024-04-14T14:26:43.967 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:26:43 smithi122 ceph-mon[67998]: from='osd.7 [v2:172.21.15.122:6824/860929673,v1:172.21.15.122:6825/860929673]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-04-14T14:26:43.967 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:26:43 smithi122 ceph-mon[67998]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-04-14T14:26:43.967 INFO:journalctl@ceph.osd.7.smithi122.stdout:Apr 14 14:26:43 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-osd-7[92447]: 2024-04-14T14:26:43.659+0000 7f97d36227c0 -1 osd.7 0 read_superblock omap replica is missing. 2024-04-14T14:26:43.967 INFO:journalctl@ceph.osd.7.smithi122.stdout:Apr 14 14:26:43 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-osd-7[92447]: 2024-04-14T14:26:43.687+0000 7f97d36227c0 -1 osd.7 150 log_to_monitors true 2024-04-14T14:26:44.066 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:26:43 smithi069 ceph-mon[83680]: from='osd.7 [v2:172.21.15.122:6824/860929673,v1:172.21.15.122:6825/860929673]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-04-14T14:26:44.066 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:26:43 smithi069 ceph-mon[83680]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-04-14T14:26:44.067 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:26:43 smithi069 ceph-mon[88636]: from='osd.7 [v2:172.21.15.122:6824/860929673,v1:172.21.15.122:6825/860929673]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-04-14T14:26:44.067 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:26:43 smithi069 ceph-mon[88636]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-04-14T14:26:44.344 INFO:journalctl@ceph.osd.7.smithi122.stdout:Apr 14 14:26:44 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-osd-7[92447]: 2024-04-14T14:26:44.185+0000 7f97c0a72700 -1 osd.7 150 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-14T14:26:44.896 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:26:44 smithi122 ceph-mon[67998]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-04-14T14:26:44.897 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:26:44 smithi122 ceph-mon[67998]: from='osd.7 [v2:172.21.15.122:6824/860929673,v1:172.21.15.122:6825/860929673]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi122", "root=default"]}]: dispatch 2024-04-14T14:26:44.897 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:26:44 smithi122 ceph-mon[67998]: osdmap e153: 8 total, 7 up, 8 in 2024-04-14T14:26:44.897 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:26:44 smithi122 ceph-mon[67998]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi122", "root=default"]}]: dispatch 2024-04-14T14:26:44.897 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:26:44 smithi122 ceph-mon[67998]: from='osd.7 ' entity='osd.7' 2024-04-14T14:26:44.897 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:26:44 smithi122 ceph-mon[67998]: pgmap v229: 161 pgs: 25 active+undersized+degraded, 40 active+undersized, 96 active+clean; 585 KiB data, 249 MiB used, 715 GiB / 715 GiB avail; 80/627 objects degraded (12.759%) 2024-04-14T14:26:45.066 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:26:44 smithi069 ceph-mon[83680]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-04-14T14:26:45.067 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:26:44 smithi069 ceph-mon[83680]: from='osd.7 [v2:172.21.15.122:6824/860929673,v1:172.21.15.122:6825/860929673]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi122", "root=default"]}]: dispatch 2024-04-14T14:26:45.067 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:26:44 smithi069 ceph-mon[83680]: osdmap e153: 8 total, 7 up, 8 in 2024-04-14T14:26:45.067 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:26:44 smithi069 ceph-mon[83680]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi122", "root=default"]}]: dispatch 2024-04-14T14:26:45.067 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:26:44 smithi069 ceph-mon[83680]: from='osd.7 ' entity='osd.7' 2024-04-14T14:26:45.067 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:26:44 smithi069 ceph-mon[83680]: pgmap v229: 161 pgs: 25 active+undersized+degraded, 40 active+undersized, 96 active+clean; 585 KiB data, 249 MiB used, 715 GiB / 715 GiB avail; 80/627 objects degraded (12.759%) 2024-04-14T14:26:45.067 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:26:44 smithi069 ceph-mon[88636]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-04-14T14:26:45.067 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:26:44 smithi069 ceph-mon[88636]: from='osd.7 [v2:172.21.15.122:6824/860929673,v1:172.21.15.122:6825/860929673]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi122", "root=default"]}]: dispatch 2024-04-14T14:26:45.067 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:26:44 smithi069 ceph-mon[88636]: osdmap e153: 8 total, 7 up, 8 in 2024-04-14T14:26:45.067 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:26:44 smithi069 ceph-mon[88636]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi122", "root=default"]}]: dispatch 2024-04-14T14:26:45.067 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:26:44 smithi069 ceph-mon[88636]: from='osd.7 ' entity='osd.7' 2024-04-14T14:26:45.067 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:26:44 smithi069 ceph-mon[88636]: pgmap v229: 161 pgs: 25 active+undersized+degraded, 40 active+undersized, 96 active+clean; 585 KiB data, 249 MiB used, 715 GiB / 715 GiB avail; 80/627 objects degraded (12.759%) 2024-04-14T14:26:46.066 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:26:45 smithi069 ceph-mon[83680]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-04-14T14:26:46.066 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:26:45 smithi069 ceph-mon[83680]: osd.7 [v2:172.21.15.122:6824/860929673,v1:172.21.15.122:6825/860929673] boot 2024-04-14T14:26:46.067 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:26:45 smithi069 ceph-mon[83680]: osdmap e154: 8 total, 8 up, 8 in 2024-04-14T14:26:46.067 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:26:45 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-14T14:26:46.067 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:26:45 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:26:46.067 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:26:45 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:26:46.067 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:26:45 smithi069 ceph-mon[88636]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-04-14T14:26:46.067 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:26:45 smithi069 ceph-mon[88636]: osd.7 [v2:172.21.15.122:6824/860929673,v1:172.21.15.122:6825/860929673] boot 2024-04-14T14:26:46.067 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:26:45 smithi069 ceph-mon[88636]: osdmap e154: 8 total, 8 up, 8 in 2024-04-14T14:26:46.067 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:26:45 smithi069 ceph-mon[88636]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-14T14:26:46.067 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:26:45 smithi069 ceph-mon[88636]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:26:46.067 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:26:45 smithi069 ceph-mon[88636]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:26:46.094 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:26:45 smithi122 ceph-mon[67998]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-04-14T14:26:46.094 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:26:45 smithi122 ceph-mon[67998]: osd.7 [v2:172.21.15.122:6824/860929673,v1:172.21.15.122:6825/860929673] boot 2024-04-14T14:26:46.094 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:26:45 smithi122 ceph-mon[67998]: osdmap e154: 8 total, 8 up, 8 in 2024-04-14T14:26:46.094 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:26:45 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-14T14:26:46.094 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:26:45 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:26:46.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:26:45 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:26:46.816 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 14 14:26:46 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-y[63038]: ::ffff:172.21.15.122 - - [14/Apr/2024:14:26:46] "GET /metrics HTTP/1.1" 200 38361 "" "Prometheus/2.43.0" 2024-04-14T14:26:47.566 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:26:47 smithi069 ceph-mon[83680]: osdmap e155: 8 total, 8 up, 8 in 2024-04-14T14:26:47.567 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:26:47 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:26:47.567 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:26:47 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:26:47.567 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:26:47 smithi069 ceph-mon[83680]: pgmap v232: 161 pgs: 15 peering, 18 active+undersized+degraded, 32 active+undersized, 96 active+clean; 585 KiB data, 249 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 54/627 objects degraded (8.612%) 2024-04-14T14:26:47.567 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:26:47 smithi069 ceph-mon[88636]: osdmap e155: 8 total, 8 up, 8 in 2024-04-14T14:26:47.567 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:26:47 smithi069 ceph-mon[88636]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:26:47.567 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:26:47 smithi069 ceph-mon[88636]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:26:47.567 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:26:47 smithi069 ceph-mon[88636]: pgmap v232: 161 pgs: 15 peering, 18 active+undersized+degraded, 32 active+undersized, 96 active+clean; 585 KiB data, 249 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 54/627 objects degraded (8.612%) 2024-04-14T14:26:47.594 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:26:47 smithi122 ceph-mon[67998]: osdmap e155: 8 total, 8 up, 8 in 2024-04-14T14:26:47.594 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:26:47 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:26:47.594 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:26:47 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:26:47.595 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:26:47 smithi122 ceph-mon[67998]: pgmap v232: 161 pgs: 15 peering, 18 active+undersized+degraded, 32 active+undersized, 96 active+clean; 585 KiB data, 249 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 54/627 objects degraded (8.612%) 2024-04-14T14:26:48.566 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:26:48 smithi069 ceph-mon[83680]: Health check update: Degraded data redundancy: 54/627 objects degraded (8.612%), 18 pgs degraded (PG_DEGRADED) 2024-04-14T14:26:48.567 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:26:48 smithi069 ceph-mon[88636]: Health check update: Degraded data redundancy: 54/627 objects degraded (8.612%), 18 pgs degraded (PG_DEGRADED) 2024-04-14T14:26:48.594 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:26:48 smithi122 ceph-mon[67998]: Health check update: Degraded data redundancy: 54/627 objects degraded (8.612%), 18 pgs degraded (PG_DEGRADED) 2024-04-14T14:26:49.566 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:26:49 smithi069 ceph-mon[83680]: from='client.25417 -' entity='client.iscsi.foo.smithi069.yuumtr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T14:26:49.566 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:26:49 smithi069 ceph-mon[83680]: pgmap v233: 161 pgs: 15 peering, 12 active+undersized+degraded, 20 active+undersized, 114 active+clean; 585 KiB data, 249 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 44/627 objects degraded (7.018%) 2024-04-14T14:26:49.567 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:26:49 smithi069 ceph-mon[88636]: from='client.25417 -' entity='client.iscsi.foo.smithi069.yuumtr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T14:26:49.567 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:26:49 smithi069 ceph-mon[88636]: pgmap v233: 161 pgs: 15 peering, 12 active+undersized+degraded, 20 active+undersized, 114 active+clean; 585 KiB data, 249 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 44/627 objects degraded (7.018%) 2024-04-14T14:26:49.594 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:26:49 smithi122 ceph-mon[67998]: from='client.25417 -' entity='client.iscsi.foo.smithi069.yuumtr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T14:26:49.594 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:26:49 smithi122 ceph-mon[67998]: pgmap v233: 161 pgs: 15 peering, 12 active+undersized+degraded, 20 active+undersized, 114 active+clean; 585 KiB data, 249 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 44/627 objects degraded (7.018%) 2024-04-14T14:26:50.520 INFO:teuthology.orchestra.run.smithi069.stdout:true 2024-04-14T14:26:50.817 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:26:50 smithi069 ceph-mon[88636]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:26:50.817 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:26:50 smithi069 ceph-mon[88636]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:26:50.817 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:26:50 smithi069 ceph-mon[88636]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T14:26:50.817 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:26:50 smithi069 ceph-mon[88636]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-14T14:26:50.817 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:26:50 smithi069 ceph-mon[88636]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:26:50.817 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:26:50 smithi069 ceph-mon[88636]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-14T14:26:50.817 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:26:50 smithi069 ceph-mon[88636]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T14:26:50.818 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:26:50 smithi069 ceph-mon[88636]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T14:26:50.818 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:26:50 smithi069 ceph-mon[88636]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T14:26:50.818 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:26:50 smithi069 ceph-mon[88636]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T14:26:50.818 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:26:50 smithi069 ceph-mon[88636]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:26:50.818 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:26:50 smithi069 ceph-mon[88636]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.0"}]: dispatch 2024-04-14T14:26:50.818 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:26:50 smithi069 ceph-mon[88636]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.0"}]': finished 2024-04-14T14:26:50.818 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:26:50 smithi069 ceph-mon[88636]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.1"}]: dispatch 2024-04-14T14:26:50.818 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:26:50 smithi069 ceph-mon[88636]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.1"}]': finished 2024-04-14T14:26:50.818 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:26:50 smithi069 ceph-mon[88636]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.2"}]: dispatch 2024-04-14T14:26:50.818 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:26:50 smithi069 ceph-mon[88636]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.2"}]': finished 2024-04-14T14:26:50.818 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:26:50 smithi069 ceph-mon[88636]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.3"}]: dispatch 2024-04-14T14:26:50.818 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:26:50 smithi069 ceph-mon[88636]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.3"}]': finished 2024-04-14T14:26:50.818 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:26:50 smithi069 ceph-mon[88636]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.4"}]: dispatch 2024-04-14T14:26:50.818 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:26:50 smithi069 ceph-mon[88636]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.4"}]': finished 2024-04-14T14:26:50.819 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:26:50 smithi069 ceph-mon[88636]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.5"}]: dispatch 2024-04-14T14:26:50.819 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:26:50 smithi069 ceph-mon[88636]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.5"}]': finished 2024-04-14T14:26:50.819 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:26:50 smithi069 ceph-mon[88636]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.6"}]: dispatch 2024-04-14T14:26:50.819 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:26:50 smithi069 ceph-mon[88636]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.6"}]': finished 2024-04-14T14:26:50.819 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:26:50 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:26:50.819 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:26:50 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:26:50.819 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:26:50 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T14:26:50.820 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:26:50 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-14T14:26:50.820 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:26:50 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:26:50.820 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:26:50 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-14T14:26:50.820 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:26:50 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T14:26:50.820 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:26:50 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T14:26:50.820 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:26:50 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T14:26:50.820 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:26:50 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T14:26:50.820 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:26:50 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:26:50.820 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:26:50 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.0"}]: dispatch 2024-04-14T14:26:50.820 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:26:50 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.0"}]': finished 2024-04-14T14:26:50.820 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:26:50 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.1"}]: dispatch 2024-04-14T14:26:50.820 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:26:50 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.1"}]': finished 2024-04-14T14:26:50.820 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:26:50 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.2"}]: dispatch 2024-04-14T14:26:50.820 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:26:50 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.2"}]': finished 2024-04-14T14:26:50.821 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:26:50 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.3"}]: dispatch 2024-04-14T14:26:50.821 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:26:50 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.3"}]': finished 2024-04-14T14:26:50.821 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:26:50 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.4"}]: dispatch 2024-04-14T14:26:50.821 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:26:50 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.4"}]': finished 2024-04-14T14:26:50.821 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:26:50 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.5"}]: dispatch 2024-04-14T14:26:50.821 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:26:50 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.5"}]': finished 2024-04-14T14:26:50.821 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:26:50 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.6"}]: dispatch 2024-04-14T14:26:50.821 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:26:50 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.6"}]': finished 2024-04-14T14:26:50.821 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:26:50 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.7"}]: dispatch 2024-04-14T14:26:50.821 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:26:50 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.7"}]': finished 2024-04-14T14:26:50.821 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:26:50 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "osd require-osd-release", "release": "squid"}]: dispatch 2024-04-14T14:26:50.822 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:26:50 smithi069 ceph-mon[88636]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.7"}]: dispatch 2024-04-14T14:26:50.822 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:26:50 smithi069 ceph-mon[88636]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.7"}]': finished 2024-04-14T14:26:50.822 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:26:50 smithi069 ceph-mon[88636]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "osd require-osd-release", "release": "squid"}]: dispatch 2024-04-14T14:26:51.094 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:26:50 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:26:51.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:26:50 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:26:51.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:26:50 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T14:26:51.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:26:50 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-14T14:26:51.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:26:50 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:26:51.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:26:50 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-14T14:26:51.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:26:50 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T14:26:51.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:26:50 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T14:26:51.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:26:50 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T14:26:51.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:26:50 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T14:26:51.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:26:50 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:26:51.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:26:50 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.0"}]: dispatch 2024-04-14T14:26:51.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:26:50 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.0"}]': finished 2024-04-14T14:26:51.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:26:50 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.1"}]: dispatch 2024-04-14T14:26:51.096 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:26:50 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.1"}]': finished 2024-04-14T14:26:51.096 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:26:50 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.2"}]: dispatch 2024-04-14T14:26:51.096 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:26:50 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.2"}]': finished 2024-04-14T14:26:51.096 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:26:50 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.3"}]: dispatch 2024-04-14T14:26:51.096 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:26:50 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.3"}]': finished 2024-04-14T14:26:51.096 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:26:50 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.4"}]: dispatch 2024-04-14T14:26:51.096 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:26:50 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.4"}]': finished 2024-04-14T14:26:51.096 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:26:50 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.5"}]: dispatch 2024-04-14T14:26:51.096 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:26:50 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.5"}]': finished 2024-04-14T14:26:51.096 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:26:50 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.6"}]: dispatch 2024-04-14T14:26:51.096 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:26:50 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.6"}]': finished 2024-04-14T14:26:51.096 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:26:50 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.7"}]: dispatch 2024-04-14T14:26:51.096 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:26:50 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.7"}]': finished 2024-04-14T14:26:51.096 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:26:50 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "osd require-osd-release", "release": "squid"}]: dispatch 2024-04-14T14:26:51.209 INFO:teuthology.orchestra.run.smithi069.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-14T14:26:51.210 INFO:teuthology.orchestra.run.smithi069.stdout:alertmanager.a smithi069 *:9093,9094 running (8m) 2m ago 30m 26.5M - 0.25.0 c8568f914cd2 7524f106e7bd 2024-04-14T14:26:51.210 INFO:teuthology.orchestra.run.smithi069.stdout:grafana.a smithi122 *:3000 running (8m) 6s ago 30m 80.9M - 9.4.12 2bacad6d85d8 5685178e3eba 2024-04-14T14:26:51.210 INFO:teuthology.orchestra.run.smithi069.stdout:iscsi.foo.smithi069.yuumtr smithi069 running (9m) 2m ago 29m 62.9M - 3.5 e1d6a67b021e 38cc81ea8cfa 2024-04-14T14:26:51.210 INFO:teuthology.orchestra.run.smithi069.stdout:mgr.x smithi122 *:8443,9283,8765 running (9m) 6s ago 34m 454M - 19.0.0-2778-ga971fd5f b96c509eef84 e29977590b43 2024-04-14T14:26:51.210 INFO:teuthology.orchestra.run.smithi069.stdout:mgr.y smithi069 *:8443,9283,8765 running (19m) 2m ago 36m 507M - 19.0.0-2778-ga971fd5f b96c509eef84 7af3d29d0e5f 2024-04-14T14:26:51.210 INFO:teuthology.orchestra.run.smithi069.stdout:mon.a smithi069 running (6m) 2m ago 36m 54.7M 2048M 19.0.0-2778-ga971fd5f b96c509eef84 8bcd6b3d3846 2024-04-14T14:26:51.210 INFO:teuthology.orchestra.run.smithi069.stdout:mon.b smithi122 running (7m) 6s ago 35m 49.0M 2048M 19.0.0-2778-ga971fd5f b96c509eef84 05f437a1a9a1 2024-04-14T14:26:51.210 INFO:teuthology.orchestra.run.smithi069.stdout:mon.c smithi069 running (5m) 2m ago 35m 39.3M 2048M 19.0.0-2778-ga971fd5f b96c509eef84 84aa95f11d0c 2024-04-14T14:26:51.210 INFO:teuthology.orchestra.run.smithi069.stdout:node-exporter.a smithi069 *:9100 running (9m) 2m ago 31m 15.0M - 1.5.0 0da6a335fe13 0de0eb2171b9 2024-04-14T14:26:51.210 INFO:teuthology.orchestra.run.smithi069.stdout:node-exporter.b smithi122 *:9100 running (9m) 6s ago 30m 16.7M - 1.5.0 0da6a335fe13 67877bf02d8e 2024-04-14T14:26:51.210 INFO:teuthology.orchestra.run.smithi069.stdout:osd.0 smithi069 running (4m) 2m ago 34m 53.9M 2992M 19.0.0-2778-ga971fd5f b96c509eef84 a0e57e3fb494 2024-04-14T14:26:51.210 INFO:teuthology.orchestra.run.smithi069.stdout:osd.1 smithi069 running (4m) 2m ago 34m 52.9M 2992M 19.0.0-2778-ga971fd5f b96c509eef84 cf73d17919bb 2024-04-14T14:26:51.211 INFO:teuthology.orchestra.run.smithi069.stdout:osd.2 smithi069 running (3m) 2m ago 33m 41.9M 2992M 19.0.0-2778-ga971fd5f b96c509eef84 c0efba5ef451 2024-04-14T14:26:51.211 INFO:teuthology.orchestra.run.smithi069.stdout:osd.3 smithi069 running (2m) 2m ago 33m 15.1M 2992M 19.0.0-2778-ga971fd5f b96c509eef84 d13460da0a65 2024-04-14T14:26:51.211 INFO:teuthology.orchestra.run.smithi069.stdout:osd.4 smithi122 running (115s) 6s ago 32m 52.8M 3248M 19.0.0-2778-ga971fd5f b96c509eef84 7a87a024f984 2024-04-14T14:26:51.211 INFO:teuthology.orchestra.run.smithi069.stdout:osd.5 smithi122 running (81s) 6s ago 32m 52.6M 3248M 19.0.0-2778-ga971fd5f b96c509eef84 02b623d44db3 2024-04-14T14:26:51.211 INFO:teuthology.orchestra.run.smithi069.stdout:osd.6 smithi122 running (46s) 6s ago 32m 49.7M 3248M 19.0.0-2778-ga971fd5f b96c509eef84 72f85ece3e75 2024-04-14T14:26:51.211 INFO:teuthology.orchestra.run.smithi069.stdout:osd.7 smithi122 running (11s) 6s ago 31m 12.6M 3248M 19.0.0-2778-ga971fd5f b96c509eef84 0c53c3cd2bc0 2024-04-14T14:26:51.211 INFO:teuthology.orchestra.run.smithi069.stdout:prometheus.a smithi122 *:9095 running (8m) 6s ago 31m 71.1M - 2.43.0 a07b618ecd1d 5c1f41d3bbe6 2024-04-14T14:26:51.211 INFO:teuthology.orchestra.run.smithi069.stdout:rgw.foo.smithi069.wesedz smithi069 *:8000 running (29m) 2m ago 29m 79.7M - 17.2.0 e1d6a67b021e 16de491440f3 2024-04-14T14:26:51.211 INFO:teuthology.orchestra.run.smithi069.stdout:rgw.foo.smithi122.jiobyv smithi122 *:8000 running (29m) 6s ago 29m 77.7M - 17.2.0 e1d6a67b021e 6d16de445dba 2024-04-14T14:26:51.612 INFO:teuthology.orchestra.run.smithi069.stdout:{ 2024-04-14T14:26:51.612 INFO:teuthology.orchestra.run.smithi069.stdout: "mon": { 2024-04-14T14:26:51.612 INFO:teuthology.orchestra.run.smithi069.stdout: "ceph version 19.0.0-2778-ga971fd5f (a971fd5f5719038c70b817bafacd6d5400b1d888) squid (dev)": 3 2024-04-14T14:26:51.612 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-14T14:26:51.612 INFO:teuthology.orchestra.run.smithi069.stdout: "mgr": { 2024-04-14T14:26:51.612 INFO:teuthology.orchestra.run.smithi069.stdout: "ceph version 19.0.0-2778-ga971fd5f (a971fd5f5719038c70b817bafacd6d5400b1d888) squid (dev)": 2 2024-04-14T14:26:51.612 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-14T14:26:51.612 INFO:teuthology.orchestra.run.smithi069.stdout: "osd": { 2024-04-14T14:26:51.612 INFO:teuthology.orchestra.run.smithi069.stdout: "ceph version 19.0.0-2778-ga971fd5f (a971fd5f5719038c70b817bafacd6d5400b1d888) squid (dev)": 8 2024-04-14T14:26:51.613 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-14T14:26:51.613 INFO:teuthology.orchestra.run.smithi069.stdout: "rgw": { 2024-04-14T14:26:51.613 INFO:teuthology.orchestra.run.smithi069.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-04-14T14:26:51.613 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-14T14:26:51.613 INFO:teuthology.orchestra.run.smithi069.stdout: "overall": { 2024-04-14T14:26:51.613 INFO:teuthology.orchestra.run.smithi069.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2, 2024-04-14T14:26:51.613 INFO:teuthology.orchestra.run.smithi069.stdout: "ceph version 19.0.0-2778-ga971fd5f (a971fd5f5719038c70b817bafacd6d5400b1d888) squid (dev)": 13 2024-04-14T14:26:51.613 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-14T14:26:51.613 INFO:teuthology.orchestra.run.smithi069.stdout:} 2024-04-14T14:26:51.951 INFO:teuthology.orchestra.run.smithi069.stdout:{ 2024-04-14T14:26:51.951 INFO:teuthology.orchestra.run.smithi069.stdout: "target_image": null, 2024-04-14T14:26:51.951 INFO:teuthology.orchestra.run.smithi069.stdout: "in_progress": false, 2024-04-14T14:26:51.951 INFO:teuthology.orchestra.run.smithi069.stdout: "which": "", 2024-04-14T14:26:51.951 INFO:teuthology.orchestra.run.smithi069.stdout: "services_complete": [], 2024-04-14T14:26:51.951 INFO:teuthology.orchestra.run.smithi069.stdout: "progress": null, 2024-04-14T14:26:51.952 INFO:teuthology.orchestra.run.smithi069.stdout: "message": "", 2024-04-14T14:26:51.952 INFO:teuthology.orchestra.run.smithi069.stdout: "is_paused": false 2024-04-14T14:26:51.952 INFO:teuthology.orchestra.run.smithi069.stdout:} 2024-04-14T14:26:51.952 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:26:51 smithi069 ceph-mon[83680]: Upgrade: Setting container_image for all osd 2024-04-14T14:26:51.952 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:26:51 smithi069 ceph-mon[83680]: Upgrade: Setting require_osd_release to 19 squid 2024-04-14T14:26:51.953 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:26:51 smithi069 ceph-mon[83680]: from='client.54514 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T14:26:51.953 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:26:51 smithi069 ceph-mon[83680]: pgmap v234: 161 pgs: 11 peering, 150 active+clean; 585 KiB data, 249 MiB used, 715 GiB / 715 GiB avail; 743 B/s rd, 0 op/s 2024-04-14T14:26:51.953 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:26:51 smithi069 ceph-mon[83680]: Health check cleared: OSD_UPGRADE_FINISHED (was: all OSDs are running squid or later but require_osd_release < squid) 2024-04-14T14:26:51.953 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:26:51 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd='[{"prefix": "osd require-osd-release", "release": "squid"}]': finished 2024-04-14T14:26:51.953 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:26:51 smithi069 ceph-mon[83680]: osdmap e156: 8 total, 8 up, 8 in 2024-04-14T14:26:51.953 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:26:51 smithi069 ceph-mon[83680]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 44/627 objects degraded (7.018%), 12 pgs degraded) 2024-04-14T14:26:51.953 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:26:51 smithi069 ceph-mon[83680]: Cluster is now healthy 2024-04-14T14:26:51.953 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:26:51 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T14:26:51.953 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:26:51 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:26:51.954 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:26:51 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T14:26:51.954 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:26:51 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:26:51.954 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:26:51 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T14:26:51.954 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:26:51 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T14:26:51.954 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:26:51 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:26:51.954 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:26:51 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T14:26:51.954 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:26:51 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:26:51.954 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:26:51 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T14:26:51.955 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:26:51 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:26:51.955 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:26:51 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T14:26:51.955 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:26:51 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T14:26:51.955 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:26:51 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T14:26:51.955 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:26:51 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T14:26:51.955 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:26:51 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T14:26:51.955 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:26:51 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T14:26:51.955 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:26:51 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]: dispatch 2024-04-14T14:26:51.956 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:26:51 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]': finished 2024-04-14T14:26:51.956 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:26:51 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-14T14:26:51.956 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:26:51 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon"}]': finished 2024-04-14T14:26:51.956 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:26:51 smithi069 ceph-mon[88636]: Upgrade: Setting container_image for all osd 2024-04-14T14:26:51.956 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:26:51 smithi069 ceph-mon[88636]: Upgrade: Setting require_osd_release to 19 squid 2024-04-14T14:26:51.956 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:26:51 smithi069 ceph-mon[88636]: from='client.54514 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T14:26:51.956 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:26:51 smithi069 ceph-mon[88636]: pgmap v234: 161 pgs: 11 peering, 150 active+clean; 585 KiB data, 249 MiB used, 715 GiB / 715 GiB avail; 743 B/s rd, 0 op/s 2024-04-14T14:26:51.956 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:26:51 smithi069 ceph-mon[88636]: Health check cleared: OSD_UPGRADE_FINISHED (was: all OSDs are running squid or later but require_osd_release < squid) 2024-04-14T14:26:51.956 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:26:51 smithi069 ceph-mon[88636]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd='[{"prefix": "osd require-osd-release", "release": "squid"}]': finished 2024-04-14T14:26:51.956 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:26:51 smithi069 ceph-mon[88636]: osdmap e156: 8 total, 8 up, 8 in 2024-04-14T14:26:51.956 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:26:51 smithi069 ceph-mon[88636]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 44/627 objects degraded (7.018%), 12 pgs degraded) 2024-04-14T14:26:51.957 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:26:51 smithi069 ceph-mon[88636]: Cluster is now healthy 2024-04-14T14:26:51.957 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:26:51 smithi069 ceph-mon[88636]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T14:26:51.957 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:26:51 smithi069 ceph-mon[88636]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:26:51.957 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:26:51 smithi069 ceph-mon[88636]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T14:26:51.957 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:26:51 smithi069 ceph-mon[88636]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:26:51.957 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:26:51 smithi069 ceph-mon[88636]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T14:26:51.957 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:26:51 smithi069 ceph-mon[88636]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T14:26:51.957 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:26:51 smithi069 ceph-mon[88636]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:26:51.957 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:26:51 smithi069 ceph-mon[88636]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T14:26:51.957 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:26:51 smithi069 ceph-mon[88636]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:26:51.957 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:26:51 smithi069 ceph-mon[88636]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T14:26:51.957 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:26:51 smithi069 ceph-mon[88636]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:26:51.957 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:26:51 smithi069 ceph-mon[88636]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T14:26:51.957 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:26:51 smithi069 ceph-mon[88636]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T14:26:51.958 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:26:51 smithi069 ceph-mon[88636]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T14:26:51.958 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:26:51 smithi069 ceph-mon[88636]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T14:26:51.958 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:26:51 smithi069 ceph-mon[88636]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T14:26:51.958 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:26:51 smithi069 ceph-mon[88636]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T14:26:51.958 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:26:51 smithi069 ceph-mon[88636]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]: dispatch 2024-04-14T14:26:51.958 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:26:51 smithi069 ceph-mon[88636]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]': finished 2024-04-14T14:26:51.958 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:26:51 smithi069 ceph-mon[88636]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-14T14:26:51.958 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:26:51 smithi069 ceph-mon[88636]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon"}]': finished 2024-04-14T14:26:51.958 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:26:51 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]: dispatch 2024-04-14T14:26:51.959 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:26:51 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]': finished 2024-04-14T14:26:51.959 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:26:51 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd"}]: dispatch 2024-04-14T14:26:51.959 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:26:51 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd"}]': finished 2024-04-14T14:26:51.959 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:26:51 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mds"}]: dispatch 2024-04-14T14:26:51.959 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:26:51 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mds"}]': finished 2024-04-14T14:26:51.959 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:26:51 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw"}]: dispatch 2024-04-14T14:26:51.959 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:26:51 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"}]: dispatch 2024-04-14T14:26:51.959 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:26:51 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"}]': finished 2024-04-14T14:26:51.959 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:26:51 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-14T14:26:51.959 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:26:51 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter"}]: dispatch 2024-04-14T14:26:51.959 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:26:51 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter"}]': finished 2024-04-14T14:26:51.959 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:26:51 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.iscsi"}]: dispatch 2024-04-14T14:26:51.959 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:26:51 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nfs"}]: dispatch 2024-04-14T14:26:51.960 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:26:51 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.nfs"}]': finished 2024-04-14T14:26:51.960 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:26:51 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]: dispatch 2024-04-14T14:26:51.960 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:26:51 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]': finished 2024-04-14T14:26:51.960 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:26:51 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-14T14:26:51.960 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:26:51 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-14T14:26:51.960 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:26:51 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-14T14:26:51.960 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:26:51 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-14T14:26:51.960 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:26:51 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-14T14:26:51.960 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:26:51 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-14T14:26:51.960 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:26:51 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]: dispatch 2024-04-14T14:26:51.960 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:26:51 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd='[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]': finished 2024-04-14T14:26:51.960 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:26:51 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T14:26:51.960 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:26:51 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-14T14:26:51.960 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:26:51 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:26:51.961 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:26:51 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T14:26:51.961 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:26:51 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-14T14:26:51.961 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:26:51 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:26:51.961 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:26:51 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:26:51.961 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:26:51 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T14:26:51.961 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:26:51 smithi069 ceph-mon[83680]: from='client.? 172.21.15.69:0/55174389' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T14:26:51.961 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:26:51 smithi069 ceph-mon[88636]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]: dispatch 2024-04-14T14:26:51.961 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:26:51 smithi069 ceph-mon[88636]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]': finished 2024-04-14T14:26:51.961 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:26:51 smithi069 ceph-mon[88636]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd"}]: dispatch 2024-04-14T14:26:51.961 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:26:51 smithi069 ceph-mon[88636]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd"}]': finished 2024-04-14T14:26:51.962 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:26:51 smithi069 ceph-mon[88636]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mds"}]: dispatch 2024-04-14T14:26:51.962 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:26:51 smithi069 ceph-mon[88636]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mds"}]': finished 2024-04-14T14:26:51.962 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:26:51 smithi069 ceph-mon[88636]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw"}]: dispatch 2024-04-14T14:26:51.962 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:26:51 smithi069 ceph-mon[88636]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"}]: dispatch 2024-04-14T14:26:51.962 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:26:51 smithi069 ceph-mon[88636]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"}]': finished 2024-04-14T14:26:51.962 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:26:51 smithi069 ceph-mon[88636]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-14T14:26:51.962 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:26:51 smithi069 ceph-mon[88636]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter"}]: dispatch 2024-04-14T14:26:51.962 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:26:51 smithi069 ceph-mon[88636]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter"}]': finished 2024-04-14T14:26:51.962 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:26:51 smithi069 ceph-mon[88636]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.iscsi"}]: dispatch 2024-04-14T14:26:51.962 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:26:51 smithi069 ceph-mon[88636]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nfs"}]: dispatch 2024-04-14T14:26:51.962 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:26:51 smithi069 ceph-mon[88636]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.nfs"}]': finished 2024-04-14T14:26:51.962 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:26:51 smithi069 ceph-mon[88636]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]: dispatch 2024-04-14T14:26:51.962 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:26:51 smithi069 ceph-mon[88636]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]': finished 2024-04-14T14:26:51.962 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:26:51 smithi069 ceph-mon[88636]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-14T14:26:51.963 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:26:51 smithi069 ceph-mon[88636]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-14T14:26:51.963 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:26:51 smithi069 ceph-mon[88636]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-14T14:26:51.963 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:26:51 smithi069 ceph-mon[88636]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-14T14:26:51.963 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:26:51 smithi069 ceph-mon[88636]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-14T14:26:51.963 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:26:51 smithi069 ceph-mon[88636]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-14T14:26:51.963 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:26:51 smithi069 ceph-mon[88636]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]: dispatch 2024-04-14T14:26:51.963 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:26:51 smithi069 ceph-mon[88636]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd='[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]': finished 2024-04-14T14:26:51.963 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:26:51 smithi069 ceph-mon[88636]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T14:26:51.963 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:26:51 smithi069 ceph-mon[88636]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-14T14:26:51.963 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:26:51 smithi069 ceph-mon[88636]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:26:51.963 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:26:51 smithi069 ceph-mon[88636]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T14:26:51.963 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:26:51 smithi069 ceph-mon[88636]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-14T14:26:51.963 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:26:51 smithi069 ceph-mon[88636]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:26:51.963 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:26:51 smithi069 ceph-mon[88636]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:26:51.963 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:26:51 smithi069 ceph-mon[88636]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T14:26:51.964 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:26:51 smithi069 ceph-mon[88636]: from='client.? 172.21.15.69:0/55174389' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T14:26:52.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:26:51 smithi122 ceph-mon[67998]: Upgrade: Setting container_image for all osd 2024-04-14T14:26:52.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:26:51 smithi122 ceph-mon[67998]: Upgrade: Setting require_osd_release to 19 squid 2024-04-14T14:26:52.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:26:51 smithi122 ceph-mon[67998]: from='client.54514 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T14:26:52.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:26:51 smithi122 ceph-mon[67998]: pgmap v234: 161 pgs: 11 peering, 150 active+clean; 585 KiB data, 249 MiB used, 715 GiB / 715 GiB avail; 743 B/s rd, 0 op/s 2024-04-14T14:26:52.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:26:51 smithi122 ceph-mon[67998]: Health check cleared: OSD_UPGRADE_FINISHED (was: all OSDs are running squid or later but require_osd_release < squid) 2024-04-14T14:26:52.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:26:51 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd='[{"prefix": "osd require-osd-release", "release": "squid"}]': finished 2024-04-14T14:26:52.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:26:51 smithi122 ceph-mon[67998]: osdmap e156: 8 total, 8 up, 8 in 2024-04-14T14:26:52.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:26:51 smithi122 ceph-mon[67998]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 44/627 objects degraded (7.018%), 12 pgs degraded) 2024-04-14T14:26:52.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:26:51 smithi122 ceph-mon[67998]: Cluster is now healthy 2024-04-14T14:26:52.096 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:26:51 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T14:26:52.096 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:26:51 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:26:52.096 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:26:51 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T14:26:52.096 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:26:51 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:26:52.096 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:26:51 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T14:26:52.096 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:26:51 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T14:26:52.096 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:26:51 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:26:52.096 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:26:51 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T14:26:52.096 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:26:51 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:26:52.096 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:26:51 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T14:26:52.096 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:26:51 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:26:52.096 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:26:51 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T14:26:52.096 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:26:51 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T14:26:52.096 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:26:51 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T14:26:52.096 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:26:51 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T14:26:52.097 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:26:51 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T14:26:52.097 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:26:51 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T14:26:52.097 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:26:51 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]: dispatch 2024-04-14T14:26:52.097 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:26:51 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]': finished 2024-04-14T14:26:52.097 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:26:51 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-14T14:26:52.097 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:26:51 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon"}]': finished 2024-04-14T14:26:52.097 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:26:51 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]: dispatch 2024-04-14T14:26:52.097 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:26:51 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]': finished 2024-04-14T14:26:52.097 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:26:51 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd"}]: dispatch 2024-04-14T14:26:52.097 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:26:51 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd"}]': finished 2024-04-14T14:26:52.097 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:26:51 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mds"}]: dispatch 2024-04-14T14:26:52.097 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:26:51 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mds"}]': finished 2024-04-14T14:26:52.097 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:26:51 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw"}]: dispatch 2024-04-14T14:26:52.097 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:26:51 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"}]: dispatch 2024-04-14T14:26:52.097 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:26:51 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"}]': finished 2024-04-14T14:26:52.098 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:26:51 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-14T14:26:52.098 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:26:51 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter"}]: dispatch 2024-04-14T14:26:52.098 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:26:51 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter"}]': finished 2024-04-14T14:26:52.098 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:26:51 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.iscsi"}]: dispatch 2024-04-14T14:26:52.098 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:26:51 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nfs"}]: dispatch 2024-04-14T14:26:52.098 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:26:51 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.nfs"}]': finished 2024-04-14T14:26:52.098 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:26:51 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]: dispatch 2024-04-14T14:26:52.098 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:26:51 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]': finished 2024-04-14T14:26:52.098 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:26:51 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-14T14:26:52.098 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:26:51 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-14T14:26:52.098 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:26:51 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-14T14:26:52.098 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:26:51 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-14T14:26:52.098 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:26:51 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-14T14:26:52.099 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:26:51 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-14T14:26:52.099 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:26:51 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]: dispatch 2024-04-14T14:26:52.099 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:26:51 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd='[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]': finished 2024-04-14T14:26:52.099 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:26:51 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T14:26:52.099 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:26:51 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-14T14:26:52.099 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:26:51 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:26:52.099 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:26:51 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T14:26:52.099 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:26:51 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-14T14:26:52.099 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:26:51 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:26:52.099 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:26:51 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:26:52.099 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:26:51 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T14:26:52.099 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:26:51 smithi122 ceph-mon[67998]: from='client.? 172.21.15.69:0/55174389' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T14:26:53.067 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:26:52 smithi069 ceph-mon[83680]: from='client.54520 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T14:26:53.067 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:26:52 smithi069 ceph-mon[83680]: Upgrade: Setting container_image for all mds 2024-04-14T14:26:53.067 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:26:52 smithi069 ceph-mon[83680]: Upgrade: Setting container_image for all rbd-mirror 2024-04-14T14:26:53.067 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:26:52 smithi069 ceph-mon[83680]: Upgrade: Setting container_image for all ceph-exporter 2024-04-14T14:26:53.067 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:26:52 smithi069 ceph-mon[83680]: Upgrade: Setting container_image for all nfs 2024-04-14T14:26:53.068 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:26:52 smithi069 ceph-mon[83680]: Upgrade: Setting container_image for all nvmeof 2024-04-14T14:26:53.068 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:26:52 smithi069 ceph-mon[83680]: Upgrade: Finalizing container_image settings 2024-04-14T14:26:53.068 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:26:52 smithi069 ceph-mon[83680]: Upgrade: Complete! 2024-04-14T14:26:53.068 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:26:52 smithi069 ceph-mon[83680]: from='client.44589 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T14:26:53.068 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:26:52 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:26:53.068 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:26:52 smithi069 ceph-mon[88636]: from='client.54520 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T14:26:53.069 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:26:52 smithi069 ceph-mon[88636]: Upgrade: Setting container_image for all mds 2024-04-14T14:26:53.069 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:26:52 smithi069 ceph-mon[88636]: Upgrade: Setting container_image for all rbd-mirror 2024-04-14T14:26:53.069 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:26:52 smithi069 ceph-mon[88636]: Upgrade: Setting container_image for all ceph-exporter 2024-04-14T14:26:53.069 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:26:52 smithi069 ceph-mon[88636]: Upgrade: Setting container_image for all nfs 2024-04-14T14:26:53.069 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:26:52 smithi069 ceph-mon[88636]: Upgrade: Setting container_image for all nvmeof 2024-04-14T14:26:53.069 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:26:52 smithi069 ceph-mon[88636]: Upgrade: Finalizing container_image settings 2024-04-14T14:26:53.069 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:26:52 smithi069 ceph-mon[88636]: Upgrade: Complete! 2024-04-14T14:26:53.070 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:26:52 smithi069 ceph-mon[88636]: from='client.44589 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T14:26:53.070 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:26:52 smithi069 ceph-mon[88636]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:26:53.094 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:26:52 smithi122 ceph-mon[67998]: from='client.54520 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T14:26:53.094 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:26:52 smithi122 ceph-mon[67998]: Upgrade: Setting container_image for all mds 2024-04-14T14:26:53.094 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:26:52 smithi122 ceph-mon[67998]: Upgrade: Setting container_image for all rbd-mirror 2024-04-14T14:26:53.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:26:52 smithi122 ceph-mon[67998]: Upgrade: Setting container_image for all ceph-exporter 2024-04-14T14:26:53.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:26:52 smithi122 ceph-mon[67998]: Upgrade: Setting container_image for all nfs 2024-04-14T14:26:53.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:26:52 smithi122 ceph-mon[67998]: Upgrade: Setting container_image for all nvmeof 2024-04-14T14:26:53.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:26:52 smithi122 ceph-mon[67998]: Upgrade: Finalizing container_image settings 2024-04-14T14:26:53.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:26:52 smithi122 ceph-mon[67998]: Upgrade: Complete! 2024-04-14T14:26:53.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:26:52 smithi122 ceph-mon[67998]: from='client.44589 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T14:26:53.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:26:52 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:26:54.066 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:26:53 smithi069 ceph-mon[83680]: from='client.44601 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T14:26:54.066 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:26:53 smithi069 ceph-mon[83680]: pgmap v236: 161 pgs: 161 active+clean; 585 KiB data, 249 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-04-14T14:26:54.067 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:26:53 smithi069 ceph-mon[88636]: from='client.44601 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T14:26:54.067 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:26:53 smithi069 ceph-mon[88636]: pgmap v236: 161 pgs: 161 active+clean; 585 KiB data, 249 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-04-14T14:26:54.094 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:26:53 smithi122 ceph-mon[67998]: from='client.44601 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T14:26:54.094 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:26:53 smithi122 ceph-mon[67998]: pgmap v236: 161 pgs: 161 active+clean; 585 KiB data, 249 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-04-14T14:26:55.067 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:26:54 smithi069 ceph-mon[83680]: pgmap v237: 161 pgs: 161 active+clean; 585 KiB data, 249 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:26:55.067 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:26:54 smithi069 ceph-mon[88636]: pgmap v237: 161 pgs: 161 active+clean; 585 KiB data, 249 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:26:55.094 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:26:54 smithi122 ceph-mon[67998]: pgmap v237: 161 pgs: 161 active+clean; 585 KiB data, 249 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:26:56.816 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 14 14:26:56 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-y[63038]: ::ffff:172.21.15.122 - - [14/Apr/2024:14:26:56] "GET /metrics HTTP/1.1" 200 38381 "" "Prometheus/2.43.0" 2024-04-14T14:26:58.066 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:26:57 smithi069 ceph-mon[83680]: pgmap v238: 161 pgs: 161 active+clean; 585 KiB data, 250 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-14T14:26:58.066 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:26:57 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:26:58.067 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:26:57 smithi069 ceph-mon[88636]: pgmap v238: 161 pgs: 161 active+clean; 585 KiB data, 250 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-14T14:26:58.067 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:26:57 smithi069 ceph-mon[88636]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:26:58.094 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:26:57 smithi122 ceph-mon[67998]: pgmap v238: 161 pgs: 161 active+clean; 585 KiB data, 250 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-14T14:26:58.094 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:26:57 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:27:00.066 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:26:59 smithi069 ceph-mon[83680]: from='client.25417 -' entity='client.iscsi.foo.smithi069.yuumtr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T14:27:00.067 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:26:59 smithi069 ceph-mon[83680]: pgmap v239: 161 pgs: 161 active+clean; 585 KiB data, 250 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-14T14:27:00.067 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:26:59 smithi069 ceph-mon[88636]: from='client.25417 -' entity='client.iscsi.foo.smithi069.yuumtr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T14:27:00.067 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:26:59 smithi069 ceph-mon[88636]: pgmap v239: 161 pgs: 161 active+clean; 585 KiB data, 250 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-14T14:27:00.094 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:26:59 smithi122 ceph-mon[67998]: from='client.25417 -' entity='client.iscsi.foo.smithi069.yuumtr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T14:27:00.094 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:26:59 smithi122 ceph-mon[67998]: pgmap v239: 161 pgs: 161 active+clean; 585 KiB data, 250 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-14T14:27:02.066 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:27:01 smithi069 ceph-mon[83680]: pgmap v240: 161 pgs: 161 active+clean; 585 KiB data, 250 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-14T14:27:02.067 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:27:01 smithi069 ceph-mon[88636]: pgmap v240: 161 pgs: 161 active+clean; 585 KiB data, 250 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-14T14:27:02.094 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:27:01 smithi122 ceph-mon[67998]: pgmap v240: 161 pgs: 161 active+clean; 585 KiB data, 250 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-14T14:27:04.066 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:27:03 smithi069 ceph-mon[83680]: pgmap v241: 161 pgs: 161 active+clean; 585 KiB data, 250 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-04-14T14:27:04.067 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:27:03 smithi069 ceph-mon[88636]: pgmap v241: 161 pgs: 161 active+clean; 585 KiB data, 250 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-04-14T14:27:04.094 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:27:03 smithi122 ceph-mon[67998]: pgmap v241: 161 pgs: 161 active+clean; 585 KiB data, 250 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-04-14T14:27:06.066 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:27:05 smithi069 ceph-mon[83680]: pgmap v242: 161 pgs: 161 active+clean; 585 KiB data, 250 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:27:06.067 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:27:05 smithi069 ceph-mon[88636]: pgmap v242: 161 pgs: 161 active+clean; 585 KiB data, 250 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:27:06.094 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:27:05 smithi122 ceph-mon[67998]: pgmap v242: 161 pgs: 161 active+clean; 585 KiB data, 250 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:27:06.816 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:27:06 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T14:27:06.816 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 14 14:27:06 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-y[63038]: ::ffff:172.21.15.122 - - [14/Apr/2024:14:27:06] "GET /metrics HTTP/1.1" 200 38381 "" "Prometheus/2.43.0" 2024-04-14T14:27:06.817 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:27:06 smithi069 ceph-mon[88636]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T14:27:07.094 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:27:06 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T14:27:08.066 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:27:07 smithi069 ceph-mon[83680]: pgmap v243: 161 pgs: 161 active+clean; 585 KiB data, 250 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:27:08.067 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:27:07 smithi069 ceph-mon[88636]: pgmap v243: 161 pgs: 161 active+clean; 585 KiB data, 250 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:27:08.094 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:27:07 smithi122 ceph-mon[67998]: pgmap v243: 161 pgs: 161 active+clean; 585 KiB data, 250 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:27:09.066 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:27:08 smithi069 ceph-mon[83680]: from='client.25417 -' entity='client.iscsi.foo.smithi069.yuumtr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T14:27:09.066 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:27:08 smithi069 ceph-mon[83680]: pgmap v244: 161 pgs: 161 active+clean; 585 KiB data, 250 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:27:09.067 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:27:08 smithi069 ceph-mon[88636]: from='client.25417 -' entity='client.iscsi.foo.smithi069.yuumtr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T14:27:09.067 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:27:08 smithi069 ceph-mon[88636]: pgmap v244: 161 pgs: 161 active+clean; 585 KiB data, 250 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:27:09.094 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:27:08 smithi122 ceph-mon[67998]: from='client.25417 -' entity='client.iscsi.foo.smithi069.yuumtr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T14:27:09.094 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:27:08 smithi122 ceph-mon[67998]: pgmap v244: 161 pgs: 161 active+clean; 585 KiB data, 250 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:27:12.066 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:27:11 smithi069 ceph-mon[83680]: pgmap v245: 161 pgs: 161 active+clean; 585 KiB data, 250 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:27:12.067 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:27:11 smithi069 ceph-mon[88636]: pgmap v245: 161 pgs: 161 active+clean; 585 KiB data, 250 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:27:12.094 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:27:11 smithi122 ceph-mon[67998]: pgmap v245: 161 pgs: 161 active+clean; 585 KiB data, 250 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:27:14.066 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:27:13 smithi069 ceph-mon[83680]: pgmap v246: 161 pgs: 161 active+clean; 585 KiB data, 250 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:27:14.066 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:27:13 smithi069 ceph-mon[88636]: pgmap v246: 161 pgs: 161 active+clean; 585 KiB data, 250 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:27:14.094 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:27:13 smithi122 ceph-mon[67998]: pgmap v246: 161 pgs: 161 active+clean; 585 KiB data, 250 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:27:16.066 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:27:15 smithi069 ceph-mon[83680]: pgmap v247: 161 pgs: 161 active+clean; 585 KiB data, 250 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:27:16.066 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:27:15 smithi069 ceph-mon[88636]: pgmap v247: 161 pgs: 161 active+clean; 585 KiB data, 250 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:27:16.094 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:27:15 smithi122 ceph-mon[67998]: pgmap v247: 161 pgs: 161 active+clean; 585 KiB data, 250 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:27:16.816 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 14 14:27:16 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-y[63038]: ::ffff:172.21.15.122 - - [14/Apr/2024:14:27:16] "GET /metrics HTTP/1.1" 200 38377 "" "Prometheus/2.43.0" 2024-04-14T14:27:18.066 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:27:17 smithi069 ceph-mon[83680]: pgmap v248: 161 pgs: 161 active+clean; 585 KiB data, 250 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:27:18.067 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:27:17 smithi069 ceph-mon[88636]: pgmap v248: 161 pgs: 161 active+clean; 585 KiB data, 250 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:27:18.094 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:27:17 smithi122 ceph-mon[67998]: pgmap v248: 161 pgs: 161 active+clean; 585 KiB data, 250 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:27:20.066 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:27:19 smithi069 ceph-mon[83680]: from='client.25417 -' entity='client.iscsi.foo.smithi069.yuumtr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T14:27:20.066 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:27:19 smithi069 ceph-mon[83680]: pgmap v249: 161 pgs: 161 active+clean; 585 KiB data, 250 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:27:20.067 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:27:19 smithi069 ceph-mon[88636]: from='client.25417 -' entity='client.iscsi.foo.smithi069.yuumtr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T14:27:20.067 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:27:19 smithi069 ceph-mon[88636]: pgmap v249: 161 pgs: 161 active+clean; 585 KiB data, 250 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:27:20.094 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:27:19 smithi122 ceph-mon[67998]: from='client.25417 -' entity='client.iscsi.foo.smithi069.yuumtr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T14:27:20.094 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:27:19 smithi122 ceph-mon[67998]: pgmap v249: 161 pgs: 161 active+clean; 585 KiB data, 250 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:27:22.066 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:27:21 smithi069 ceph-mon[83680]: pgmap v250: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 29 op/s 2024-04-14T14:27:22.066 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:27:21 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T14:27:22.067 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:27:21 smithi069 ceph-mon[88636]: pgmap v250: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 29 op/s 2024-04-14T14:27:22.067 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:27:21 smithi069 ceph-mon[88636]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T14:27:22.094 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:27:21 smithi122 ceph-mon[67998]: pgmap v250: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 29 op/s 2024-04-14T14:27:22.094 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:27:21 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T14:27:23.040 DEBUG:teuthology.orchestra.run.smithi069:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid dc326fc2-fa65-11ee-bc8e-c7b262605968 -e sha1=a971fd5f5719038c70b817bafacd6d5400b1d888 -- bash -c 'ceph orch ps' 2024-04-14T14:27:23.946 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:27:23 smithi069 ceph-mon[83680]: from='client.44607 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T14:27:23.947 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:27:23 smithi069 ceph-mon[83680]: pgmap v251: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail; 34 KiB/s rd, 0 B/s wr, 55 op/s 2024-04-14T14:27:23.947 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:27:23 smithi069 ceph-mon[88636]: from='client.44607 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T14:27:23.947 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:27:23 smithi069 ceph-mon[88636]: pgmap v251: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail; 34 KiB/s rd, 0 B/s wr, 55 op/s 2024-04-14T14:27:24.094 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:27:23 smithi122 ceph-mon[67998]: from='client.44607 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T14:27:24.094 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:27:23 smithi122 ceph-mon[67998]: pgmap v251: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail; 34 KiB/s rd, 0 B/s wr, 55 op/s 2024-04-14T14:27:24.928 INFO:teuthology.orchestra.run.smithi069.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-14T14:27:24.928 INFO:teuthology.orchestra.run.smithi069.stdout:alertmanager.a smithi069 *:9093,9094 running (9m) 2m ago 30m 26.5M - 0.25.0 c8568f914cd2 7524f106e7bd 2024-04-14T14:27:24.928 INFO:teuthology.orchestra.run.smithi069.stdout:grafana.a smithi122 *:3000 running (8m) 39s ago 30m 80.9M - 9.4.12 2bacad6d85d8 5685178e3eba 2024-04-14T14:27:24.928 INFO:teuthology.orchestra.run.smithi069.stdout:iscsi.foo.smithi069.yuumtr smithi069 running (10m) 2m ago 30m 62.9M - 3.5 e1d6a67b021e 38cc81ea8cfa 2024-04-14T14:27:24.928 INFO:teuthology.orchestra.run.smithi069.stdout:mgr.x smithi122 *:8443,9283,8765 running (10m) 39s ago 35m 454M - 19.0.0-2778-ga971fd5f b96c509eef84 e29977590b43 2024-04-14T14:27:24.928 INFO:teuthology.orchestra.run.smithi069.stdout:mgr.y smithi069 *:8443,9283,8765 running (19m) 2m ago 36m 507M - 19.0.0-2778-ga971fd5f b96c509eef84 7af3d29d0e5f 2024-04-14T14:27:24.928 INFO:teuthology.orchestra.run.smithi069.stdout:mon.a smithi069 running (6m) 2m ago 37m 54.7M 2048M 19.0.0-2778-ga971fd5f b96c509eef84 8bcd6b3d3846 2024-04-14T14:27:24.928 INFO:teuthology.orchestra.run.smithi069.stdout:mon.b smithi122 running (7m) 39s ago 35m 49.0M 2048M 19.0.0-2778-ga971fd5f b96c509eef84 05f437a1a9a1 2024-04-14T14:27:24.929 INFO:teuthology.orchestra.run.smithi069.stdout:mon.c smithi069 running (6m) 2m ago 35m 39.3M 2048M 19.0.0-2778-ga971fd5f b96c509eef84 84aa95f11d0c 2024-04-14T14:27:24.929 INFO:teuthology.orchestra.run.smithi069.stdout:node-exporter.a smithi069 *:9100 running (9m) 2m ago 31m 15.0M - 1.5.0 0da6a335fe13 0de0eb2171b9 2024-04-14T14:27:24.929 INFO:teuthology.orchestra.run.smithi069.stdout:node-exporter.b smithi122 *:9100 running (9m) 39s ago 31m 16.7M - 1.5.0 0da6a335fe13 67877bf02d8e 2024-04-14T14:27:24.929 INFO:teuthology.orchestra.run.smithi069.stdout:osd.0 smithi069 running (5m) 2m ago 35m 53.9M 2992M 19.0.0-2778-ga971fd5f b96c509eef84 a0e57e3fb494 2024-04-14T14:27:24.929 INFO:teuthology.orchestra.run.smithi069.stdout:osd.1 smithi069 running (4m) 2m ago 34m 52.9M 2992M 19.0.0-2778-ga971fd5f b96c509eef84 cf73d17919bb 2024-04-14T14:27:24.929 INFO:teuthology.orchestra.run.smithi069.stdout:osd.2 smithi069 running (3m) 2m ago 34m 41.9M 2992M 19.0.0-2778-ga971fd5f b96c509eef84 c0efba5ef451 2024-04-14T14:27:24.929 INFO:teuthology.orchestra.run.smithi069.stdout:osd.3 smithi069 running (3m) 2m ago 33m 15.1M 2992M 19.0.0-2778-ga971fd5f b96c509eef84 d13460da0a65 2024-04-14T14:27:24.929 INFO:teuthology.orchestra.run.smithi069.stdout:osd.4 smithi122 running (2m) 39s ago 33m 52.8M 3248M 19.0.0-2778-ga971fd5f b96c509eef84 7a87a024f984 2024-04-14T14:27:24.929 INFO:teuthology.orchestra.run.smithi069.stdout:osd.5 smithi122 running (114s) 39s ago 33m 52.6M 3248M 19.0.0-2778-ga971fd5f b96c509eef84 02b623d44db3 2024-04-14T14:27:24.929 INFO:teuthology.orchestra.run.smithi069.stdout:osd.6 smithi122 running (79s) 39s ago 32m 49.7M 3248M 19.0.0-2778-ga971fd5f b96c509eef84 72f85ece3e75 2024-04-14T14:27:24.929 INFO:teuthology.orchestra.run.smithi069.stdout:osd.7 smithi122 running (44s) 39s ago 32m 12.6M 3248M 19.0.0-2778-ga971fd5f b96c509eef84 0c53c3cd2bc0 2024-04-14T14:27:24.929 INFO:teuthology.orchestra.run.smithi069.stdout:prometheus.a smithi122 *:9095 running (9m) 39s ago 31m 71.1M - 2.43.0 a07b618ecd1d 5c1f41d3bbe6 2024-04-14T14:27:24.929 INFO:teuthology.orchestra.run.smithi069.stdout:rgw.foo.smithi069.wesedz smithi069 *:8000 running (30m) 2m ago 30m 79.7M - 17.2.0 e1d6a67b021e 16de491440f3 2024-04-14T14:27:24.930 INFO:teuthology.orchestra.run.smithi069.stdout:rgw.foo.smithi122.jiobyv smithi122 *:8000 running (30m) 39s ago 30m 77.7M - 17.2.0 e1d6a67b021e 6d16de445dba 2024-04-14T14:27:25.587 DEBUG:teuthology.orchestra.run.smithi069:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid dc326fc2-fa65-11ee-bc8e-c7b262605968 -e sha1=a971fd5f5719038c70b817bafacd6d5400b1d888 -- bash -c 'ceph versions | jq -e '"'"'.osd | length == 1'"'"'' 2024-04-14T14:27:25.816 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:27:25 smithi069 ceph-mon[83680]: pgmap v252: 161 pgs: 161 active+clean; 585 KiB data, 258 MiB used, 715 GiB / 715 GiB avail; 55 KiB/s rd, 0 B/s wr, 91 op/s 2024-04-14T14:27:25.816 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:27:25 smithi069 ceph-mon[88636]: pgmap v252: 161 pgs: 161 active+clean; 585 KiB data, 258 MiB used, 715 GiB / 715 GiB avail; 55 KiB/s rd, 0 B/s wr, 91 op/s 2024-04-14T14:27:26.094 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:27:25 smithi122 ceph-mon[67998]: pgmap v252: 161 pgs: 161 active+clean; 585 KiB data, 258 MiB used, 715 GiB / 715 GiB avail; 55 KiB/s rd, 0 B/s wr, 91 op/s 2024-04-14T14:27:26.762 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:27:26 smithi069 ceph-mon[83680]: from='client.54550 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T14:27:26.762 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:27:26 smithi069 ceph-mon[83680]: pgmap v253: 161 pgs: 161 active+clean; 585 KiB data, 258 MiB used, 715 GiB / 715 GiB avail; 72 KiB/s rd, 0 B/s wr, 119 op/s 2024-04-14T14:27:26.762 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 14 14:27:26 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-y[63038]: ::ffff:172.21.15.122 - - [14/Apr/2024:14:27:26] "GET /metrics HTTP/1.1" 200 38376 "" "Prometheus/2.43.0" 2024-04-14T14:27:26.762 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:27:26 smithi069 ceph-mon[88636]: from='client.54550 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T14:27:26.762 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:27:26 smithi069 ceph-mon[88636]: pgmap v253: 161 pgs: 161 active+clean; 585 KiB data, 258 MiB used, 715 GiB / 715 GiB avail; 72 KiB/s rd, 0 B/s wr, 119 op/s 2024-04-14T14:27:27.094 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:27:26 smithi122 ceph-mon[67998]: from='client.54550 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T14:27:27.094 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:27:26 smithi122 ceph-mon[67998]: pgmap v253: 161 pgs: 161 active+clean; 585 KiB data, 258 MiB used, 715 GiB / 715 GiB avail; 72 KiB/s rd, 0 B/s wr, 119 op/s 2024-04-14T14:27:27.501 INFO:teuthology.orchestra.run.smithi069.stdout:true 2024-04-14T14:27:27.816 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:27:27 smithi069 ceph-mon[83680]: from='client.? 172.21.15.69:0/3243709175' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T14:27:27.816 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:27:27 smithi069 ceph-mon[88636]: from='client.? 172.21.15.69:0/3243709175' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T14:27:28.094 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:27:27 smithi122 ceph-mon[67998]: from='client.? 172.21.15.69:0/3243709175' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T14:27:28.136 DEBUG:teuthology.orchestra.run.smithi069:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid dc326fc2-fa65-11ee-bc8e-c7b262605968 -e sha1=a971fd5f5719038c70b817bafacd6d5400b1d888 -- bash -c 'ceph versions | jq -e '"'"'.osd | keys'"'"' | grep $sha1' 2024-04-14T14:27:29.066 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:27:28 smithi069 ceph-mon[88636]: from='client.25417 -' entity='client.iscsi.foo.smithi069.yuumtr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T14:27:29.066 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:27:28 smithi069 ceph-mon[88636]: pgmap v254: 161 pgs: 161 active+clean; 585 KiB data, 258 MiB used, 715 GiB / 715 GiB avail; 73 KiB/s rd, 0 B/s wr, 120 op/s 2024-04-14T14:27:29.067 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:27:28 smithi069 ceph-mon[83680]: from='client.25417 -' entity='client.iscsi.foo.smithi069.yuumtr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T14:27:29.067 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:27:28 smithi069 ceph-mon[83680]: pgmap v254: 161 pgs: 161 active+clean; 585 KiB data, 258 MiB used, 715 GiB / 715 GiB avail; 73 KiB/s rd, 0 B/s wr, 120 op/s 2024-04-14T14:27:29.094 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:27:28 smithi122 ceph-mon[67998]: from='client.25417 -' entity='client.iscsi.foo.smithi069.yuumtr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T14:27:29.094 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:27:28 smithi122 ceph-mon[67998]: pgmap v254: 161 pgs: 161 active+clean; 585 KiB data, 258 MiB used, 715 GiB / 715 GiB avail; 73 KiB/s rd, 0 B/s wr, 120 op/s 2024-04-14T14:27:30.243 INFO:teuthology.orchestra.run.smithi069.stdout: "ceph version 19.0.0-2778-ga971fd5f (a971fd5f5719038c70b817bafacd6d5400b1d888) squid (dev)" 2024-04-14T14:27:30.542 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:27:30 smithi069 ceph-mon[83680]: from='client.? 172.21.15.69:0/3386018250' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T14:27:30.542 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:27:30 smithi069 ceph-mon[88636]: from='client.? 172.21.15.69:0/3386018250' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T14:27:30.594 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:27:30 smithi122 ceph-mon[67998]: from='client.? 172.21.15.69:0/3386018250' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T14:27:30.932 DEBUG:teuthology.orchestra.run.smithi069:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid dc326fc2-fa65-11ee-bc8e-c7b262605968 -e sha1=a971fd5f5719038c70b817bafacd6d5400b1d888 -- bash -c 'ceph orch upgrade status' 2024-04-14T14:27:31.512 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:27:31 smithi069 ceph-mon[83680]: pgmap v255: 161 pgs: 161 active+clean; 585 KiB data, 258 MiB used, 715 GiB / 715 GiB avail; 72 KiB/s rd, 0 B/s wr, 119 op/s 2024-04-14T14:27:31.512 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:27:31 smithi069 ceph-mon[88636]: pgmap v255: 161 pgs: 161 active+clean; 585 KiB data, 258 MiB used, 715 GiB / 715 GiB avail; 72 KiB/s rd, 0 B/s wr, 119 op/s 2024-04-14T14:27:31.594 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:27:31 smithi122 ceph-mon[67998]: pgmap v255: 161 pgs: 161 active+clean; 585 KiB data, 258 MiB used, 715 GiB / 715 GiB avail; 72 KiB/s rd, 0 B/s wr, 119 op/s 2024-04-14T14:27:33.091 INFO:teuthology.orchestra.run.smithi069.stdout:{ 2024-04-14T14:27:33.091 INFO:teuthology.orchestra.run.smithi069.stdout: "target_image": null, 2024-04-14T14:27:33.091 INFO:teuthology.orchestra.run.smithi069.stdout: "in_progress": false, 2024-04-14T14:27:33.091 INFO:teuthology.orchestra.run.smithi069.stdout: "which": "", 2024-04-14T14:27:33.092 INFO:teuthology.orchestra.run.smithi069.stdout: "services_complete": [], 2024-04-14T14:27:33.092 INFO:teuthology.orchestra.run.smithi069.stdout: "progress": null, 2024-04-14T14:27:33.092 INFO:teuthology.orchestra.run.smithi069.stdout: "message": "", 2024-04-14T14:27:33.092 INFO:teuthology.orchestra.run.smithi069.stdout: "is_paused": false 2024-04-14T14:27:33.092 INFO:teuthology.orchestra.run.smithi069.stdout:} 2024-04-14T14:27:33.799 DEBUG:teuthology.orchestra.run.smithi069:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid dc326fc2-fa65-11ee-bc8e-c7b262605968 -e sha1=a971fd5f5719038c70b817bafacd6d5400b1d888 -- bash -c 'ceph health detail' 2024-04-14T14:27:34.066 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:27:33 smithi069 ceph-mon[83680]: pgmap v256: 161 pgs: 161 active+clean; 585 KiB data, 258 MiB used, 715 GiB / 715 GiB avail; 56 KiB/s rd, 0 B/s wr, 92 op/s 2024-04-14T14:27:34.066 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:27:33 smithi069 ceph-mon[88636]: pgmap v256: 161 pgs: 161 active+clean; 585 KiB data, 258 MiB used, 715 GiB / 715 GiB avail; 56 KiB/s rd, 0 B/s wr, 92 op/s 2024-04-14T14:27:34.094 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:27:33 smithi122 ceph-mon[67998]: pgmap v256: 161 pgs: 161 active+clean; 585 KiB data, 258 MiB used, 715 GiB / 715 GiB avail; 56 KiB/s rd, 0 B/s wr, 92 op/s 2024-04-14T14:27:34.844 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:27:34 smithi122 ceph-mon[67998]: from='client.44631 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T14:27:35.047 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:27:34 smithi069 ceph-mon[83680]: from='client.44631 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T14:27:35.047 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:27:34 smithi069 ceph-mon[88636]: from='client.44631 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T14:27:35.766 INFO:teuthology.orchestra.run.smithi069.stdout:HEALTH_OK 2024-04-14T14:27:36.066 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:27:35 smithi069 ceph-mon[83680]: pgmap v257: 161 pgs: 161 active+clean; 585 KiB data, 258 MiB used, 715 GiB / 715 GiB avail; 40 KiB/s rd, 0 B/s wr, 65 op/s 2024-04-14T14:27:36.066 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:27:35 smithi069 ceph-mon[88636]: pgmap v257: 161 pgs: 161 active+clean; 585 KiB data, 258 MiB used, 715 GiB / 715 GiB avail; 40 KiB/s rd, 0 B/s wr, 65 op/s 2024-04-14T14:27:36.094 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:27:35 smithi122 ceph-mon[67998]: pgmap v257: 161 pgs: 161 active+clean; 585 KiB data, 258 MiB used, 715 GiB / 715 GiB avail; 40 KiB/s rd, 0 B/s wr, 65 op/s 2024-04-14T14:27:36.478 DEBUG:teuthology.orchestra.run.smithi069:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid dc326fc2-fa65-11ee-bc8e-c7b262605968 -e sha1=a971fd5f5719038c70b817bafacd6d5400b1d888 -- bash -c 'ceph orch upgrade start --image quay.ceph.io/ceph-ci/ceph:$sha1 --services rgw.foo' 2024-04-14T14:27:36.785 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 14 14:27:36 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-y[63038]: ::ffff:172.21.15.122 - - [14/Apr/2024:14:27:36] "GET /metrics HTTP/1.1" 200 38376 "" "Prometheus/2.43.0" 2024-04-14T14:27:36.786 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:27:36 smithi069 ceph-mon[83680]: from='client.? 172.21.15.69:0/2260182394' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-14T14:27:36.786 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:27:36 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T14:27:36.786 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:27:36 smithi069 ceph-mon[88636]: from='client.? 172.21.15.69:0/2260182394' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-14T14:27:36.786 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:27:36 smithi069 ceph-mon[88636]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T14:27:37.094 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:27:36 smithi122 ceph-mon[67998]: from='client.? 172.21.15.69:0/2260182394' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-14T14:27:37.094 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:27:36 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T14:27:37.804 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:27:37 smithi069 ceph-mon[83680]: pgmap v258: 161 pgs: 161 active+clean; 585 KiB data, 258 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 29 op/s 2024-04-14T14:27:37.804 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:27:37 smithi069 ceph-mon[88636]: pgmap v258: 161 pgs: 161 active+clean; 585 KiB data, 258 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 29 op/s 2024-04-14T14:27:38.094 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:27:37 smithi122 ceph-mon[67998]: pgmap v258: 161 pgs: 161 active+clean; 585 KiB data, 258 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 29 op/s 2024-04-14T14:27:39.066 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:27:38 smithi069 ceph-mon[83680]: from='client.25417 -' entity='client.iscsi.foo.smithi069.yuumtr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T14:27:39.066 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:27:38 smithi069 ceph-mon[83680]: from='client.54580 -' entity='client.admin' cmd=[{"prefix": "orch upgrade start", "image": "quay.ceph.io/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888", "services": "rgw.foo", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T14:27:39.066 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:27:38 smithi069 ceph-mon[83680]: pgmap v259: 161 pgs: 161 active+clean; 585 KiB data, 258 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:27:39.067 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:27:38 smithi069 ceph-mon[88636]: from='client.25417 -' entity='client.iscsi.foo.smithi069.yuumtr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T14:27:39.067 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:27:38 smithi069 ceph-mon[88636]: from='client.54580 -' entity='client.admin' cmd=[{"prefix": "orch upgrade start", "image": "quay.ceph.io/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888", "services": "rgw.foo", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T14:27:39.067 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:27:38 smithi069 ceph-mon[88636]: pgmap v259: 161 pgs: 161 active+clean; 585 KiB data, 258 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:27:39.094 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:27:38 smithi122 ceph-mon[67998]: from='client.25417 -' entity='client.iscsi.foo.smithi069.yuumtr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T14:27:39.094 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:27:38 smithi122 ceph-mon[67998]: from='client.54580 -' entity='client.admin' cmd=[{"prefix": "orch upgrade start", "image": "quay.ceph.io/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888", "services": "rgw.foo", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T14:27:39.094 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:27:38 smithi122 ceph-mon[67998]: pgmap v259: 161 pgs: 161 active+clean; 585 KiB data, 258 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:27:40.049 INFO:teuthology.orchestra.run.smithi069.stdout:Initiating upgrade to quay.ceph.io/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888 2024-04-14T14:27:40.725 DEBUG:teuthology.orchestra.run.smithi069:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid dc326fc2-fa65-11ee-bc8e-c7b262605968 -e sha1=a971fd5f5719038c70b817bafacd6d5400b1d888 -- bash -c 'while ceph orch upgrade status | jq '"'"'.in_progress'"'"' | grep true && ! ceph orch upgrade status | jq '"'"'.message'"'"' | grep Error ; do ceph orch ps ; ceph versions ; ceph orch upgrade status ; sleep 30 ; done' 2024-04-14T14:27:41.298 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:27:41 smithi069 ceph-mon[83680]: Upgrade: Started with target quay.ceph.io/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888 2024-04-14T14:27:41.298 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:27:41 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:27:41.298 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:27:41 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T14:27:41.298 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:27:41 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-14T14:27:41.299 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:27:41 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:27:41.299 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:27:41 smithi069 ceph-mon[83680]: Upgrade: First pull of quay.ceph.io/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888 2024-04-14T14:27:41.299 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:27:41 smithi069 ceph-mon[83680]: pgmap v260: 161 pgs: 161 active+clean; 585 KiB data, 258 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:27:41.299 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:27:41 smithi069 ceph-mon[88636]: Upgrade: Started with target quay.ceph.io/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888 2024-04-14T14:27:41.299 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:27:41 smithi069 ceph-mon[88636]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:27:41.299 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:27:41 smithi069 ceph-mon[88636]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T14:27:41.299 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:27:41 smithi069 ceph-mon[88636]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-14T14:27:41.299 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:27:41 smithi069 ceph-mon[88636]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:27:41.299 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:27:41 smithi069 ceph-mon[88636]: Upgrade: First pull of quay.ceph.io/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888 2024-04-14T14:27:41.299 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:27:41 smithi069 ceph-mon[88636]: pgmap v260: 161 pgs: 161 active+clean; 585 KiB data, 258 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:27:41.344 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:27:41 smithi122 ceph-mon[67998]: Upgrade: Started with target quay.ceph.io/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888 2024-04-14T14:27:41.344 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:27:41 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:27:41.344 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:27:41 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T14:27:41.344 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:27:41 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-14T14:27:41.344 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:27:41 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:27:41.345 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:27:41 smithi122 ceph-mon[67998]: Upgrade: First pull of quay.ceph.io/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888 2024-04-14T14:27:41.345 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:27:41 smithi122 ceph-mon[67998]: pgmap v260: 161 pgs: 161 active+clean; 585 KiB data, 258 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:27:43.642 INFO:teuthology.orchestra.run.smithi069.stdout:true 2024-04-14T14:27:43.817 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:27:43 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:27:43.817 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:27:43 smithi069 ceph-mon[83680]: Upgrade: Target is version 19.0.0-2778-ga971fd5f (squid) 2024-04-14T14:27:43.817 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:27:43 smithi069 ceph-mon[83680]: Upgrade: Target container is quay.ceph.io/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888, digests ['quay.ceph.io/ceph-ci/ceph@sha256:02f0a191d4d693952654c64f93cc9c0152602c0915251b3c94c2c4e32855c608'] 2024-04-14T14:27:43.817 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:27:43 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-14T14:27:43.817 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:27:43 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T14:27:43.817 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:27:43 smithi069 ceph-mon[83680]: Upgrade: Setting container_image for all mgr 2024-04-14T14:27:43.817 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:27:43 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:27:43.817 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:27:43 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T14:27:43.817 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:27:43 smithi069 ceph-mon[83680]: Upgrade: Setting container_image for all mon 2024-04-14T14:27:43.817 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:27:43 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:27:43.817 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:27:43 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T14:27:43.818 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:27:43 smithi069 ceph-mon[83680]: Upgrade: Setting container_image for all crash 2024-04-14T14:27:43.818 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:27:43 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:27:43.818 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:27:43 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T14:27:43.818 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:27:43 smithi069 ceph-mon[83680]: Upgrade: Setting container_image for all osd 2024-04-14T14:27:43.818 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:27:43 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:27:43.818 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:27:43 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T14:27:43.818 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:27:43 smithi069 ceph-mon[83680]: Upgrade: Setting container_image for all mds 2024-04-14T14:27:43.818 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:27:43 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:27:43.818 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:27:43 smithi069 ceph-mon[83680]: pgmap v261: 161 pgs: 161 active+clean; 585 KiB data, 258 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:27:43.818 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:27:43 smithi069 ceph-mon[88636]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:27:43.818 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:27:43 smithi069 ceph-mon[88636]: Upgrade: Target is version 19.0.0-2778-ga971fd5f (squid) 2024-04-14T14:27:43.818 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:27:43 smithi069 ceph-mon[88636]: Upgrade: Target container is quay.ceph.io/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888, digests ['quay.ceph.io/ceph-ci/ceph@sha256:02f0a191d4d693952654c64f93cc9c0152602c0915251b3c94c2c4e32855c608'] 2024-04-14T14:27:43.819 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:27:43 smithi069 ceph-mon[88636]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-14T14:27:43.819 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:27:43 smithi069 ceph-mon[88636]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T14:27:43.819 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:27:43 smithi069 ceph-mon[88636]: Upgrade: Setting container_image for all mgr 2024-04-14T14:27:43.819 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:27:43 smithi069 ceph-mon[88636]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:27:43.819 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:27:43 smithi069 ceph-mon[88636]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T14:27:43.819 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:27:43 smithi069 ceph-mon[88636]: Upgrade: Setting container_image for all mon 2024-04-14T14:27:43.819 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:27:43 smithi069 ceph-mon[88636]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:27:43.819 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:27:43 smithi069 ceph-mon[88636]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T14:27:43.819 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:27:43 smithi069 ceph-mon[88636]: Upgrade: Setting container_image for all crash 2024-04-14T14:27:43.819 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:27:43 smithi069 ceph-mon[88636]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:27:43.819 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:27:43 smithi069 ceph-mon[88636]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T14:27:43.819 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:27:43 smithi069 ceph-mon[88636]: Upgrade: Setting container_image for all osd 2024-04-14T14:27:43.819 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:27:43 smithi069 ceph-mon[88636]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:27:43.819 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:27:43 smithi069 ceph-mon[88636]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T14:27:43.819 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:27:43 smithi069 ceph-mon[88636]: Upgrade: Setting container_image for all mds 2024-04-14T14:27:43.820 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:27:43 smithi069 ceph-mon[88636]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:27:43.820 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:27:43 smithi069 ceph-mon[88636]: pgmap v261: 161 pgs: 161 active+clean; 585 KiB data, 258 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:27:43.844 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:27:43 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:27:43.844 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:27:43 smithi122 ceph-mon[67998]: Upgrade: Target is version 19.0.0-2778-ga971fd5f (squid) 2024-04-14T14:27:43.844 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:27:43 smithi122 ceph-mon[67998]: Upgrade: Target container is quay.ceph.io/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888, digests ['quay.ceph.io/ceph-ci/ceph@sha256:02f0a191d4d693952654c64f93cc9c0152602c0915251b3c94c2c4e32855c608'] 2024-04-14T14:27:43.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:27:43 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-14T14:27:43.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:27:43 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T14:27:43.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:27:43 smithi122 ceph-mon[67998]: Upgrade: Setting container_image for all mgr 2024-04-14T14:27:43.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:27:43 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:27:43.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:27:43 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T14:27:43.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:27:43 smithi122 ceph-mon[67998]: Upgrade: Setting container_image for all mon 2024-04-14T14:27:43.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:27:43 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:27:43.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:27:43 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T14:27:43.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:27:43 smithi122 ceph-mon[67998]: Upgrade: Setting container_image for all crash 2024-04-14T14:27:43.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:27:43 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:27:43.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:27:43 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T14:27:43.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:27:43 smithi122 ceph-mon[67998]: Upgrade: Setting container_image for all osd 2024-04-14T14:27:43.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:27:43 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:27:43.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:27:43 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T14:27:43.846 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:27:43 smithi122 ceph-mon[67998]: Upgrade: Setting container_image for all mds 2024-04-14T14:27:43.846 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:27:43 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:27:43.846 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:27:43 smithi122 ceph-mon[67998]: pgmap v261: 161 pgs: 161 active+clean; 585 KiB data, 258 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:27:44.301 INFO:teuthology.orchestra.run.smithi069.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-14T14:27:44.301 INFO:teuthology.orchestra.run.smithi069.stdout:alertmanager.a smithi069 *:9093,9094 running (9m) 3m ago 31m 26.5M - 0.25.0 c8568f914cd2 7524f106e7bd 2024-04-14T14:27:44.302 INFO:teuthology.orchestra.run.smithi069.stdout:grafana.a smithi122 *:3000 running (9m) 59s ago 30m 80.9M - 9.4.12 2bacad6d85d8 5685178e3eba 2024-04-14T14:27:44.302 INFO:teuthology.orchestra.run.smithi069.stdout:iscsi.foo.smithi069.yuumtr smithi069 running (10m) 3m ago 30m 62.9M - 3.5 e1d6a67b021e 38cc81ea8cfa 2024-04-14T14:27:44.302 INFO:teuthology.orchestra.run.smithi069.stdout:mgr.x smithi122 *:8443,9283,8765 running (10m) 59s ago 35m 454M - 19.0.0-2778-ga971fd5f b96c509eef84 e29977590b43 2024-04-14T14:27:44.302 INFO:teuthology.orchestra.run.smithi069.stdout:mgr.y smithi069 *:8443,9283,8765 running (20m) 3m ago 37m 507M - 19.0.0-2778-ga971fd5f b96c509eef84 7af3d29d0e5f 2024-04-14T14:27:44.302 INFO:teuthology.orchestra.run.smithi069.stdout:mon.a smithi069 running (7m) 3m ago 37m 54.7M 2048M 19.0.0-2778-ga971fd5f b96c509eef84 8bcd6b3d3846 2024-04-14T14:27:44.302 INFO:teuthology.orchestra.run.smithi069.stdout:mon.b smithi122 running (8m) 59s ago 35m 49.0M 2048M 19.0.0-2778-ga971fd5f b96c509eef84 05f437a1a9a1 2024-04-14T14:27:44.302 INFO:teuthology.orchestra.run.smithi069.stdout:mon.c smithi069 running (6m) 3m ago 36m 39.3M 2048M 19.0.0-2778-ga971fd5f b96c509eef84 84aa95f11d0c 2024-04-14T14:27:44.302 INFO:teuthology.orchestra.run.smithi069.stdout:node-exporter.a smithi069 *:9100 running (10m) 3m ago 31m 15.0M - 1.5.0 0da6a335fe13 0de0eb2171b9 2024-04-14T14:27:44.302 INFO:teuthology.orchestra.run.smithi069.stdout:node-exporter.b smithi122 *:9100 running (10m) 59s ago 31m 16.7M - 1.5.0 0da6a335fe13 67877bf02d8e 2024-04-14T14:27:44.302 INFO:teuthology.orchestra.run.smithi069.stdout:osd.0 smithi069 running (5m) 3m ago 35m 53.9M 2992M 19.0.0-2778-ga971fd5f b96c509eef84 a0e57e3fb494 2024-04-14T14:27:44.302 INFO:teuthology.orchestra.run.smithi069.stdout:osd.1 smithi069 running (5m) 3m ago 34m 52.9M 2992M 19.0.0-2778-ga971fd5f b96c509eef84 cf73d17919bb 2024-04-14T14:27:44.302 INFO:teuthology.orchestra.run.smithi069.stdout:osd.2 smithi069 running (4m) 3m ago 34m 41.9M 2992M 19.0.0-2778-ga971fd5f b96c509eef84 c0efba5ef451 2024-04-14T14:27:44.302 INFO:teuthology.orchestra.run.smithi069.stdout:osd.3 smithi069 running (3m) 3m ago 34m 15.1M 2992M 19.0.0-2778-ga971fd5f b96c509eef84 d13460da0a65 2024-04-14T14:27:44.302 INFO:teuthology.orchestra.run.smithi069.stdout:osd.4 smithi122 running (2m) 59s ago 33m 52.8M 3248M 19.0.0-2778-ga971fd5f b96c509eef84 7a87a024f984 2024-04-14T14:27:44.303 INFO:teuthology.orchestra.run.smithi069.stdout:osd.5 smithi122 running (2m) 59s ago 33m 52.6M 3248M 19.0.0-2778-ga971fd5f b96c509eef84 02b623d44db3 2024-04-14T14:27:44.303 INFO:teuthology.orchestra.run.smithi069.stdout:osd.6 smithi122 running (99s) 59s ago 33m 49.7M 3248M 19.0.0-2778-ga971fd5f b96c509eef84 72f85ece3e75 2024-04-14T14:27:44.303 INFO:teuthology.orchestra.run.smithi069.stdout:osd.7 smithi122 running (64s) 59s ago 32m 12.6M 3248M 19.0.0-2778-ga971fd5f b96c509eef84 0c53c3cd2bc0 2024-04-14T14:27:44.303 INFO:teuthology.orchestra.run.smithi069.stdout:prometheus.a smithi122 *:9095 running (9m) 59s ago 32m 71.1M - 2.43.0 a07b618ecd1d 5c1f41d3bbe6 2024-04-14T14:27:44.303 INFO:teuthology.orchestra.run.smithi069.stdout:rgw.foo.smithi069.wesedz smithi069 *:8000 running (30m) 3m ago 30m 79.7M - 17.2.0 e1d6a67b021e 16de491440f3 2024-04-14T14:27:44.303 INFO:teuthology.orchestra.run.smithi069.stdout:rgw.foo.smithi122.jiobyv smithi122 *:8000 running (30m) 59s ago 30m 77.7M - 17.2.0 e1d6a67b021e 6d16de445dba 2024-04-14T14:27:44.727 INFO:teuthology.orchestra.run.smithi069.stdout:{ 2024-04-14T14:27:44.728 INFO:teuthology.orchestra.run.smithi069.stdout: "mon": { 2024-04-14T14:27:44.728 INFO:teuthology.orchestra.run.smithi069.stdout: "ceph version 19.0.0-2778-ga971fd5f (a971fd5f5719038c70b817bafacd6d5400b1d888) squid (dev)": 3 2024-04-14T14:27:44.728 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-14T14:27:44.728 INFO:teuthology.orchestra.run.smithi069.stdout: "mgr": { 2024-04-14T14:27:44.728 INFO:teuthology.orchestra.run.smithi069.stdout: "ceph version 19.0.0-2778-ga971fd5f (a971fd5f5719038c70b817bafacd6d5400b1d888) squid (dev)": 2 2024-04-14T14:27:44.728 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-14T14:27:44.728 INFO:teuthology.orchestra.run.smithi069.stdout: "osd": { 2024-04-14T14:27:44.728 INFO:teuthology.orchestra.run.smithi069.stdout: "ceph version 19.0.0-2778-ga971fd5f (a971fd5f5719038c70b817bafacd6d5400b1d888) squid (dev)": 8 2024-04-14T14:27:44.728 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-14T14:27:44.728 INFO:teuthology.orchestra.run.smithi069.stdout: "rgw": { 2024-04-14T14:27:44.728 INFO:teuthology.orchestra.run.smithi069.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-04-14T14:27:44.728 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-14T14:27:44.728 INFO:teuthology.orchestra.run.smithi069.stdout: "overall": { 2024-04-14T14:27:44.729 INFO:teuthology.orchestra.run.smithi069.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2, 2024-04-14T14:27:44.729 INFO:teuthology.orchestra.run.smithi069.stdout: "ceph version 19.0.0-2778-ga971fd5f (a971fd5f5719038c70b817bafacd6d5400b1d888) squid (dev)": 13 2024-04-14T14:27:44.729 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-14T14:27:44.729 INFO:teuthology.orchestra.run.smithi069.stdout:} 2024-04-14T14:27:44.816 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:27:44 smithi069 ceph-mon[88636]: from='client.44649 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T14:27:44.816 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:27:44 smithi069 ceph-mon[88636]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:27:44.816 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:27:44 smithi069 ceph-mon[88636]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi069.wesedz", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-04-14T14:27:44.817 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:27:44 smithi069 ceph-mon[88636]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T14:27:44.817 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:27:44 smithi069 ceph-mon[83680]: from='client.44649 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T14:27:44.817 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:27:44 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:27:44.817 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:27:44 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi069.wesedz", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-04-14T14:27:44.817 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:27:44 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T14:27:45.081 INFO:teuthology.orchestra.run.smithi069.stdout:{ 2024-04-14T14:27:45.081 INFO:teuthology.orchestra.run.smithi069.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888", 2024-04-14T14:27:45.081 INFO:teuthology.orchestra.run.smithi069.stdout: "in_progress": true, 2024-04-14T14:27:45.081 INFO:teuthology.orchestra.run.smithi069.stdout: "which": "Upgrading daemons in service(s) rgw.foo", 2024-04-14T14:27:45.081 INFO:teuthology.orchestra.run.smithi069.stdout: "services_complete": [], 2024-04-14T14:27:45.081 INFO:teuthology.orchestra.run.smithi069.stdout: "progress": "0/2 daemons upgraded", 2024-04-14T14:27:45.081 INFO:teuthology.orchestra.run.smithi069.stdout: "message": "Currently upgrading rgw daemons", 2024-04-14T14:27:45.081 INFO:teuthology.orchestra.run.smithi069.stdout: "is_paused": false 2024-04-14T14:27:45.081 INFO:teuthology.orchestra.run.smithi069.stdout:} 2024-04-14T14:27:45.094 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:27:44 smithi122 ceph-mon[67998]: from='client.44649 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T14:27:45.094 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:27:44 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:27:45.094 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:27:44 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi069.wesedz", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-04-14T14:27:45.094 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:27:44 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T14:27:45.816 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:27:45 smithi069 ceph-mon[83680]: from='client.44655 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T14:27:45.816 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:27:45 smithi069 ceph-mon[83680]: from='client.54598 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T14:27:45.816 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:27:45 smithi069 ceph-mon[83680]: Upgrade: Updating rgw.foo.smithi069.wesedz (1/2) 2024-04-14T14:27:45.816 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:27:45 smithi069 ceph-mon[83680]: Deploying daemon rgw.foo.smithi069.wesedz on smithi069 2024-04-14T14:27:45.817 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:27:45 smithi069 ceph-mon[83680]: pgmap v262: 161 pgs: 161 active+clean; 585 KiB data, 258 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:27:45.817 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:27:45 smithi069 ceph-mon[83680]: from='client.? 172.21.15.69:0/3876709707' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T14:27:45.817 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:27:45 smithi069 ceph-mon[88636]: from='client.44655 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T14:27:45.817 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:27:45 smithi069 ceph-mon[88636]: from='client.54598 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T14:27:45.817 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:27:45 smithi069 ceph-mon[88636]: Upgrade: Updating rgw.foo.smithi069.wesedz (1/2) 2024-04-14T14:27:45.817 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:27:45 smithi069 ceph-mon[88636]: Deploying daemon rgw.foo.smithi069.wesedz on smithi069 2024-04-14T14:27:45.817 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:27:45 smithi069 ceph-mon[88636]: pgmap v262: 161 pgs: 161 active+clean; 585 KiB data, 258 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:27:45.817 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:27:45 smithi069 ceph-mon[88636]: from='client.? 172.21.15.69:0/3876709707' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T14:27:46.094 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:27:45 smithi122 ceph-mon[67998]: from='client.44655 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T14:27:46.094 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:27:45 smithi122 ceph-mon[67998]: from='client.54598 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T14:27:46.094 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:27:45 smithi122 ceph-mon[67998]: Upgrade: Updating rgw.foo.smithi069.wesedz (1/2) 2024-04-14T14:27:46.094 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:27:45 smithi122 ceph-mon[67998]: Deploying daemon rgw.foo.smithi069.wesedz on smithi069 2024-04-14T14:27:46.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:27:45 smithi122 ceph-mon[67998]: pgmap v262: 161 pgs: 161 active+clean; 585 KiB data, 258 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:27:46.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:27:45 smithi122 ceph-mon[67998]: from='client.? 172.21.15.69:0/3876709707' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T14:27:46.816 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:27:46 smithi069 ceph-mon[83680]: from='client.54610 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T14:27:46.816 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 14 14:27:46 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-y[63038]: ::ffff:172.21.15.122 - - [14/Apr/2024:14:27:46] "GET /metrics HTTP/1.1" 200 38375 "" "Prometheus/2.43.0" 2024-04-14T14:27:46.817 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:27:46 smithi069 ceph-mon[88636]: from='client.54610 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T14:27:47.094 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:27:46 smithi122 ceph-mon[67998]: from='client.54610 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T14:27:47.830 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:27:47 smithi069 ceph-mon[83680]: pgmap v263: 161 pgs: 161 active+clean; 585 KiB data, 258 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:27:47.830 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:27:47 smithi069 ceph-mon[88636]: pgmap v263: 161 pgs: 161 active+clean; 585 KiB data, 258 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:27:48.094 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:27:47 smithi122 ceph-mon[67998]: pgmap v263: 161 pgs: 161 active+clean; 585 KiB data, 258 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:27:49.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:27:49 smithi122 ceph-mon[67998]: from='client.25417 -' entity='client.iscsi.foo.smithi069.yuumtr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T14:27:49.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:27:49 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:27:49.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:27:49 smithi122 ceph-mon[67998]: pgmap v264: 161 pgs: 161 active+clean; 585 KiB data, 258 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:27:49.845 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:27:49 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:27:50.036 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:27:49 smithi069 ceph-mon[83680]: from='client.25417 -' entity='client.iscsi.foo.smithi069.yuumtr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T14:27:50.036 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:27:49 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:27:50.037 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:27:49 smithi069 ceph-mon[83680]: pgmap v264: 161 pgs: 161 active+clean; 585 KiB data, 258 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:27:50.037 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:27:49 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:27:50.037 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:27:49 smithi069 ceph-mon[88636]: from='client.25417 -' entity='client.iscsi.foo.smithi069.yuumtr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T14:27:50.037 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:27:49 smithi069 ceph-mon[88636]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:27:50.037 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:27:49 smithi069 ceph-mon[88636]: pgmap v264: 161 pgs: 161 active+clean; 585 KiB data, 258 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:27:50.037 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:27:49 smithi069 ceph-mon[88636]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:27:51.344 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:27:51 smithi122 ceph-mon[67998]: Upgrade: Updating rgw.foo.smithi122.jiobyv (2/2) 2024-04-14T14:27:51.344 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:27:51 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:27:51.345 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:27:51 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi122.jiobyv", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-04-14T14:27:51.345 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:27:51 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T14:27:51.345 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:27:51 smithi122 ceph-mon[67998]: Deploying daemon rgw.foo.smithi122.jiobyv on smithi122 2024-04-14T14:27:51.345 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:27:51 smithi122 ceph-mon[67998]: pgmap v265: 161 pgs: 161 active+clean; 585 KiB data, 258 MiB used, 715 GiB / 715 GiB avail; 26 KiB/s rd, 0 B/s wr, 40 op/s 2024-04-14T14:27:51.566 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:27:51 smithi069 ceph-mon[88636]: Upgrade: Updating rgw.foo.smithi122.jiobyv (2/2) 2024-04-14T14:27:51.566 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:27:51 smithi069 ceph-mon[88636]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:27:51.566 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:27:51 smithi069 ceph-mon[88636]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi122.jiobyv", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-04-14T14:27:51.566 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:27:51 smithi069 ceph-mon[88636]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T14:27:51.566 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:27:51 smithi069 ceph-mon[88636]: Deploying daemon rgw.foo.smithi122.jiobyv on smithi122 2024-04-14T14:27:51.567 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:27:51 smithi069 ceph-mon[88636]: pgmap v265: 161 pgs: 161 active+clean; 585 KiB data, 258 MiB used, 715 GiB / 715 GiB avail; 26 KiB/s rd, 0 B/s wr, 40 op/s 2024-04-14T14:27:51.567 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:27:51 smithi069 ceph-mon[83680]: Upgrade: Updating rgw.foo.smithi122.jiobyv (2/2) 2024-04-14T14:27:51.567 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:27:51 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:27:51.567 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:27:51 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi122.jiobyv", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-04-14T14:27:51.567 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:27:51 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T14:27:51.567 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:27:51 smithi069 ceph-mon[83680]: Deploying daemon rgw.foo.smithi122.jiobyv on smithi122 2024-04-14T14:27:51.567 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:27:51 smithi069 ceph-mon[83680]: pgmap v265: 161 pgs: 161 active+clean; 585 KiB data, 258 MiB used, 715 GiB / 715 GiB avail; 26 KiB/s rd, 0 B/s wr, 40 op/s 2024-04-14T14:27:52.344 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:27:52 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T14:27:52.566 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:27:52 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T14:27:52.566 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:27:52 smithi069 ceph-mon[88636]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T14:27:53.546 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:27:53 smithi122 ceph-mon[67998]: pgmap v266: 161 pgs: 161 active+clean; 585 KiB data, 258 MiB used, 715 GiB / 715 GiB avail; 44 KiB/s rd, 85 B/s wr, 67 op/s 2024-04-14T14:27:53.566 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:27:53 smithi069 ceph-mon[83680]: pgmap v266: 161 pgs: 161 active+clean; 585 KiB data, 258 MiB used, 715 GiB / 715 GiB avail; 44 KiB/s rd, 85 B/s wr, 67 op/s 2024-04-14T14:27:53.566 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:27:53 smithi069 ceph-mon[88636]: pgmap v266: 161 pgs: 161 active+clean; 585 KiB data, 258 MiB used, 715 GiB / 715 GiB avail; 44 KiB/s rd, 85 B/s wr, 67 op/s 2024-04-14T14:27:55.316 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:27:55 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:27:55.316 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:27:55 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:27:55.317 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:27:55 smithi069 ceph-mon[83680]: pgmap v267: 161 pgs: 161 active+clean; 585 KiB data, 262 MiB used, 715 GiB / 715 GiB avail; 55 KiB/s rd, 85 B/s wr, 84 op/s 2024-04-14T14:27:55.317 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:27:55 smithi069 ceph-mon[88636]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:27:55.317 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:27:55 smithi069 ceph-mon[88636]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:27:55.317 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:27:55 smithi069 ceph-mon[88636]: pgmap v267: 161 pgs: 161 active+clean; 585 KiB data, 262 MiB used, 715 GiB / 715 GiB avail; 55 KiB/s rd, 85 B/s wr, 84 op/s 2024-04-14T14:27:55.344 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:27:55 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:27:55.344 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:27:55 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:27:55.344 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:27:55 smithi122 ceph-mon[67998]: pgmap v267: 161 pgs: 161 active+clean; 585 KiB data, 262 MiB used, 715 GiB / 715 GiB avail; 55 KiB/s rd, 85 B/s wr, 84 op/s 2024-04-14T14:27:56.816 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 14 14:27:56 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-y[63038]: ::ffff:172.21.15.122 - - [14/Apr/2024:14:27:56] "GET /metrics HTTP/1.1" 200 38379 "" "Prometheus/2.43.0" 2024-04-14T14:27:58.066 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:27:57 smithi069 ceph-mon[83680]: pgmap v268: 161 pgs: 161 active+clean; 585 KiB data, 262 MiB used, 715 GiB / 715 GiB avail; 82 KiB/s rd, 85 B/s wr, 127 op/s 2024-04-14T14:27:58.066 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:27:57 smithi069 ceph-mon[88636]: pgmap v268: 161 pgs: 161 active+clean; 585 KiB data, 262 MiB used, 715 GiB / 715 GiB avail; 82 KiB/s rd, 85 B/s wr, 127 op/s 2024-04-14T14:27:58.094 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:27:57 smithi122 ceph-mon[67998]: pgmap v268: 161 pgs: 161 active+clean; 585 KiB data, 262 MiB used, 715 GiB / 715 GiB avail; 82 KiB/s rd, 85 B/s wr, 127 op/s 2024-04-14T14:27:59.066 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:27:58 smithi069 ceph-mon[88636]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:27:59.066 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:27:58 smithi069 ceph-mon[88636]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:27:59.066 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:27:58 smithi069 ceph-mon[88636]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:27:59.066 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:27:58 smithi069 ceph-mon[88636]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:27:59.067 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:27:58 smithi069 ceph-mon[88636]: from='client.25417 -' entity='client.iscsi.foo.smithi069.yuumtr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T14:27:59.067 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:27:58 smithi069 ceph-mon[88636]: pgmap v269: 161 pgs: 161 active+clean; 585 KiB data, 266 MiB used, 715 GiB / 715 GiB avail; 99 KiB/s rd, 170 B/s wr, 152 op/s 2024-04-14T14:27:59.067 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:27:58 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:27:59.067 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:27:58 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:27:59.067 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:27:58 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:27:59.067 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:27:58 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:27:59.067 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:27:58 smithi069 ceph-mon[83680]: from='client.25417 -' entity='client.iscsi.foo.smithi069.yuumtr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T14:27:59.067 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:27:58 smithi069 ceph-mon[83680]: pgmap v269: 161 pgs: 161 active+clean; 585 KiB data, 266 MiB used, 715 GiB / 715 GiB avail; 99 KiB/s rd, 170 B/s wr, 152 op/s 2024-04-14T14:27:59.094 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:27:58 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:27:59.094 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:27:58 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:27:59.094 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:27:58 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:27:59.094 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:27:58 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:27:59.094 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:27:58 smithi122 ceph-mon[67998]: from='client.25417 -' entity='client.iscsi.foo.smithi069.yuumtr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T14:27:59.094 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:27:58 smithi122 ceph-mon[67998]: pgmap v269: 161 pgs: 161 active+clean; 585 KiB data, 266 MiB used, 715 GiB / 715 GiB avail; 99 KiB/s rd, 170 B/s wr, 152 op/s 2024-04-14T14:27:59.816 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:27:59 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:27:59.816 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:27:59 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:27:59.817 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:27:59 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:27:59.817 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:27:59 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:27:59.817 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:27:59 smithi069 ceph-mon[88636]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:27:59.817 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:27:59 smithi069 ceph-mon[88636]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:27:59.817 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:27:59 smithi069 ceph-mon[88636]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:27:59.817 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:27:59 smithi069 ceph-mon[88636]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:28:00.094 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:27:59 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:28:00.094 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:27:59 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:28:00.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:27:59 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:28:00.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:27:59 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:28:00.816 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:28:00 smithi069 ceph-mon[88636]: pgmap v270: 161 pgs: 161 active+clean; 585 KiB data, 266 MiB used, 715 GiB / 715 GiB avail; 109 KiB/s rd, 170 B/s wr, 168 op/s 2024-04-14T14:28:00.817 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:28:00 smithi069 ceph-mon[83680]: pgmap v270: 161 pgs: 161 active+clean; 585 KiB data, 266 MiB used, 715 GiB / 715 GiB avail; 109 KiB/s rd, 170 B/s wr, 168 op/s 2024-04-14T14:28:01.094 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:28:00 smithi122 ceph-mon[67998]: pgmap v270: 161 pgs: 161 active+clean; 585 KiB data, 266 MiB used, 715 GiB / 715 GiB avail; 109 KiB/s rd, 170 B/s wr, 168 op/s 2024-04-14T14:28:03.345 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:28:03 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:28:03.345 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:28:03 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:28:03.345 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:28:03 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:28:03.345 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:28:03 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:28:03.345 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:28:03 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T14:28:03.345 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:28:03 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-14T14:28:03.345 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:28:03 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:28:03.345 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:28:03 smithi122 ceph-mon[67998]: pgmap v271: 161 pgs: 161 active+clean; 585 KiB data, 266 MiB used, 715 GiB / 715 GiB avail; 84 KiB/s rd, 170 B/s wr, 129 op/s 2024-04-14T14:28:03.345 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:28:03 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-14T14:28:03.346 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:28:03 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T14:28:03.346 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:28:03 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T14:28:03.346 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:28:03 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T14:28:03.346 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:28:03 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T14:28:03.346 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:28:03 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T14:28:03.346 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:28:03 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T14:28:03.346 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:28:03 smithi122 ceph-mon[67998]: Upgrade: Setting container_image for all rgw 2024-04-14T14:28:03.346 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:28:03 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:28:03.346 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:28:03 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw.foo.smithi069.wesedz"}]: dispatch 2024-04-14T14:28:03.346 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:28:03 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rgw.foo.smithi069.wesedz"}]': finished 2024-04-14T14:28:03.346 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:28:03 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw.foo.smithi122.jiobyv"}]: dispatch 2024-04-14T14:28:03.346 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:28:03 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rgw.foo.smithi122.jiobyv"}]': finished 2024-04-14T14:28:03.346 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:28:03 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T14:28:03.347 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:28:03 smithi122 ceph-mon[67998]: Upgrade: Setting container_image for all rbd-mirror 2024-04-14T14:28:03.347 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:28:03 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:28:03.347 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:28:03 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T14:28:03.347 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:28:03 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T14:28:03.347 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:28:03 smithi122 ceph-mon[67998]: Upgrade: Setting container_image for all ceph-exporter 2024-04-14T14:28:03.347 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:28:03 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:28:03.347 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:28:03 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T14:28:03.347 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:28:03 smithi122 ceph-mon[67998]: Upgrade: Setting container_image for all nfs 2024-04-14T14:28:03.347 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:28:03 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:28:03.347 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:28:03 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T14:28:03.347 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:28:03 smithi122 ceph-mon[67998]: Upgrade: Setting container_image for all nvmeof 2024-04-14T14:28:03.347 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:28:03 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:28:03.347 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:28:03 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T14:28:03.347 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:28:03 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T14:28:03.347 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:28:03 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T14:28:03.348 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:28:03 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T14:28:03.348 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:28:03 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T14:28:03.348 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:28:03 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T14:28:03.348 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:28:03 smithi122 ceph-mon[67998]: Upgrade: Finalizing container_image settings 2024-04-14T14:28:03.348 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:28:03 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]: dispatch 2024-04-14T14:28:03.348 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:28:03 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]': finished 2024-04-14T14:28:03.348 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:28:03 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-14T14:28:03.348 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:28:03 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon"}]': finished 2024-04-14T14:28:03.348 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:28:03 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]: dispatch 2024-04-14T14:28:03.348 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:28:03 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]': finished 2024-04-14T14:28:03.348 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:28:03 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd"}]: dispatch 2024-04-14T14:28:03.348 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:28:03 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd"}]': finished 2024-04-14T14:28:03.348 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:28:03 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mds"}]: dispatch 2024-04-14T14:28:03.348 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:28:03 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mds"}]': finished 2024-04-14T14:28:03.349 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:28:03 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw"}]: dispatch 2024-04-14T14:28:03.349 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:28:03 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rgw"}]': finished 2024-04-14T14:28:03.349 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:28:03 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"}]: dispatch 2024-04-14T14:28:03.349 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:28:03 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"}]': finished 2024-04-14T14:28:03.349 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:28:03 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-14T14:28:03.349 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:28:03 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter"}]: dispatch 2024-04-14T14:28:03.349 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:28:03 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter"}]': finished 2024-04-14T14:28:03.349 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:28:03 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.iscsi"}]: dispatch 2024-04-14T14:28:03.349 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:28:03 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nfs"}]: dispatch 2024-04-14T14:28:03.350 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:28:03 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.nfs"}]': finished 2024-04-14T14:28:03.350 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:28:03 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]: dispatch 2024-04-14T14:28:03.350 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:28:03 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]': finished 2024-04-14T14:28:03.350 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:28:03 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-14T14:28:03.350 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:28:03 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-14T14:28:03.350 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:28:03 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-14T14:28:03.350 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:28:03 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-14T14:28:03.350 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:28:03 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-14T14:28:03.350 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:28:03 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-14T14:28:03.350 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:28:03 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]: dispatch 2024-04-14T14:28:03.350 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:28:03 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd='[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]': finished 2024-04-14T14:28:03.350 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:28:03 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T14:28:03.350 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:28:03 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-14T14:28:03.350 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:28:03 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:28:03.351 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:28:03 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T14:28:03.351 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:28:03 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-14T14:28:03.351 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:28:03 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:28:03.566 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:28:03 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:28:03.567 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:28:03 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:28:03.567 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:28:03 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:28:03.567 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:28:03 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:28:03.567 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:28:03 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T14:28:03.567 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:28:03 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-14T14:28:03.567 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:28:03 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:28:03.567 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:28:03 smithi069 ceph-mon[83680]: pgmap v271: 161 pgs: 161 active+clean; 585 KiB data, 266 MiB used, 715 GiB / 715 GiB avail; 84 KiB/s rd, 170 B/s wr, 129 op/s 2024-04-14T14:28:03.567 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:28:03 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-14T14:28:03.567 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:28:03 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T14:28:03.567 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:28:03 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T14:28:03.567 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:28:03 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T14:28:03.567 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:28:03 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T14:28:03.567 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:28:03 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T14:28:03.568 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:28:03 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T14:28:03.568 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:28:03 smithi069 ceph-mon[83680]: Upgrade: Setting container_image for all rgw 2024-04-14T14:28:03.568 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:28:03 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:28:03.568 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:28:03 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw.foo.smithi069.wesedz"}]: dispatch 2024-04-14T14:28:03.568 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:28:03 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rgw.foo.smithi069.wesedz"}]': finished 2024-04-14T14:28:03.568 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:28:03 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw.foo.smithi122.jiobyv"}]: dispatch 2024-04-14T14:28:03.568 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:28:03 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rgw.foo.smithi122.jiobyv"}]': finished 2024-04-14T14:28:03.568 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:28:03 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T14:28:03.568 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:28:03 smithi069 ceph-mon[83680]: Upgrade: Setting container_image for all rbd-mirror 2024-04-14T14:28:03.568 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:28:03 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:28:03.568 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:28:03 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T14:28:03.568 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:28:03 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T14:28:03.568 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:28:03 smithi069 ceph-mon[83680]: Upgrade: Setting container_image for all ceph-exporter 2024-04-14T14:28:03.568 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:28:03 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:28:03.568 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:28:03 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T14:28:03.569 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:28:03 smithi069 ceph-mon[83680]: Upgrade: Setting container_image for all nfs 2024-04-14T14:28:03.569 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:28:03 smithi069 ceph-mon[88636]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:28:03.569 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:28:03 smithi069 ceph-mon[88636]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:28:03.569 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:28:03 smithi069 ceph-mon[88636]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:28:03.569 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:28:03 smithi069 ceph-mon[88636]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:28:03.569 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:28:03 smithi069 ceph-mon[88636]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T14:28:03.569 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:28:03 smithi069 ceph-mon[88636]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-14T14:28:03.569 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:28:03 smithi069 ceph-mon[88636]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:28:03.569 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:28:03 smithi069 ceph-mon[88636]: pgmap v271: 161 pgs: 161 active+clean; 585 KiB data, 266 MiB used, 715 GiB / 715 GiB avail; 84 KiB/s rd, 170 B/s wr, 129 op/s 2024-04-14T14:28:03.569 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:28:03 smithi069 ceph-mon[88636]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-14T14:28:03.569 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:28:03 smithi069 ceph-mon[88636]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T14:28:03.570 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:28:03 smithi069 ceph-mon[88636]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T14:28:03.570 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:28:03 smithi069 ceph-mon[88636]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T14:28:03.570 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:28:03 smithi069 ceph-mon[88636]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T14:28:03.570 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:28:03 smithi069 ceph-mon[88636]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T14:28:03.570 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:28:03 smithi069 ceph-mon[88636]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T14:28:03.570 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:28:03 smithi069 ceph-mon[88636]: Upgrade: Setting container_image for all rgw 2024-04-14T14:28:03.570 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:28:03 smithi069 ceph-mon[88636]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:28:03.570 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:28:03 smithi069 ceph-mon[88636]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw.foo.smithi069.wesedz"}]: dispatch 2024-04-14T14:28:03.570 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:28:03 smithi069 ceph-mon[88636]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rgw.foo.smithi069.wesedz"}]': finished 2024-04-14T14:28:03.570 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:28:03 smithi069 ceph-mon[88636]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw.foo.smithi122.jiobyv"}]: dispatch 2024-04-14T14:28:03.570 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:28:03 smithi069 ceph-mon[88636]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rgw.foo.smithi122.jiobyv"}]': finished 2024-04-14T14:28:03.570 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:28:03 smithi069 ceph-mon[88636]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T14:28:03.570 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:28:03 smithi069 ceph-mon[88636]: Upgrade: Setting container_image for all rbd-mirror 2024-04-14T14:28:03.570 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:28:03 smithi069 ceph-mon[88636]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:28:03.571 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:28:03 smithi069 ceph-mon[88636]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T14:28:03.571 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:28:03 smithi069 ceph-mon[88636]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T14:28:03.571 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:28:03 smithi069 ceph-mon[88636]: Upgrade: Setting container_image for all ceph-exporter 2024-04-14T14:28:03.571 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:28:03 smithi069 ceph-mon[88636]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:28:03.571 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:28:03 smithi069 ceph-mon[88636]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T14:28:03.571 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:28:03 smithi069 ceph-mon[88636]: Upgrade: Setting container_image for all nfs 2024-04-14T14:28:03.571 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:28:03 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:28:03.571 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:28:03 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T14:28:03.571 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:28:03 smithi069 ceph-mon[83680]: Upgrade: Setting container_image for all nvmeof 2024-04-14T14:28:03.571 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:28:03 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:28:03.571 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:28:03 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T14:28:03.572 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:28:03 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T14:28:03.572 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:28:03 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T14:28:03.572 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:28:03 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T14:28:03.572 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:28:03 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T14:28:03.572 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:28:03 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T14:28:03.572 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:28:03 smithi069 ceph-mon[83680]: Upgrade: Finalizing container_image settings 2024-04-14T14:28:03.572 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:28:03 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]: dispatch 2024-04-14T14:28:03.572 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:28:03 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]': finished 2024-04-14T14:28:03.572 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:28:03 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-14T14:28:03.572 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:28:03 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon"}]': finished 2024-04-14T14:28:03.572 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:28:03 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]: dispatch 2024-04-14T14:28:03.572 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:28:03 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]': finished 2024-04-14T14:28:03.572 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:28:03 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd"}]: dispatch 2024-04-14T14:28:03.572 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:28:03 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd"}]': finished 2024-04-14T14:28:03.572 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:28:03 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mds"}]: dispatch 2024-04-14T14:28:03.573 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:28:03 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mds"}]': finished 2024-04-14T14:28:03.573 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:28:03 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw"}]: dispatch 2024-04-14T14:28:03.573 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:28:03 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rgw"}]': finished 2024-04-14T14:28:03.573 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:28:03 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"}]: dispatch 2024-04-14T14:28:03.573 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:28:03 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"}]': finished 2024-04-14T14:28:03.573 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:28:03 smithi069 ceph-mon[88636]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:28:03.573 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:28:03 smithi069 ceph-mon[88636]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T14:28:03.573 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:28:03 smithi069 ceph-mon[88636]: Upgrade: Setting container_image for all nvmeof 2024-04-14T14:28:03.573 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:28:03 smithi069 ceph-mon[88636]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:28:03.573 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:28:03 smithi069 ceph-mon[88636]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T14:28:03.573 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:28:03 smithi069 ceph-mon[88636]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T14:28:03.574 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:28:03 smithi069 ceph-mon[88636]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T14:28:03.574 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:28:03 smithi069 ceph-mon[88636]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T14:28:03.574 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:28:03 smithi069 ceph-mon[88636]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T14:28:03.574 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:28:03 smithi069 ceph-mon[88636]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T14:28:03.574 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:28:03 smithi069 ceph-mon[88636]: Upgrade: Finalizing container_image settings 2024-04-14T14:28:03.574 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:28:03 smithi069 ceph-mon[88636]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]: dispatch 2024-04-14T14:28:03.574 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:28:03 smithi069 ceph-mon[88636]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]': finished 2024-04-14T14:28:03.574 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:28:03 smithi069 ceph-mon[88636]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-14T14:28:03.574 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:28:03 smithi069 ceph-mon[88636]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon"}]': finished 2024-04-14T14:28:03.574 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:28:03 smithi069 ceph-mon[88636]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]: dispatch 2024-04-14T14:28:03.574 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:28:03 smithi069 ceph-mon[88636]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]': finished 2024-04-14T14:28:03.574 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:28:03 smithi069 ceph-mon[88636]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd"}]: dispatch 2024-04-14T14:28:03.574 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:28:03 smithi069 ceph-mon[88636]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd"}]': finished 2024-04-14T14:28:03.574 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:28:03 smithi069 ceph-mon[88636]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mds"}]: dispatch 2024-04-14T14:28:03.575 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:28:03 smithi069 ceph-mon[88636]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mds"}]': finished 2024-04-14T14:28:03.575 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:28:03 smithi069 ceph-mon[88636]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw"}]: dispatch 2024-04-14T14:28:03.575 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:28:03 smithi069 ceph-mon[88636]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rgw"}]': finished 2024-04-14T14:28:03.575 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:28:03 smithi069 ceph-mon[88636]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"}]: dispatch 2024-04-14T14:28:03.575 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:28:03 smithi069 ceph-mon[88636]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"}]': finished 2024-04-14T14:28:03.575 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:28:03 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-14T14:28:03.575 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:28:03 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter"}]: dispatch 2024-04-14T14:28:03.575 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:28:03 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter"}]': finished 2024-04-14T14:28:03.575 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:28:03 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.iscsi"}]: dispatch 2024-04-14T14:28:03.575 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:28:03 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nfs"}]: dispatch 2024-04-14T14:28:03.575 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:28:03 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.nfs"}]': finished 2024-04-14T14:28:03.576 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:28:03 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]: dispatch 2024-04-14T14:28:03.576 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:28:03 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]': finished 2024-04-14T14:28:03.576 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:28:03 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-14T14:28:03.576 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:28:03 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-14T14:28:03.576 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:28:03 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-14T14:28:03.576 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:28:03 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-14T14:28:03.576 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:28:03 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-14T14:28:03.576 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:28:03 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-14T14:28:03.576 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:28:03 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]: dispatch 2024-04-14T14:28:03.576 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:28:03 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd='[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]': finished 2024-04-14T14:28:03.576 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:28:03 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T14:28:03.576 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:28:03 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-14T14:28:03.576 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:28:03 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:28:03.576 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:28:03 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T14:28:03.577 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:28:03 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-14T14:28:03.577 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:28:03 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:28:03.577 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:28:03 smithi069 ceph-mon[88636]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-14T14:28:03.577 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:28:03 smithi069 ceph-mon[88636]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter"}]: dispatch 2024-04-14T14:28:03.577 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:28:03 smithi069 ceph-mon[88636]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter"}]': finished 2024-04-14T14:28:03.577 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:28:03 smithi069 ceph-mon[88636]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.iscsi"}]: dispatch 2024-04-14T14:28:03.577 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:28:03 smithi069 ceph-mon[88636]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nfs"}]: dispatch 2024-04-14T14:28:03.577 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:28:03 smithi069 ceph-mon[88636]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.nfs"}]': finished 2024-04-14T14:28:03.577 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:28:03 smithi069 ceph-mon[88636]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]: dispatch 2024-04-14T14:28:03.577 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:28:03 smithi069 ceph-mon[88636]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]': finished 2024-04-14T14:28:03.577 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:28:03 smithi069 ceph-mon[88636]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-14T14:28:03.577 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:28:03 smithi069 ceph-mon[88636]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-14T14:28:03.578 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:28:03 smithi069 ceph-mon[88636]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-14T14:28:03.578 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:28:03 smithi069 ceph-mon[88636]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-14T14:28:03.578 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:28:03 smithi069 ceph-mon[88636]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-14T14:28:03.578 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:28:03 smithi069 ceph-mon[88636]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-14T14:28:03.578 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:28:03 smithi069 ceph-mon[88636]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]: dispatch 2024-04-14T14:28:03.578 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:28:03 smithi069 ceph-mon[88636]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd='[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]': finished 2024-04-14T14:28:03.578 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:28:03 smithi069 ceph-mon[88636]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T14:28:03.578 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:28:03 smithi069 ceph-mon[88636]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-14T14:28:03.578 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:28:03 smithi069 ceph-mon[88636]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:28:03.578 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:28:03 smithi069 ceph-mon[88636]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T14:28:03.578 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:28:03 smithi069 ceph-mon[88636]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-14T14:28:03.578 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:28:03 smithi069 ceph-mon[88636]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:28:04.816 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:28:04 smithi069 ceph-mon[83680]: Upgrade: Complete! 2024-04-14T14:28:04.816 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:28:04 smithi069 ceph-mon[83680]: Checking dashboard <-> RGW credentials 2024-04-14T14:28:04.816 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:28:04 smithi069 ceph-mon[88636]: Upgrade: Complete! 2024-04-14T14:28:04.817 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:28:04 smithi069 ceph-mon[88636]: Checking dashboard <-> RGW credentials 2024-04-14T14:28:04.844 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:28:04 smithi122 ceph-mon[67998]: Upgrade: Complete! 2024-04-14T14:28:04.844 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:28:04 smithi122 ceph-mon[67998]: Checking dashboard <-> RGW credentials 2024-04-14T14:28:05.816 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:28:05 smithi069 ceph-mon[83680]: pgmap v272: 161 pgs: 161 active+clean; 585 KiB data, 266 MiB used, 715 GiB / 715 GiB avail; 66 KiB/s rd, 85 B/s wr, 101 op/s 2024-04-14T14:28:05.816 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:28:05 smithi069 ceph-mon[88636]: pgmap v272: 161 pgs: 161 active+clean; 585 KiB data, 266 MiB used, 715 GiB / 715 GiB avail; 66 KiB/s rd, 85 B/s wr, 101 op/s 2024-04-14T14:28:05.844 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:28:05 smithi122 ceph-mon[67998]: pgmap v272: 161 pgs: 161 active+clean; 585 KiB data, 266 MiB used, 715 GiB / 715 GiB avail; 66 KiB/s rd, 85 B/s wr, 101 op/s 2024-04-14T14:28:06.818 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:28:06 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T14:28:06.845 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 14 14:28:06 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-y[63038]: ::ffff:172.21.15.122 - - [14/Apr/2024:14:28:06] "GET /metrics HTTP/1.1" 200 38379 "" "Prometheus/2.43.0" 2024-04-14T14:28:06.845 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:28:06 smithi069 ceph-mon[88636]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T14:28:07.094 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:28:06 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T14:28:07.901 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:28:07 smithi069 ceph-mon[83680]: pgmap v273: 161 pgs: 161 active+clean; 585 KiB data, 266 MiB used, 715 GiB / 715 GiB avail; 56 KiB/s rd, 85 B/s wr, 85 op/s 2024-04-14T14:28:07.901 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:28:07 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:28:07.902 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:28:07 smithi069 ceph-mon[88636]: pgmap v273: 161 pgs: 161 active+clean; 585 KiB data, 266 MiB used, 715 GiB / 715 GiB avail; 56 KiB/s rd, 85 B/s wr, 85 op/s 2024-04-14T14:28:07.902 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:28:07 smithi069 ceph-mon[88636]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:28:08.094 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:28:07 smithi122 ceph-mon[67998]: pgmap v273: 161 pgs: 161 active+clean; 585 KiB data, 266 MiB used, 715 GiB / 715 GiB avail; 56 KiB/s rd, 85 B/s wr, 85 op/s 2024-04-14T14:28:08.094 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:28:07 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:28:10.066 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:28:09 smithi069 ceph-mon[83680]: from='client.25417 -' entity='client.iscsi.foo.smithi069.yuumtr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T14:28:10.066 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:28:09 smithi069 ceph-mon[83680]: pgmap v274: 161 pgs: 161 active+clean; 585 KiB data, 266 MiB used, 715 GiB / 715 GiB avail; 30 KiB/s rd, 85 B/s wr, 44 op/s 2024-04-14T14:28:10.067 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:28:09 smithi069 ceph-mon[88636]: from='client.25417 -' entity='client.iscsi.foo.smithi069.yuumtr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T14:28:10.067 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:28:09 smithi069 ceph-mon[88636]: pgmap v274: 161 pgs: 161 active+clean; 585 KiB data, 266 MiB used, 715 GiB / 715 GiB avail; 30 KiB/s rd, 85 B/s wr, 44 op/s 2024-04-14T14:28:10.094 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:28:09 smithi122 ceph-mon[67998]: from='client.25417 -' entity='client.iscsi.foo.smithi069.yuumtr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T14:28:10.094 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:28:09 smithi122 ceph-mon[67998]: pgmap v274: 161 pgs: 161 active+clean; 585 KiB data, 266 MiB used, 715 GiB / 715 GiB avail; 30 KiB/s rd, 85 B/s wr, 44 op/s 2024-04-14T14:28:12.040 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:28:11 smithi069 ceph-mon[83680]: pgmap v275: 161 pgs: 161 active+clean; 585 KiB data, 266 MiB used, 715 GiB / 715 GiB avail; 13 KiB/s rd, 0 B/s wr, 18 op/s 2024-04-14T14:28:12.041 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:28:11 smithi069 ceph-mon[88636]: pgmap v275: 161 pgs: 161 active+clean; 585 KiB data, 266 MiB used, 715 GiB / 715 GiB avail; 13 KiB/s rd, 0 B/s wr, 18 op/s 2024-04-14T14:28:12.094 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:28:11 smithi122 ceph-mon[67998]: pgmap v275: 161 pgs: 161 active+clean; 585 KiB data, 266 MiB used, 715 GiB / 715 GiB avail; 13 KiB/s rd, 0 B/s wr, 18 op/s 2024-04-14T14:28:14.066 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:28:13 smithi069 ceph-mon[83680]: pgmap v276: 161 pgs: 161 active+clean; 585 KiB data, 266 MiB used, 715 GiB / 715 GiB avail; 3.1 KiB/s rd, 3 op/s 2024-04-14T14:28:14.067 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:28:13 smithi069 ceph-mon[88636]: pgmap v276: 161 pgs: 161 active+clean; 585 KiB data, 266 MiB used, 715 GiB / 715 GiB avail; 3.1 KiB/s rd, 3 op/s 2024-04-14T14:28:14.094 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:28:13 smithi122 ceph-mon[67998]: pgmap v276: 161 pgs: 161 active+clean; 585 KiB data, 266 MiB used, 715 GiB / 715 GiB avail; 3.1 KiB/s rd, 3 op/s 2024-04-14T14:28:15.066 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:28:14 smithi069 ceph-mon[83680]: pgmap v277: 161 pgs: 161 active+clean; 585 KiB data, 266 MiB used, 715 GiB / 715 GiB avail; 2.7 KiB/s rd, 2 op/s 2024-04-14T14:28:15.066 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:28:14 smithi069 ceph-mon[88636]: pgmap v277: 161 pgs: 161 active+clean; 585 KiB data, 266 MiB used, 715 GiB / 715 GiB avail; 2.7 KiB/s rd, 2 op/s 2024-04-14T14:28:15.094 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:28:14 smithi122 ceph-mon[67998]: pgmap v277: 161 pgs: 161 active+clean; 585 KiB data, 266 MiB used, 715 GiB / 715 GiB avail; 2.7 KiB/s rd, 2 op/s 2024-04-14T14:28:16.066 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:28:15 smithi069 ceph-mon[83680]: from='client.54700 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T14:28:16.066 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:28:15 smithi069 ceph-mon[88636]: from='client.54700 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T14:28:16.094 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:28:15 smithi122 ceph-mon[67998]: from='client.54700 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T14:28:16.138 DEBUG:teuthology.orchestra.run.smithi069:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid dc326fc2-fa65-11ee-bc8e-c7b262605968 -e sha1=a971fd5f5719038c70b817bafacd6d5400b1d888 -- bash -c 'ceph orch ps' 2024-04-14T14:28:16.722 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 14 14:28:16 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-y[63038]: ::ffff:172.21.15.122 - - [14/Apr/2024:14:28:16] "GET /metrics HTTP/1.1" 200 38413 "" "Prometheus/2.43.0" 2024-04-14T14:28:16.994 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:28:16 smithi069 ceph-mon[83680]: pgmap v278: 161 pgs: 161 active+clean; 585 KiB data, 266 MiB used, 715 GiB / 715 GiB avail; 2.7 KiB/s rd, 2 op/s 2024-04-14T14:28:16.994 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:28:16 smithi069 ceph-mon[88636]: pgmap v278: 161 pgs: 161 active+clean; 585 KiB data, 266 MiB used, 715 GiB / 715 GiB avail; 2.7 KiB/s rd, 2 op/s 2024-04-14T14:28:17.094 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:28:16 smithi122 ceph-mon[67998]: pgmap v278: 161 pgs: 161 active+clean; 585 KiB data, 266 MiB used, 715 GiB / 715 GiB avail; 2.7 KiB/s rd, 2 op/s 2024-04-14T14:28:18.057 INFO:teuthology.orchestra.run.smithi069.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-14T14:28:18.057 INFO:teuthology.orchestra.run.smithi069.stdout:alertmanager.a smithi069 *:9093,9094 running (10m) 20s ago 31m 25.9M - 0.25.0 c8568f914cd2 7524f106e7bd 2024-04-14T14:28:18.057 INFO:teuthology.orchestra.run.smithi069.stdout:grafana.a smithi122 *:3000 running (9m) 20s ago 31m 82.4M - 9.4.12 2bacad6d85d8 5685178e3eba 2024-04-14T14:28:18.057 INFO:teuthology.orchestra.run.smithi069.stdout:iscsi.foo.smithi069.yuumtr smithi069 running (11m) 20s ago 30m 62.9M - 3.5 e1d6a67b021e 38cc81ea8cfa 2024-04-14T14:28:18.057 INFO:teuthology.orchestra.run.smithi069.stdout:mgr.x smithi122 *:8443,9283,8765 running (10m) 20s ago 36m 455M - 19.0.0-2778-ga971fd5f b96c509eef84 e29977590b43 2024-04-14T14:28:18.057 INFO:teuthology.orchestra.run.smithi069.stdout:mgr.y smithi069 *:8443,9283,8765 running (20m) 20s ago 37m 513M - 19.0.0-2778-ga971fd5f b96c509eef84 7af3d29d0e5f 2024-04-14T14:28:18.057 INFO:teuthology.orchestra.run.smithi069.stdout:mon.a smithi069 running (7m) 20s ago 37m 63.2M 2048M 19.0.0-2778-ga971fd5f b96c509eef84 8bcd6b3d3846 2024-04-14T14:28:18.057 INFO:teuthology.orchestra.run.smithi069.stdout:mon.b smithi122 running (8m) 20s ago 36m 49.5M 2048M 19.0.0-2778-ga971fd5f b96c509eef84 05f437a1a9a1 2024-04-14T14:28:18.058 INFO:teuthology.orchestra.run.smithi069.stdout:mon.c smithi069 running (7m) 20s ago 36m 44.8M 2048M 19.0.0-2778-ga971fd5f b96c509eef84 84aa95f11d0c 2024-04-14T14:28:18.058 INFO:teuthology.orchestra.run.smithi069.stdout:node-exporter.a smithi069 *:9100 running (10m) 20s ago 32m 15.7M - 1.5.0 0da6a335fe13 0de0eb2171b9 2024-04-14T14:28:18.058 INFO:teuthology.orchestra.run.smithi069.stdout:node-exporter.b smithi122 *:9100 running (10m) 20s ago 32m 16.3M - 1.5.0 0da6a335fe13 67877bf02d8e 2024-04-14T14:28:18.058 INFO:teuthology.orchestra.run.smithi069.stdout:osd.0 smithi069 running (6m) 20s ago 35m 62.2M 2992M 19.0.0-2778-ga971fd5f b96c509eef84 a0e57e3fb494 2024-04-14T14:28:18.058 INFO:teuthology.orchestra.run.smithi069.stdout:osd.1 smithi069 running (5m) 20s ago 35m 62.6M 2992M 19.0.0-2778-ga971fd5f b96c509eef84 cf73d17919bb 2024-04-14T14:28:18.058 INFO:teuthology.orchestra.run.smithi069.stdout:osd.2 smithi069 running (4m) 20s ago 35m 49.7M 2992M 19.0.0-2778-ga971fd5f b96c509eef84 c0efba5ef451 2024-04-14T14:28:18.058 INFO:teuthology.orchestra.run.smithi069.stdout:osd.3 smithi069 running (3m) 20s ago 34m 60.6M 2992M 19.0.0-2778-ga971fd5f b96c509eef84 d13460da0a65 2024-04-14T14:28:18.058 INFO:teuthology.orchestra.run.smithi069.stdout:osd.4 smithi122 running (3m) 20s ago 34m 58.6M 3248M 19.0.0-2778-ga971fd5f b96c509eef84 7a87a024f984 2024-04-14T14:28:18.058 INFO:teuthology.orchestra.run.smithi069.stdout:osd.5 smithi122 running (2m) 20s ago 34m 56.7M 3248M 19.0.0-2778-ga971fd5f b96c509eef84 02b623d44db3 2024-04-14T14:28:18.058 INFO:teuthology.orchestra.run.smithi069.stdout:osd.6 smithi122 running (2m) 20s ago 33m 53.9M 3248M 19.0.0-2778-ga971fd5f b96c509eef84 72f85ece3e75 2024-04-14T14:28:18.058 INFO:teuthology.orchestra.run.smithi069.stdout:osd.7 smithi122 running (98s) 20s ago 33m 55.3M 3248M 19.0.0-2778-ga971fd5f b96c509eef84 0c53c3cd2bc0 2024-04-14T14:28:18.058 INFO:teuthology.orchestra.run.smithi069.stdout:prometheus.a smithi122 *:9095 running (10m) 20s ago 32m 71.1M - 2.43.0 a07b618ecd1d 5c1f41d3bbe6 2024-04-14T14:28:18.058 INFO:teuthology.orchestra.run.smithi069.stdout:rgw.foo.smithi069.wesedz smithi069 *:8000 running (29s) 20s ago 31m 77.2M - 19.0.0-2778-ga971fd5f b96c509eef84 f135fd86285d 2024-04-14T14:28:18.058 INFO:teuthology.orchestra.run.smithi069.stdout:rgw.foo.smithi122.jiobyv smithi122 *:8000 running (24s) 20s ago 30m 76.3M - 19.0.0-2778-ga971fd5f b96c509eef84 7d37b0b84199 2024-04-14T14:28:18.675 DEBUG:teuthology.orchestra.run.smithi069:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid dc326fc2-fa65-11ee-bc8e-c7b262605968 -e sha1=a971fd5f5719038c70b817bafacd6d5400b1d888 -- bash -c 'ceph versions | jq -e '"'"'.rgw | length == 1'"'"'' 2024-04-14T14:28:20.030 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:28:19 smithi069 ceph-mon[88636]: from='client.25417 -' entity='client.iscsi.foo.smithi069.yuumtr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T14:28:20.031 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:28:19 smithi069 ceph-mon[88636]: from='client.54706 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T14:28:20.031 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:28:19 smithi069 ceph-mon[88636]: pgmap v279: 161 pgs: 161 active+clean; 585 KiB data, 266 MiB used, 715 GiB / 715 GiB avail; 2.2 KiB/s rd, 2 op/s 2024-04-14T14:28:20.031 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:28:19 smithi069 ceph-mon[83680]: from='client.25417 -' entity='client.iscsi.foo.smithi069.yuumtr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T14:28:20.031 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:28:19 smithi069 ceph-mon[83680]: from='client.54706 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T14:28:20.031 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:28:19 smithi069 ceph-mon[83680]: pgmap v279: 161 pgs: 161 active+clean; 585 KiB data, 266 MiB used, 715 GiB / 715 GiB avail; 2.2 KiB/s rd, 2 op/s 2024-04-14T14:28:20.094 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:28:19 smithi122 ceph-mon[67998]: from='client.25417 -' entity='client.iscsi.foo.smithi069.yuumtr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T14:28:20.094 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:28:19 smithi122 ceph-mon[67998]: from='client.54706 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T14:28:20.094 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:28:19 smithi122 ceph-mon[67998]: pgmap v279: 161 pgs: 161 active+clean; 585 KiB data, 266 MiB used, 715 GiB / 715 GiB avail; 2.2 KiB/s rd, 2 op/s 2024-04-14T14:28:20.936 INFO:teuthology.orchestra.run.smithi069.stdout:true 2024-04-14T14:28:21.649 DEBUG:teuthology.orchestra.run.smithi069:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid dc326fc2-fa65-11ee-bc8e-c7b262605968 -e sha1=a971fd5f5719038c70b817bafacd6d5400b1d888 -- bash -c 'ceph versions | jq -e '"'"'.rgw | keys'"'"' | grep $sha1' 2024-04-14T14:28:21.958 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:28:21 smithi069 ceph-mon[83680]: pgmap v280: 161 pgs: 161 active+clean; 585 KiB data, 266 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:28:21.958 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:28:21 smithi069 ceph-mon[83680]: from='client.? 172.21.15.69:0/862666865' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T14:28:21.959 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:28:21 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T14:28:21.959 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:28:21 smithi069 ceph-mon[88636]: pgmap v280: 161 pgs: 161 active+clean; 585 KiB data, 266 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:28:21.959 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:28:21 smithi069 ceph-mon[88636]: from='client.? 172.21.15.69:0/862666865' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T14:28:21.959 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:28:21 smithi069 ceph-mon[88636]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T14:28:22.094 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:28:21 smithi122 ceph-mon[67998]: pgmap v280: 161 pgs: 161 active+clean; 585 KiB data, 266 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:28:22.094 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:28:21 smithi122 ceph-mon[67998]: from='client.? 172.21.15.69:0/862666865' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T14:28:22.094 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:28:21 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T14:28:23.805 INFO:teuthology.orchestra.run.smithi069.stdout: "ceph version 19.0.0-2778-ga971fd5f (a971fd5f5719038c70b817bafacd6d5400b1d888) squid (dev)" 2024-04-14T14:28:24.066 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:28:23 smithi069 ceph-mon[83680]: pgmap v281: 161 pgs: 161 active+clean; 585 KiB data, 266 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:28:24.066 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:28:23 smithi069 ceph-mon[88636]: pgmap v281: 161 pgs: 161 active+clean; 585 KiB data, 266 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:28:24.094 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:28:23 smithi122 ceph-mon[67998]: pgmap v281: 161 pgs: 161 active+clean; 585 KiB data, 266 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:28:24.468 DEBUG:teuthology.orchestra.run.smithi069:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid dc326fc2-fa65-11ee-bc8e-c7b262605968 -e sha1=a971fd5f5719038c70b817bafacd6d5400b1d888 -- bash -c 'ceph orch upgrade status' 2024-04-14T14:28:25.011 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:28:24 smithi069 ceph-mon[83680]: from='client.? 172.21.15.69:0/1782942624' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T14:28:25.011 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:28:24 smithi069 ceph-mon[88636]: from='client.? 172.21.15.69:0/1782942624' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T14:28:25.094 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:28:24 smithi122 ceph-mon[67998]: from='client.? 172.21.15.69:0/1782942624' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T14:28:25.964 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:28:25 smithi069 ceph-mon[83680]: pgmap v282: 161 pgs: 161 active+clean; 585 KiB data, 266 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:28:25.964 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:28:25 smithi069 ceph-mon[88636]: pgmap v282: 161 pgs: 161 active+clean; 585 KiB data, 266 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:28:26.094 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:28:25 smithi122 ceph-mon[67998]: pgmap v282: 161 pgs: 161 active+clean; 585 KiB data, 266 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:28:26.430 INFO:teuthology.orchestra.run.smithi069.stdout:{ 2024-04-14T14:28:26.430 INFO:teuthology.orchestra.run.smithi069.stdout: "target_image": null, 2024-04-14T14:28:26.430 INFO:teuthology.orchestra.run.smithi069.stdout: "in_progress": false, 2024-04-14T14:28:26.430 INFO:teuthology.orchestra.run.smithi069.stdout: "which": "", 2024-04-14T14:28:26.430 INFO:teuthology.orchestra.run.smithi069.stdout: "services_complete": [], 2024-04-14T14:28:26.430 INFO:teuthology.orchestra.run.smithi069.stdout: "progress": null, 2024-04-14T14:28:26.430 INFO:teuthology.orchestra.run.smithi069.stdout: "message": "", 2024-04-14T14:28:26.430 INFO:teuthology.orchestra.run.smithi069.stdout: "is_paused": false 2024-04-14T14:28:26.430 INFO:teuthology.orchestra.run.smithi069.stdout:} 2024-04-14T14:28:26.713 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 14 14:28:26 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-y[63038]: ::ffff:172.21.15.122 - - [14/Apr/2024:14:28:26] "GET /metrics HTTP/1.1" 200 38412 "" "Prometheus/2.43.0" 2024-04-14T14:28:27.066 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:28:26 smithi069 ceph-mon[83680]: from='client.54724 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T14:28:27.066 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:28:26 smithi069 ceph-mon[83680]: pgmap v283: 161 pgs: 161 active+clean; 585 KiB data, 266 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:28:27.066 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:28:26 smithi069 ceph-mon[88636]: from='client.54724 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T14:28:27.066 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:28:26 smithi069 ceph-mon[88636]: pgmap v283: 161 pgs: 161 active+clean; 585 KiB data, 266 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:28:27.094 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:28:26 smithi122 ceph-mon[67998]: from='client.54724 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T14:28:27.094 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:28:26 smithi122 ceph-mon[67998]: pgmap v283: 161 pgs: 161 active+clean; 585 KiB data, 266 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:28:27.196 DEBUG:teuthology.orchestra.run.smithi069:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid dc326fc2-fa65-11ee-bc8e-c7b262605968 -e sha1=a971fd5f5719038c70b817bafacd6d5400b1d888 -- bash -c 'ceph health detail' 2024-04-14T14:28:29.470 INFO:teuthology.orchestra.run.smithi069.stdout:HEALTH_OK 2024-04-14T14:28:29.803 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:28:29 smithi069 ceph-mon[83680]: from='client.25417 -' entity='client.iscsi.foo.smithi069.yuumtr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T14:28:29.803 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:28:29 smithi069 ceph-mon[83680]: pgmap v284: 161 pgs: 161 active+clean; 585 KiB data, 266 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:28:29.804 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:28:29 smithi069 ceph-mon[83680]: from='client.? 172.21.15.69:0/3837632856' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-14T14:28:29.804 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:28:29 smithi069 ceph-mon[88636]: from='client.25417 -' entity='client.iscsi.foo.smithi069.yuumtr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T14:28:29.804 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:28:29 smithi069 ceph-mon[88636]: pgmap v284: 161 pgs: 161 active+clean; 585 KiB data, 266 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:28:29.804 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:28:29 smithi069 ceph-mon[88636]: from='client.? 172.21.15.69:0/3837632856' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-14T14:28:30.094 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:28:29 smithi122 ceph-mon[67998]: from='client.25417 -' entity='client.iscsi.foo.smithi069.yuumtr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T14:28:30.094 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:28:29 smithi122 ceph-mon[67998]: pgmap v284: 161 pgs: 161 active+clean; 585 KiB data, 266 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:28:30.094 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:28:29 smithi122 ceph-mon[67998]: from='client.? 172.21.15.69:0/3837632856' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-14T14:28:30.101 DEBUG:teuthology.orchestra.run.smithi069:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid dc326fc2-fa65-11ee-bc8e-c7b262605968 -e sha1=a971fd5f5719038c70b817bafacd6d5400b1d888 -- bash -c 'ceph orch upgrade start --image quay.ceph.io/ceph-ci/ceph:$sha1' 2024-04-14T14:28:31.344 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Apr 14 14:28:30 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-grafana-a[66394]: logger=cleanup t=2024-04-14T14:28:30.922764542Z level=info msg="Completed cleanup jobs" duration=2.431543ms 2024-04-14T14:28:31.344 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Apr 14 14:28:30 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-grafana-a[66394]: logger=sqlstore.transactions t=2024-04-14T14:28:30.930966655Z level=info msg="Database locked, sleeping then retrying" error="database is locked" retry=0 code="database is locked" 2024-04-14T14:28:32.056 INFO:teuthology.orchestra.run.smithi069.stdout:Initiating upgrade to quay.ceph.io/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888 2024-04-14T14:28:32.056 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:28:31 smithi069 ceph-mon[83680]: pgmap v285: 161 pgs: 161 active+clean; 585 KiB data, 266 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:28:32.057 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:28:31 smithi069 ceph-mon[88636]: pgmap v285: 161 pgs: 161 active+clean; 585 KiB data, 266 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:28:32.094 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:28:31 smithi122 ceph-mon[67998]: pgmap v285: 161 pgs: 161 active+clean; 585 KiB data, 266 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:28:32.654 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-04-14T14:28:32.665 INFO:tasks.cephadm:Running commands on role mon.a host ubuntu@smithi069.front.sepia.ceph.com 2024-04-14T14:28:32.665 DEBUG:teuthology.orchestra.run.smithi069:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid dc326fc2-fa65-11ee-bc8e-c7b262605968 -e sha1=a971fd5f5719038c70b817bafacd6d5400b1d888 -- bash -c 'while ceph orch upgrade status | jq '"'"'.in_progress'"'"' | grep true && ! ceph orch upgrade status | jq '"'"'.message'"'"' | grep Error ; do ceph orch ps ; ceph versions ; ceph orch upgrade status ; ceph health detail ; sleep 30 ; done' 2024-04-14T14:28:33.288 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:28:33 smithi069 ceph-mon[88636]: from='client.54736 -' entity='client.admin' cmd=[{"prefix": "orch upgrade start", "image": "quay.ceph.io/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T14:28:33.288 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:28:33 smithi069 ceph-mon[88636]: Upgrade: Started with target quay.ceph.io/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888 2024-04-14T14:28:33.288 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:28:33 smithi069 ceph-mon[88636]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:28:33.288 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:28:33 smithi069 ceph-mon[88636]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T14:28:33.288 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:28:33 smithi069 ceph-mon[88636]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-14T14:28:33.288 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:28:33 smithi069 ceph-mon[88636]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:28:33.288 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:28:33 smithi069 ceph-mon[88636]: Upgrade: First pull of quay.ceph.io/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888 2024-04-14T14:28:33.288 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:28:33 smithi069 ceph-mon[88636]: pgmap v286: 161 pgs: 161 active+clean; 585 KiB data, 266 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:28:33.289 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:28:33 smithi069 ceph-mon[83680]: from='client.54736 -' entity='client.admin' cmd=[{"prefix": "orch upgrade start", "image": "quay.ceph.io/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T14:28:33.289 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:28:33 smithi069 ceph-mon[83680]: Upgrade: Started with target quay.ceph.io/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888 2024-04-14T14:28:33.289 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:28:33 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:28:33.289 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:28:33 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T14:28:33.289 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:28:33 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-14T14:28:33.289 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:28:33 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:28:33.289 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:28:33 smithi069 ceph-mon[83680]: Upgrade: First pull of quay.ceph.io/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888 2024-04-14T14:28:33.289 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:28:33 smithi069 ceph-mon[83680]: pgmap v286: 161 pgs: 161 active+clean; 585 KiB data, 266 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:28:33.344 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:28:33 smithi122 ceph-mon[67998]: from='client.54736 -' entity='client.admin' cmd=[{"prefix": "orch upgrade start", "image": "quay.ceph.io/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T14:28:33.344 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:28:33 smithi122 ceph-mon[67998]: Upgrade: Started with target quay.ceph.io/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888 2024-04-14T14:28:33.345 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:28:33 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:28:33.345 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:28:33 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T14:28:33.345 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:28:33 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-14T14:28:33.345 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:28:33 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:28:33.345 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:28:33 smithi122 ceph-mon[67998]: Upgrade: First pull of quay.ceph.io/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888 2024-04-14T14:28:33.345 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:28:33 smithi122 ceph-mon[67998]: pgmap v286: 161 pgs: 161 active+clean; 585 KiB data, 266 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:28:34.874 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:28:34 smithi069 ceph-mon[88636]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:28:34.875 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:28:34 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:28:34.875 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:28:34 smithi069 ceph-mon[83680]: Upgrade: Target is version 19.0.0-2778-ga971fd5f (squid) 2024-04-14T14:28:34.875 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:28:34 smithi069 ceph-mon[83680]: Upgrade: Target container is quay.ceph.io/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888, digests ['quay.ceph.io/ceph-ci/ceph@sha256:02f0a191d4d693952654c64f93cc9c0152602c0915251b3c94c2c4e32855c608'] 2024-04-14T14:28:34.875 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:28:34 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-14T14:28:34.875 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:28:34 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T14:28:35.147 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:28:34 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:28:35.147 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:28:34 smithi122 ceph-mon[67998]: Upgrade: Target is version 19.0.0-2778-ga971fd5f (squid) 2024-04-14T14:28:35.147 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:28:34 smithi122 ceph-mon[67998]: Upgrade: Target container is quay.ceph.io/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888, digests ['quay.ceph.io/ceph-ci/ceph@sha256:02f0a191d4d693952654c64f93cc9c0152602c0915251b3c94c2c4e32855c608'] 2024-04-14T14:28:35.147 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:28:34 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-14T14:28:35.147 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:28:34 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T14:28:35.147 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:28:34 smithi122 ceph-mon[67998]: Upgrade: Setting container_image for all mgr 2024-04-14T14:28:35.147 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:28:34 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:28:35.147 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:28:34 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T14:28:35.147 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:28:34 smithi122 ceph-mon[67998]: Upgrade: Setting container_image for all mon 2024-04-14T14:28:35.147 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:28:34 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:28:35.147 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:28:34 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T14:28:35.147 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:28:34 smithi122 ceph-mon[67998]: Upgrade: Setting container_image for all crash 2024-04-14T14:28:35.147 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:28:34 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:28:35.148 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:28:34 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T14:28:35.148 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:28:34 smithi122 ceph-mon[67998]: Upgrade: Setting container_image for all osd 2024-04-14T14:28:35.148 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:28:34 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:28:35.148 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:28:34 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T14:28:35.148 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:28:34 smithi122 ceph-mon[67998]: Upgrade: Setting container_image for all mds 2024-04-14T14:28:35.148 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:28:34 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:28:35.148 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:28:34 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T14:28:35.148 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:28:34 smithi122 ceph-mon[67998]: Upgrade: Setting container_image for all rgw 2024-04-14T14:28:35.148 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:28:34 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:28:35.148 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:28:34 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T14:28:35.148 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:28:34 smithi122 ceph-mon[67998]: Upgrade: Setting container_image for all rbd-mirror 2024-04-14T14:28:35.148 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:28:34 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:28:35.148 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:28:34 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T14:28:35.148 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:28:34 smithi122 ceph-mon[67998]: Upgrade: Setting container_image for all cephfs-mirror 2024-04-14T14:28:35.149 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:28:34 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T14:28:35.149 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:28:34 smithi122 ceph-mon[67998]: Upgrade: Setting container_image for all ceph-exporter 2024-04-14T14:28:35.149 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:28:34 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:28:35.149 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:28:34 smithi122 ceph-mon[67998]: pgmap v287: 161 pgs: 161 active+clean; 585 KiB data, 266 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:28:35.264 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:28:34 smithi069 ceph-mon[88636]: Upgrade: Target is version 19.0.0-2778-ga971fd5f (squid) 2024-04-14T14:28:35.264 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:28:34 smithi069 ceph-mon[88636]: Upgrade: Target container is quay.ceph.io/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888, digests ['quay.ceph.io/ceph-ci/ceph@sha256:02f0a191d4d693952654c64f93cc9c0152602c0915251b3c94c2c4e32855c608'] 2024-04-14T14:28:35.264 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:28:34 smithi069 ceph-mon[88636]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-14T14:28:35.264 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:28:34 smithi069 ceph-mon[88636]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T14:28:35.264 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:28:34 smithi069 ceph-mon[88636]: Upgrade: Setting container_image for all mgr 2024-04-14T14:28:35.264 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:28:34 smithi069 ceph-mon[88636]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:28:35.264 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:28:34 smithi069 ceph-mon[88636]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T14:28:35.264 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:28:34 smithi069 ceph-mon[88636]: Upgrade: Setting container_image for all mon 2024-04-14T14:28:35.264 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:28:34 smithi069 ceph-mon[88636]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:28:35.264 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:28:34 smithi069 ceph-mon[88636]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T14:28:35.265 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:28:34 smithi069 ceph-mon[88636]: Upgrade: Setting container_image for all crash 2024-04-14T14:28:35.265 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:28:34 smithi069 ceph-mon[88636]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:28:35.265 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:28:34 smithi069 ceph-mon[88636]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T14:28:35.265 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:28:34 smithi069 ceph-mon[88636]: Upgrade: Setting container_image for all osd 2024-04-14T14:28:35.265 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:28:34 smithi069 ceph-mon[88636]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:28:35.265 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:28:34 smithi069 ceph-mon[88636]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T14:28:35.265 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:28:34 smithi069 ceph-mon[88636]: Upgrade: Setting container_image for all mds 2024-04-14T14:28:35.265 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:28:34 smithi069 ceph-mon[88636]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:28:35.265 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:28:34 smithi069 ceph-mon[88636]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T14:28:35.265 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:28:34 smithi069 ceph-mon[88636]: Upgrade: Setting container_image for all rgw 2024-04-14T14:28:35.265 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:28:34 smithi069 ceph-mon[88636]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:28:35.265 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:28:34 smithi069 ceph-mon[88636]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T14:28:35.265 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:28:34 smithi069 ceph-mon[88636]: Upgrade: Setting container_image for all rbd-mirror 2024-04-14T14:28:35.265 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:28:34 smithi069 ceph-mon[88636]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:28:35.265 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:28:34 smithi069 ceph-mon[88636]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T14:28:35.266 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:28:34 smithi069 ceph-mon[88636]: Upgrade: Setting container_image for all cephfs-mirror 2024-04-14T14:28:35.266 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:28:34 smithi069 ceph-mon[88636]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T14:28:35.266 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:28:34 smithi069 ceph-mon[88636]: Upgrade: Setting container_image for all ceph-exporter 2024-04-14T14:28:35.266 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:28:34 smithi069 ceph-mon[88636]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:28:35.266 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:28:34 smithi069 ceph-mon[88636]: pgmap v287: 161 pgs: 161 active+clean; 585 KiB data, 266 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:28:35.266 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:28:34 smithi069 ceph-mon[83680]: Upgrade: Setting container_image for all mgr 2024-04-14T14:28:35.266 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:28:34 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:28:35.266 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:28:34 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T14:28:35.266 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:28:34 smithi069 ceph-mon[83680]: Upgrade: Setting container_image for all mon 2024-04-14T14:28:35.266 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:28:34 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:28:35.266 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:28:34 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T14:28:35.267 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:28:34 smithi069 ceph-mon[83680]: Upgrade: Setting container_image for all crash 2024-04-14T14:28:35.267 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:28:34 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:28:35.267 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:28:34 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T14:28:35.267 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:28:34 smithi069 ceph-mon[83680]: Upgrade: Setting container_image for all osd 2024-04-14T14:28:35.267 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:28:34 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:28:35.267 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:28:34 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T14:28:35.267 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:28:34 smithi069 ceph-mon[83680]: Upgrade: Setting container_image for all mds 2024-04-14T14:28:35.267 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:28:34 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:28:35.267 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:28:34 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T14:28:35.267 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:28:34 smithi069 ceph-mon[83680]: Upgrade: Setting container_image for all rgw 2024-04-14T14:28:35.267 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:28:34 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:28:35.267 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:28:34 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T14:28:35.267 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:28:34 smithi069 ceph-mon[83680]: Upgrade: Setting container_image for all rbd-mirror 2024-04-14T14:28:35.268 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:28:34 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:28:35.268 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:28:34 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T14:28:35.268 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:28:34 smithi069 ceph-mon[83680]: Upgrade: Setting container_image for all cephfs-mirror 2024-04-14T14:28:35.268 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:28:34 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T14:28:35.268 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:28:34 smithi069 ceph-mon[83680]: Upgrade: Setting container_image for all ceph-exporter 2024-04-14T14:28:35.268 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:28:34 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:28:35.268 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:28:34 smithi069 ceph-mon[83680]: pgmap v287: 161 pgs: 161 active+clean; 585 KiB data, 266 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:28:36.668 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 14 14:28:36 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-y[63038]: ::ffff:172.21.15.122 - - [14/Apr/2024:14:28:36] "GET /metrics HTTP/1.1" 200 38412 "" "Prometheus/2.43.0" 2024-04-14T14:28:36.809 INFO:teuthology.orchestra.run.smithi069.stdout:true 2024-04-14T14:28:37.316 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:28:37 smithi069 ceph-mon[83680]: Upgrade: Updating iscsi.foo.smithi069.yuumtr 2024-04-14T14:28:37.316 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:28:37 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:28:37.316 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:28:37 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi069.yuumtr", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-04-14T14:28:37.316 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:28:37 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T14:28:37.316 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:28:37 smithi069 ceph-mon[83680]: Deploying daemon iscsi.foo.smithi069.yuumtr on smithi069 2024-04-14T14:28:37.316 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:28:37 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T14:28:37.317 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:28:37 smithi069 ceph-mon[83680]: pgmap v288: 161 pgs: 161 active+clean; 585 KiB data, 266 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:28:37.317 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:28:37 smithi069 ceph-mon[88636]: Upgrade: Updating iscsi.foo.smithi069.yuumtr 2024-04-14T14:28:37.317 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:28:37 smithi069 ceph-mon[88636]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:28:37.317 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:28:37 smithi069 ceph-mon[88636]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi069.yuumtr", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-04-14T14:28:37.317 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:28:37 smithi069 ceph-mon[88636]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T14:28:37.317 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:28:37 smithi069 ceph-mon[88636]: Deploying daemon iscsi.foo.smithi069.yuumtr on smithi069 2024-04-14T14:28:37.317 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:28:37 smithi069 ceph-mon[88636]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T14:28:37.317 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:28:37 smithi069 ceph-mon[88636]: pgmap v288: 161 pgs: 161 active+clean; 585 KiB data, 266 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:28:37.344 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:28:37 smithi122 ceph-mon[67998]: Upgrade: Updating iscsi.foo.smithi069.yuumtr 2024-04-14T14:28:37.344 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:28:37 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:28:37.344 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:28:37 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi069.yuumtr", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-04-14T14:28:37.344 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:28:37 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T14:28:37.344 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:28:37 smithi122 ceph-mon[67998]: Deploying daemon iscsi.foo.smithi069.yuumtr on smithi069 2024-04-14T14:28:37.344 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:28:37 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T14:28:37.344 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:28:37 smithi122 ceph-mon[67998]: pgmap v288: 161 pgs: 161 active+clean; 585 KiB data, 266 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:28:37.474 INFO:teuthology.orchestra.run.smithi069.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-14T14:28:37.474 INFO:teuthology.orchestra.run.smithi069.stdout:alertmanager.a smithi069 *:9093,9094 running (10m) 39s ago 32m 25.9M - 0.25.0 c8568f914cd2 7524f106e7bd 2024-04-14T14:28:37.474 INFO:teuthology.orchestra.run.smithi069.stdout:grafana.a smithi122 *:3000 running (10m) 39s ago 31m 82.4M - 9.4.12 2bacad6d85d8 5685178e3eba 2024-04-14T14:28:37.475 INFO:teuthology.orchestra.run.smithi069.stdout:iscsi.foo.smithi069.yuumtr smithi069 running (11m) 39s ago 31m 62.9M - 3.5 e1d6a67b021e 38cc81ea8cfa 2024-04-14T14:28:37.475 INFO:teuthology.orchestra.run.smithi069.stdout:mgr.x smithi122 *:8443,9283,8765 running (11m) 39s ago 36m 455M - 19.0.0-2778-ga971fd5f b96c509eef84 e29977590b43 2024-04-14T14:28:37.475 INFO:teuthology.orchestra.run.smithi069.stdout:mgr.y smithi069 *:8443,9283,8765 running (21m) 39s ago 38m 513M - 19.0.0-2778-ga971fd5f b96c509eef84 7af3d29d0e5f 2024-04-14T14:28:37.475 INFO:teuthology.orchestra.run.smithi069.stdout:mon.a smithi069 running (8m) 39s ago 38m 63.2M 2048M 19.0.0-2778-ga971fd5f b96c509eef84 8bcd6b3d3846 2024-04-14T14:28:37.475 INFO:teuthology.orchestra.run.smithi069.stdout:mon.b smithi122 running (8m) 39s ago 36m 49.5M 2048M 19.0.0-2778-ga971fd5f b96c509eef84 05f437a1a9a1 2024-04-14T14:28:37.475 INFO:teuthology.orchestra.run.smithi069.stdout:mon.c smithi069 running (7m) 39s ago 36m 44.8M 2048M 19.0.0-2778-ga971fd5f b96c509eef84 84aa95f11d0c 2024-04-14T14:28:37.475 INFO:teuthology.orchestra.run.smithi069.stdout:node-exporter.a smithi069 *:9100 running (11m) 39s ago 32m 15.7M - 1.5.0 0da6a335fe13 0de0eb2171b9 2024-04-14T14:28:37.475 INFO:teuthology.orchestra.run.smithi069.stdout:node-exporter.b smithi122 *:9100 running (10m) 39s ago 32m 16.3M - 1.5.0 0da6a335fe13 67877bf02d8e 2024-04-14T14:28:37.475 INFO:teuthology.orchestra.run.smithi069.stdout:osd.0 smithi069 running (6m) 39s ago 36m 62.2M 2992M 19.0.0-2778-ga971fd5f b96c509eef84 a0e57e3fb494 2024-04-14T14:28:37.475 INFO:teuthology.orchestra.run.smithi069.stdout:osd.1 smithi069 running (5m) 39s ago 35m 62.6M 2992M 19.0.0-2778-ga971fd5f b96c509eef84 cf73d17919bb 2024-04-14T14:28:37.475 INFO:teuthology.orchestra.run.smithi069.stdout:osd.2 smithi069 running (5m) 39s ago 35m 49.7M 2992M 19.0.0-2778-ga971fd5f b96c509eef84 c0efba5ef451 2024-04-14T14:28:37.475 INFO:teuthology.orchestra.run.smithi069.stdout:osd.3 smithi069 running (4m) 39s ago 35m 60.6M 2992M 19.0.0-2778-ga971fd5f b96c509eef84 d13460da0a65 2024-04-14T14:28:37.475 INFO:teuthology.orchestra.run.smithi069.stdout:osd.4 smithi122 running (3m) 39s ago 34m 58.6M 3248M 19.0.0-2778-ga971fd5f b96c509eef84 7a87a024f984 2024-04-14T14:28:37.475 INFO:teuthology.orchestra.run.smithi069.stdout:osd.5 smithi122 running (3m) 39s ago 34m 56.7M 3248M 19.0.0-2778-ga971fd5f b96c509eef84 02b623d44db3 2024-04-14T14:28:37.475 INFO:teuthology.orchestra.run.smithi069.stdout:osd.6 smithi122 running (2m) 39s ago 33m 53.9M 3248M 19.0.0-2778-ga971fd5f b96c509eef84 72f85ece3e75 2024-04-14T14:28:37.476 INFO:teuthology.orchestra.run.smithi069.stdout:osd.7 smithi122 running (117s) 39s ago 33m 55.3M 3248M 19.0.0-2778-ga971fd5f b96c509eef84 0c53c3cd2bc0 2024-04-14T14:28:37.476 INFO:teuthology.orchestra.run.smithi069.stdout:prometheus.a smithi122 *:9095 running (10m) 39s ago 32m 71.1M - 2.43.0 a07b618ecd1d 5c1f41d3bbe6 2024-04-14T14:28:37.476 INFO:teuthology.orchestra.run.smithi069.stdout:rgw.foo.smithi069.wesedz smithi069 *:8000 running (49s) 39s ago 31m 77.2M - 19.0.0-2778-ga971fd5f b96c509eef84 f135fd86285d 2024-04-14T14:28:37.476 INFO:teuthology.orchestra.run.smithi069.stdout:rgw.foo.smithi122.jiobyv smithi122 *:8000 running (43s) 39s ago 31m 76.3M - 19.0.0-2778-ga971fd5f b96c509eef84 7d37b0b84199 2024-04-14T14:28:37.878 INFO:teuthology.orchestra.run.smithi069.stdout:{ 2024-04-14T14:28:37.878 INFO:teuthology.orchestra.run.smithi069.stdout: "mon": { 2024-04-14T14:28:37.878 INFO:teuthology.orchestra.run.smithi069.stdout: "ceph version 19.0.0-2778-ga971fd5f (a971fd5f5719038c70b817bafacd6d5400b1d888) squid (dev)": 3 2024-04-14T14:28:37.878 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-14T14:28:37.878 INFO:teuthology.orchestra.run.smithi069.stdout: "mgr": { 2024-04-14T14:28:37.878 INFO:teuthology.orchestra.run.smithi069.stdout: "ceph version 19.0.0-2778-ga971fd5f (a971fd5f5719038c70b817bafacd6d5400b1d888) squid (dev)": 2 2024-04-14T14:28:37.879 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-14T14:28:37.879 INFO:teuthology.orchestra.run.smithi069.stdout: "osd": { 2024-04-14T14:28:37.879 INFO:teuthology.orchestra.run.smithi069.stdout: "ceph version 19.0.0-2778-ga971fd5f (a971fd5f5719038c70b817bafacd6d5400b1d888) squid (dev)": 8 2024-04-14T14:28:37.879 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-14T14:28:37.879 INFO:teuthology.orchestra.run.smithi069.stdout: "rgw": { 2024-04-14T14:28:37.879 INFO:teuthology.orchestra.run.smithi069.stdout: "ceph version 19.0.0-2778-ga971fd5f (a971fd5f5719038c70b817bafacd6d5400b1d888) squid (dev)": 2 2024-04-14T14:28:37.879 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-14T14:28:37.879 INFO:teuthology.orchestra.run.smithi069.stdout: "overall": { 2024-04-14T14:28:37.879 INFO:teuthology.orchestra.run.smithi069.stdout: "ceph version 19.0.0-2778-ga971fd5f (a971fd5f5719038c70b817bafacd6d5400b1d888) squid (dev)": 15 2024-04-14T14:28:37.879 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-14T14:28:37.879 INFO:teuthology.orchestra.run.smithi069.stdout:} 2024-04-14T14:28:38.244 INFO:teuthology.orchestra.run.smithi069.stdout:{ 2024-04-14T14:28:38.244 INFO:teuthology.orchestra.run.smithi069.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888", 2024-04-14T14:28:38.244 INFO:teuthology.orchestra.run.smithi069.stdout: "in_progress": true, 2024-04-14T14:28:38.244 INFO:teuthology.orchestra.run.smithi069.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-14T14:28:38.244 INFO:teuthology.orchestra.run.smithi069.stdout: "services_complete": [ 2024-04-14T14:28:38.244 INFO:teuthology.orchestra.run.smithi069.stdout: "rgw", 2024-04-14T14:28:38.244 INFO:teuthology.orchestra.run.smithi069.stdout: "mon", 2024-04-14T14:28:38.244 INFO:teuthology.orchestra.run.smithi069.stdout: "osd", 2024-04-14T14:28:38.244 INFO:teuthology.orchestra.run.smithi069.stdout: "mgr" 2024-04-14T14:28:38.245 INFO:teuthology.orchestra.run.smithi069.stdout: ], 2024-04-14T14:28:38.245 INFO:teuthology.orchestra.run.smithi069.stdout: "progress": "15/21 daemons upgraded", 2024-04-14T14:28:38.245 INFO:teuthology.orchestra.run.smithi069.stdout: "message": "Currently upgrading iscsi daemons", 2024-04-14T14:28:38.245 INFO:teuthology.orchestra.run.smithi069.stdout: "is_paused": false 2024-04-14T14:28:38.245 INFO:teuthology.orchestra.run.smithi069.stdout:} 2024-04-14T14:28:38.316 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:28:38 smithi069 ceph-mon[83680]: from='client.54742 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T14:28:38.316 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:28:38 smithi069 ceph-mon[83680]: from='client.54748 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T14:28:38.316 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:28:38 smithi069 ceph-mon[83680]: from='client.44811 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T14:28:38.316 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:28:38 smithi069 ceph-mon[83680]: from='client.? 172.21.15.69:0/3573162170' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T14:28:38.316 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:28:38 smithi069 ceph-mon[88636]: from='client.54742 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T14:28:38.317 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:28:38 smithi069 ceph-mon[88636]: from='client.54748 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T14:28:38.317 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:28:38 smithi069 ceph-mon[88636]: from='client.44811 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T14:28:38.317 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:28:38 smithi069 ceph-mon[88636]: from='client.? 172.21.15.69:0/3573162170' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T14:28:38.344 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:28:38 smithi122 ceph-mon[67998]: from='client.54742 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T14:28:38.344 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:28:38 smithi122 ceph-mon[67998]: from='client.54748 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T14:28:38.344 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:28:38 smithi122 ceph-mon[67998]: from='client.44811 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T14:28:38.344 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:28:38 smithi122 ceph-mon[67998]: from='client.? 172.21.15.69:0/3573162170' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T14:28:38.658 INFO:teuthology.orchestra.run.smithi069.stdout:HEALTH_OK 2024-04-14T14:28:39.316 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:28:39 smithi069 ceph-mon[83680]: from='client.25417 -' entity='client.iscsi.foo.smithi069.yuumtr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T14:28:39.316 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:28:39 smithi069 ceph-mon[83680]: from='client.54766 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T14:28:39.316 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:28:39 smithi069 ceph-mon[83680]: pgmap v289: 161 pgs: 161 active+clean; 585 KiB data, 266 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:28:39.316 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:28:39 smithi069 ceph-mon[83680]: from='client.? 172.21.15.69:0/4257042674' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-14T14:28:39.317 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:28:39 smithi069 ceph-mon[88636]: from='client.25417 -' entity='client.iscsi.foo.smithi069.yuumtr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T14:28:39.317 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:28:39 smithi069 ceph-mon[88636]: from='client.54766 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T14:28:39.317 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:28:39 smithi069 ceph-mon[88636]: pgmap v289: 161 pgs: 161 active+clean; 585 KiB data, 266 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:28:39.317 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:28:39 smithi069 ceph-mon[88636]: from='client.? 172.21.15.69:0/4257042674' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-14T14:28:39.344 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:28:39 smithi122 ceph-mon[67998]: from='client.25417 -' entity='client.iscsi.foo.smithi069.yuumtr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T14:28:39.344 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:28:39 smithi122 ceph-mon[67998]: from='client.54766 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T14:28:39.344 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:28:39 smithi122 ceph-mon[67998]: pgmap v289: 161 pgs: 161 active+clean; 585 KiB data, 266 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:28:39.344 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:28:39 smithi122 ceph-mon[67998]: from='client.? 172.21.15.69:0/4257042674' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-14T14:28:40.066 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 14 14:28:39 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-y[63038]: 2024-04-14T14:28:39.728+0000 7fdd08bbf700 -1 log_channel(cephadm) log [ERR] : Upgrade: Paused due to UPGRADE_REDEPLOY_DAEMON: Upgrading daemon iscsi.foo.smithi069.yuumtr on host smithi069 failed. 2024-04-14T14:28:41.067 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:28:40 smithi069 ceph-mon[83680]: Upgrade: Paused due to UPGRADE_REDEPLOY_DAEMON: Upgrading daemon iscsi.foo.smithi069.yuumtr on host smithi069 failed. 2024-04-14T14:28:41.067 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:28:40 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:28:41.067 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:28:40 smithi069 ceph-mon[83680]: pgmap v290: 161 pgs: 161 active+clean; 585 KiB data, 266 MiB used, 715 GiB / 715 GiB avail; 923 B/s rd, 0 op/s 2024-04-14T14:28:41.067 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:28:40 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T14:28:41.067 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:28:40 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-14T14:28:41.067 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:28:40 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:28:41.067 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:28:40 smithi069 ceph-mon[88636]: Upgrade: Paused due to UPGRADE_REDEPLOY_DAEMON: Upgrading daemon iscsi.foo.smithi069.yuumtr on host smithi069 failed. 2024-04-14T14:28:41.067 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:28:40 smithi069 ceph-mon[88636]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:28:41.067 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:28:40 smithi069 ceph-mon[88636]: pgmap v290: 161 pgs: 161 active+clean; 585 KiB data, 266 MiB used, 715 GiB / 715 GiB avail; 923 B/s rd, 0 op/s 2024-04-14T14:28:41.068 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:28:40 smithi069 ceph-mon[88636]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T14:28:41.068 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:28:40 smithi069 ceph-mon[88636]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-14T14:28:41.068 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:28:40 smithi069 ceph-mon[88636]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:28:41.094 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:28:40 smithi122 ceph-mon[67998]: Upgrade: Paused due to UPGRADE_REDEPLOY_DAEMON: Upgrading daemon iscsi.foo.smithi069.yuumtr on host smithi069 failed. 2024-04-14T14:28:41.094 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:28:40 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:28:41.094 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:28:40 smithi122 ceph-mon[67998]: pgmap v290: 161 pgs: 161 active+clean; 585 KiB data, 266 MiB used, 715 GiB / 715 GiB avail; 923 B/s rd, 0 op/s 2024-04-14T14:28:41.094 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:28:40 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T14:28:41.094 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:28:40 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-14T14:28:41.094 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:28:40 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:28:42.066 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:28:41 smithi069 ceph-mon[83680]: Health check failed: Upgrading daemon iscsi.foo.smithi069.yuumtr on host smithi069 failed. (UPGRADE_REDEPLOY_DAEMON) 2024-04-14T14:28:42.066 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:28:41 smithi069 ceph-mon[88636]: Health check failed: Upgrading daemon iscsi.foo.smithi069.yuumtr on host smithi069 failed. (UPGRADE_REDEPLOY_DAEMON) 2024-04-14T14:28:42.094 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:28:41 smithi122 ceph-mon[67998]: Health check failed: Upgrading daemon iscsi.foo.smithi069.yuumtr on host smithi069 failed. (UPGRADE_REDEPLOY_DAEMON) 2024-04-14T14:28:43.066 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:28:42 smithi069 ceph-mon[83680]: pgmap v291: 161 pgs: 161 active+clean; 585 KiB data, 266 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:28:43.066 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:28:42 smithi069 ceph-mon[88636]: pgmap v291: 161 pgs: 161 active+clean; 585 KiB data, 266 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:28:43.094 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:28:42 smithi122 ceph-mon[67998]: pgmap v291: 161 pgs: 161 active+clean; 585 KiB data, 266 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:28:45.066 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:28:44 smithi069 ceph-mon[83680]: pgmap v292: 161 pgs: 161 active+clean; 585 KiB data, 266 MiB used, 715 GiB / 715 GiB avail; 738 B/s rd, 0 op/s 2024-04-14T14:28:45.066 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:28:44 smithi069 ceph-mon[88636]: pgmap v292: 161 pgs: 161 active+clean; 585 KiB data, 266 MiB used, 715 GiB / 715 GiB avail; 738 B/s rd, 0 op/s 2024-04-14T14:28:45.094 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:28:44 smithi122 ceph-mon[67998]: pgmap v292: 161 pgs: 161 active+clean; 585 KiB data, 266 MiB used, 715 GiB / 715 GiB avail; 738 B/s rd, 0 op/s 2024-04-14T14:28:46.816 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:28:46 smithi069 ceph-mon[83680]: pgmap v293: 161 pgs: 161 active+clean; 585 KiB data, 266 MiB used, 715 GiB / 715 GiB avail; 738 B/s rd, 0 op/s 2024-04-14T14:28:46.816 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:28:46 smithi069 ceph-mon[88636]: pgmap v293: 161 pgs: 161 active+clean; 585 KiB data, 266 MiB used, 715 GiB / 715 GiB avail; 738 B/s rd, 0 op/s 2024-04-14T14:28:46.817 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 14 14:28:46 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-y[63038]: ::ffff:172.21.15.122 - - [14/Apr/2024:14:28:46] "GET /metrics HTTP/1.1" 200 38411 "" "Prometheus/2.43.0" 2024-04-14T14:28:47.097 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:28:46 smithi122 ceph-mon[67998]: pgmap v293: 161 pgs: 161 active+clean; 585 KiB data, 266 MiB used, 715 GiB / 715 GiB avail; 738 B/s rd, 0 op/s 2024-04-14T14:28:49.519 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:28:49 smithi069 ceph-mon[83680]: pgmap v294: 161 pgs: 161 active+clean; 585 KiB data, 266 MiB used, 715 GiB / 715 GiB avail; 738 B/s rd, 0 op/s 2024-04-14T14:28:49.520 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:28:49 smithi069 ceph-mon[88636]: pgmap v294: 161 pgs: 161 active+clean; 585 KiB data, 266 MiB used, 715 GiB / 715 GiB avail; 738 B/s rd, 0 op/s 2024-04-14T14:28:49.594 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:28:49 smithi122 ceph-mon[67998]: pgmap v294: 161 pgs: 161 active+clean; 585 KiB data, 266 MiB used, 715 GiB / 715 GiB avail; 738 B/s rd, 0 op/s 2024-04-14T14:28:51.566 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:28:51 smithi069 ceph-mon[83680]: pgmap v295: 161 pgs: 161 active+clean; 585 KiB data, 266 MiB used, 715 GiB / 715 GiB avail; 277 B/s rd, 0 op/s 2024-04-14T14:28:51.566 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:28:51 smithi069 ceph-mon[83680]: from='client.? 172.21.15.69:0/3938789734' entity='client.iscsi.foo.smithi069.yuumtr' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-04-14T14:28:51.567 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:28:51 smithi069 ceph-mon[83680]: from='client.? 172.21.15.69:0/1416815995' entity='client.iscsi.foo.smithi069.yuumtr' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.69:0/2045430248"}]: dispatch 2024-04-14T14:28:51.567 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:28:51 smithi069 ceph-mon[83680]: from='client.? ' entity='client.iscsi.foo.smithi069.yuumtr' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.69:0/2045430248"}]: dispatch 2024-04-14T14:28:51.567 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:28:51 smithi069 ceph-mon[88636]: pgmap v295: 161 pgs: 161 active+clean; 585 KiB data, 266 MiB used, 715 GiB / 715 GiB avail; 277 B/s rd, 0 op/s 2024-04-14T14:28:51.567 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:28:51 smithi069 ceph-mon[88636]: from='client.? 172.21.15.69:0/3938789734' entity='client.iscsi.foo.smithi069.yuumtr' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-04-14T14:28:51.567 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:28:51 smithi069 ceph-mon[88636]: from='client.? 172.21.15.69:0/1416815995' entity='client.iscsi.foo.smithi069.yuumtr' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.69:0/2045430248"}]: dispatch 2024-04-14T14:28:51.567 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:28:51 smithi069 ceph-mon[88636]: from='client.? ' entity='client.iscsi.foo.smithi069.yuumtr' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.69:0/2045430248"}]: dispatch 2024-04-14T14:28:51.594 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:28:51 smithi122 ceph-mon[67998]: pgmap v295: 161 pgs: 161 active+clean; 585 KiB data, 266 MiB used, 715 GiB / 715 GiB avail; 277 B/s rd, 0 op/s 2024-04-14T14:28:51.594 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:28:51 smithi122 ceph-mon[67998]: from='client.? 172.21.15.69:0/3938789734' entity='client.iscsi.foo.smithi069.yuumtr' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-04-14T14:28:51.594 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:28:51 smithi122 ceph-mon[67998]: from='client.? 172.21.15.69:0/1416815995' entity='client.iscsi.foo.smithi069.yuumtr' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.69:0/2045430248"}]: dispatch 2024-04-14T14:28:51.594 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:28:51 smithi122 ceph-mon[67998]: from='client.? ' entity='client.iscsi.foo.smithi069.yuumtr' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.69:0/2045430248"}]: dispatch 2024-04-14T14:28:52.566 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:28:52 smithi069 ceph-mon[83680]: from='client.? ' entity='client.iscsi.foo.smithi069.yuumtr' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.69:0/2045430248"}]': finished 2024-04-14T14:28:52.567 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:28:52 smithi069 ceph-mon[83680]: osdmap e157: 8 total, 8 up, 8 in 2024-04-14T14:28:52.567 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:28:52 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:28:52.567 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:28:52 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T14:28:52.567 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:28:52 smithi069 ceph-mon[83680]: from='client.? 172.21.15.69:0/165239257' entity='client.iscsi.foo.smithi069.yuumtr' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.69:0/958383841"}]: dispatch 2024-04-14T14:28:52.567 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:28:52 smithi069 ceph-mon[88636]: from='client.? ' entity='client.iscsi.foo.smithi069.yuumtr' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.69:0/2045430248"}]': finished 2024-04-14T14:28:52.567 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:28:52 smithi069 ceph-mon[88636]: osdmap e157: 8 total, 8 up, 8 in 2024-04-14T14:28:52.567 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:28:52 smithi069 ceph-mon[88636]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:28:52.567 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:28:52 smithi069 ceph-mon[88636]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T14:28:52.567 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:28:52 smithi069 ceph-mon[88636]: from='client.? 172.21.15.69:0/165239257' entity='client.iscsi.foo.smithi069.yuumtr' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.69:0/958383841"}]: dispatch 2024-04-14T14:28:52.594 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:28:52 smithi122 ceph-mon[67998]: from='client.? ' entity='client.iscsi.foo.smithi069.yuumtr' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.69:0/2045430248"}]': finished 2024-04-14T14:28:52.594 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:28:52 smithi122 ceph-mon[67998]: osdmap e157: 8 total, 8 up, 8 in 2024-04-14T14:28:52.594 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:28:52 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:28:52.594 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:28:52 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T14:28:52.594 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:28:52 smithi122 ceph-mon[67998]: from='client.? 172.21.15.69:0/165239257' entity='client.iscsi.foo.smithi069.yuumtr' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.69:0/958383841"}]: dispatch 2024-04-14T14:28:53.844 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:28:53 smithi122 ceph-mon[67998]: pgmap v297: 161 pgs: 161 active+clean; 585 KiB data, 266 MiB used, 715 GiB / 715 GiB avail; 204 B/s rd, 0 op/s 2024-04-14T14:28:53.844 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:28:53 smithi122 ceph-mon[67998]: from='client.? 172.21.15.69:0/165239257' entity='client.iscsi.foo.smithi069.yuumtr' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.69:0/958383841"}]': finished 2024-04-14T14:28:53.844 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:28:53 smithi122 ceph-mon[67998]: osdmap e158: 8 total, 8 up, 8 in 2024-04-14T14:28:53.844 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:28:53 smithi122 ceph-mon[67998]: from='client.? 172.21.15.69:0/149945894' entity='client.iscsi.foo.smithi069.yuumtr' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.69:6801/1481587909"}]: dispatch 2024-04-14T14:28:54.066 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:28:53 smithi069 ceph-mon[83680]: pgmap v297: 161 pgs: 161 active+clean; 585 KiB data, 266 MiB used, 715 GiB / 715 GiB avail; 204 B/s rd, 0 op/s 2024-04-14T14:28:54.066 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:28:53 smithi069 ceph-mon[83680]: from='client.? 172.21.15.69:0/165239257' entity='client.iscsi.foo.smithi069.yuumtr' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.69:0/958383841"}]': finished 2024-04-14T14:28:54.066 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:28:53 smithi069 ceph-mon[83680]: osdmap e158: 8 total, 8 up, 8 in 2024-04-14T14:28:54.066 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:28:53 smithi069 ceph-mon[83680]: from='client.? 172.21.15.69:0/149945894' entity='client.iscsi.foo.smithi069.yuumtr' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.69:6801/1481587909"}]: dispatch 2024-04-14T14:28:54.067 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:28:53 smithi069 ceph-mon[88636]: pgmap v297: 161 pgs: 161 active+clean; 585 KiB data, 266 MiB used, 715 GiB / 715 GiB avail; 204 B/s rd, 0 op/s 2024-04-14T14:28:54.067 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:28:53 smithi069 ceph-mon[88636]: from='client.? 172.21.15.69:0/165239257' entity='client.iscsi.foo.smithi069.yuumtr' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.69:0/958383841"}]': finished 2024-04-14T14:28:54.067 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:28:53 smithi069 ceph-mon[88636]: osdmap e158: 8 total, 8 up, 8 in 2024-04-14T14:28:54.067 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:28:53 smithi069 ceph-mon[88636]: from='client.? 172.21.15.69:0/149945894' entity='client.iscsi.foo.smithi069.yuumtr' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.69:6801/1481587909"}]: dispatch 2024-04-14T14:28:55.066 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:28:54 smithi069 ceph-mon[83680]: from='client.? 172.21.15.69:0/149945894' entity='client.iscsi.foo.smithi069.yuumtr' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.69:6801/1481587909"}]': finished 2024-04-14T14:28:55.066 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:28:54 smithi069 ceph-mon[83680]: osdmap e159: 8 total, 8 up, 8 in 2024-04-14T14:28:55.066 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:28:54 smithi069 ceph-mon[83680]: from='client.? 172.21.15.69:0/2010428535' entity='client.iscsi.foo.smithi069.yuumtr' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.69:6800/1481587909"}]: dispatch 2024-04-14T14:28:55.066 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:28:54 smithi069 ceph-mon[83680]: from='client.? ' entity='client.iscsi.foo.smithi069.yuumtr' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.69:6800/1481587909"}]: dispatch 2024-04-14T14:28:55.067 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:28:54 smithi069 ceph-mon[88636]: from='client.? 172.21.15.69:0/149945894' entity='client.iscsi.foo.smithi069.yuumtr' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.69:6801/1481587909"}]': finished 2024-04-14T14:28:55.067 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:28:54 smithi069 ceph-mon[88636]: osdmap e159: 8 total, 8 up, 8 in 2024-04-14T14:28:55.067 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:28:54 smithi069 ceph-mon[88636]: from='client.? 172.21.15.69:0/2010428535' entity='client.iscsi.foo.smithi069.yuumtr' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.69:6800/1481587909"}]: dispatch 2024-04-14T14:28:55.067 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:28:54 smithi069 ceph-mon[88636]: from='client.? ' entity='client.iscsi.foo.smithi069.yuumtr' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.69:6800/1481587909"}]: dispatch 2024-04-14T14:28:55.094 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:28:54 smithi122 ceph-mon[67998]: from='client.? 172.21.15.69:0/149945894' entity='client.iscsi.foo.smithi069.yuumtr' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.69:6801/1481587909"}]': finished 2024-04-14T14:28:55.094 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:28:54 smithi122 ceph-mon[67998]: osdmap e159: 8 total, 8 up, 8 in 2024-04-14T14:28:55.094 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:28:54 smithi122 ceph-mon[67998]: from='client.? 172.21.15.69:0/2010428535' entity='client.iscsi.foo.smithi069.yuumtr' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.69:6800/1481587909"}]: dispatch 2024-04-14T14:28:55.094 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:28:54 smithi122 ceph-mon[67998]: from='client.? ' entity='client.iscsi.foo.smithi069.yuumtr' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.69:6800/1481587909"}]: dispatch 2024-04-14T14:28:56.066 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:28:55 smithi069 ceph-mon[83680]: pgmap v300: 161 pgs: 161 active+clean; 585 KiB data, 266 MiB used, 715 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2024-04-14T14:28:56.066 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:28:55 smithi069 ceph-mon[83680]: from='client.? ' entity='client.iscsi.foo.smithi069.yuumtr' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.69:6800/1481587909"}]': finished 2024-04-14T14:28:56.066 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:28:55 smithi069 ceph-mon[83680]: osdmap e160: 8 total, 8 up, 8 in 2024-04-14T14:28:56.066 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:28:55 smithi069 ceph-mon[83680]: from='client.? 172.21.15.69:0/2660015770' entity='client.iscsi.foo.smithi069.yuumtr' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.69:0/3279178559"}]: dispatch 2024-04-14T14:28:56.067 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:28:55 smithi069 ceph-mon[88636]: pgmap v300: 161 pgs: 161 active+clean; 585 KiB data, 266 MiB used, 715 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2024-04-14T14:28:56.067 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:28:55 smithi069 ceph-mon[88636]: from='client.? ' entity='client.iscsi.foo.smithi069.yuumtr' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.69:6800/1481587909"}]': finished 2024-04-14T14:28:56.067 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:28:55 smithi069 ceph-mon[88636]: osdmap e160: 8 total, 8 up, 8 in 2024-04-14T14:28:56.067 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:28:55 smithi069 ceph-mon[88636]: from='client.? 172.21.15.69:0/2660015770' entity='client.iscsi.foo.smithi069.yuumtr' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.69:0/3279178559"}]: dispatch 2024-04-14T14:28:56.094 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:28:55 smithi122 ceph-mon[67998]: pgmap v300: 161 pgs: 161 active+clean; 585 KiB data, 266 MiB used, 715 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2024-04-14T14:28:56.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:28:55 smithi122 ceph-mon[67998]: from='client.? ' entity='client.iscsi.foo.smithi069.yuumtr' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.69:6800/1481587909"}]': finished 2024-04-14T14:28:56.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:28:55 smithi122 ceph-mon[67998]: osdmap e160: 8 total, 8 up, 8 in 2024-04-14T14:28:56.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:28:55 smithi122 ceph-mon[67998]: from='client.? 172.21.15.69:0/2660015770' entity='client.iscsi.foo.smithi069.yuumtr' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.69:0/3279178559"}]: dispatch 2024-04-14T14:28:56.816 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 14 14:28:56 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-y[63038]: ::ffff:172.21.15.122 - - [14/Apr/2024:14:28:56] "GET /metrics HTTP/1.1" 200 38490 "" "Prometheus/2.43.0" 2024-04-14T14:28:56.817 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:28:56 smithi069 ceph-mon[83680]: from='client.? 172.21.15.69:0/2660015770' entity='client.iscsi.foo.smithi069.yuumtr' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.69:0/3279178559"}]': finished 2024-04-14T14:28:56.817 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:28:56 smithi069 ceph-mon[83680]: osdmap e161: 8 total, 8 up, 8 in 2024-04-14T14:28:56.817 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:28:56 smithi069 ceph-mon[83680]: from='client.? 172.21.15.69:0/222273309' entity='client.iscsi.foo.smithi069.yuumtr' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.69:0/4061531143"}]: dispatch 2024-04-14T14:28:56.817 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:28:56 smithi069 ceph-mon[83680]: from='client.? ' entity='client.iscsi.foo.smithi069.yuumtr' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.69:0/4061531143"}]: dispatch 2024-04-14T14:28:56.817 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:28:56 smithi069 ceph-mon[88636]: from='client.? 172.21.15.69:0/2660015770' entity='client.iscsi.foo.smithi069.yuumtr' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.69:0/3279178559"}]': finished 2024-04-14T14:28:56.817 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:28:56 smithi069 ceph-mon[88636]: osdmap e161: 8 total, 8 up, 8 in 2024-04-14T14:28:56.817 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:28:56 smithi069 ceph-mon[88636]: from='client.? 172.21.15.69:0/222273309' entity='client.iscsi.foo.smithi069.yuumtr' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.69:0/4061531143"}]: dispatch 2024-04-14T14:28:56.817 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:28:56 smithi069 ceph-mon[88636]: from='client.? ' entity='client.iscsi.foo.smithi069.yuumtr' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.69:0/4061531143"}]: dispatch 2024-04-14T14:28:57.094 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:28:56 smithi122 ceph-mon[67998]: from='client.? 172.21.15.69:0/2660015770' entity='client.iscsi.foo.smithi069.yuumtr' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.69:0/3279178559"}]': finished 2024-04-14T14:28:57.094 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:28:56 smithi122 ceph-mon[67998]: osdmap e161: 8 total, 8 up, 8 in 2024-04-14T14:28:57.094 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:28:56 smithi122 ceph-mon[67998]: from='client.? 172.21.15.69:0/222273309' entity='client.iscsi.foo.smithi069.yuumtr' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.69:0/4061531143"}]: dispatch 2024-04-14T14:28:57.094 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:28:56 smithi122 ceph-mon[67998]: from='client.? ' entity='client.iscsi.foo.smithi069.yuumtr' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.69:0/4061531143"}]: dispatch 2024-04-14T14:28:58.068 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:28:57 smithi069 ceph-mon[83680]: pgmap v303: 161 pgs: 161 active+clean; 585 KiB data, 266 MiB used, 715 GiB / 715 GiB avail 2024-04-14T14:28:58.069 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:28:57 smithi069 ceph-mon[83680]: from='client.? ' entity='client.iscsi.foo.smithi069.yuumtr' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.69:0/4061531143"}]': finished 2024-04-14T14:28:58.069 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:28:57 smithi069 ceph-mon[83680]: osdmap e162: 8 total, 8 up, 8 in 2024-04-14T14:28:58.070 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:28:57 smithi069 ceph-mon[88636]: pgmap v303: 161 pgs: 161 active+clean; 585 KiB data, 266 MiB used, 715 GiB / 715 GiB avail 2024-04-14T14:28:58.070 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:28:57 smithi069 ceph-mon[88636]: from='client.? ' entity='client.iscsi.foo.smithi069.yuumtr' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.69:0/4061531143"}]': finished 2024-04-14T14:28:58.070 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:28:57 smithi069 ceph-mon[88636]: osdmap e162: 8 total, 8 up, 8 in 2024-04-14T14:28:58.094 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:28:57 smithi122 ceph-mon[67998]: pgmap v303: 161 pgs: 161 active+clean; 585 KiB data, 266 MiB used, 715 GiB / 715 GiB avail 2024-04-14T14:28:58.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:28:57 smithi122 ceph-mon[67998]: from='client.? ' entity='client.iscsi.foo.smithi069.yuumtr' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.69:0/4061531143"}]': finished 2024-04-14T14:28:58.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:28:57 smithi122 ceph-mon[67998]: osdmap e162: 8 total, 8 up, 8 in 2024-04-14T14:29:00.066 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:28:59 smithi069 ceph-mon[83680]: pgmap v305: 161 pgs: 161 active+clean; 585 KiB data, 266 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:29:00.067 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:28:59 smithi069 ceph-mon[88636]: pgmap v305: 161 pgs: 161 active+clean; 585 KiB data, 266 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:29:00.094 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:28:59 smithi122 ceph-mon[67998]: pgmap v305: 161 pgs: 161 active+clean; 585 KiB data, 266 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:29:02.066 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:29:01 smithi069 ceph-mon[83680]: pgmap v306: 161 pgs: 161 active+clean; 585 KiB data, 266 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:29:02.066 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:29:01 smithi069 ceph-mon[83680]: from='client.44844 -' entity='client.iscsi.foo.smithi069.yuumtr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T14:29:02.067 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:29:01 smithi069 ceph-mon[88636]: pgmap v306: 161 pgs: 161 active+clean; 585 KiB data, 266 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:29:02.067 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:29:01 smithi069 ceph-mon[88636]: from='client.44844 -' entity='client.iscsi.foo.smithi069.yuumtr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T14:29:02.094 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:29:01 smithi122 ceph-mon[67998]: pgmap v306: 161 pgs: 161 active+clean; 585 KiB data, 266 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:29:02.094 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:29:01 smithi122 ceph-mon[67998]: from='client.44844 -' entity='client.iscsi.foo.smithi069.yuumtr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T14:29:04.066 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:29:03 smithi069 ceph-mon[83680]: pgmap v307: 161 pgs: 161 active+clean; 585 KiB data, 266 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s 2024-04-14T14:29:04.067 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:29:03 smithi069 ceph-mon[88636]: pgmap v307: 161 pgs: 161 active+clean; 585 KiB data, 266 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s 2024-04-14T14:29:04.094 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:29:03 smithi122 ceph-mon[67998]: pgmap v307: 161 pgs: 161 active+clean; 585 KiB data, 266 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s 2024-04-14T14:29:05.896 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:29:05 smithi122 ceph-mon[67998]: pgmap v308: 161 pgs: 161 active+clean; 585 KiB data, 266 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:29:06.066 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:29:05 smithi069 ceph-mon[83680]: pgmap v308: 161 pgs: 161 active+clean; 585 KiB data, 266 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:29:06.066 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:29:05 smithi069 ceph-mon[88636]: pgmap v308: 161 pgs: 161 active+clean; 585 KiB data, 266 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:29:06.816 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:29:06 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T14:29:06.817 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:29:06 smithi069 ceph-mon[88636]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T14:29:06.817 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 14 14:29:06 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-y[63038]: ::ffff:172.21.15.122 - - [14/Apr/2024:14:29:06] "GET /metrics HTTP/1.1" 200 38490 "" "Prometheus/2.43.0" 2024-04-14T14:29:07.094 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:29:06 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T14:29:08.066 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:29:07 smithi069 ceph-mon[83680]: pgmap v309: 161 pgs: 161 active+clean; 585 KiB data, 266 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-14T14:29:08.066 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:29:07 smithi069 ceph-mon[88636]: pgmap v309: 161 pgs: 161 active+clean; 585 KiB data, 266 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-14T14:29:08.094 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:29:07 smithi122 ceph-mon[67998]: pgmap v309: 161 pgs: 161 active+clean; 585 KiB data, 266 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-14T14:29:09.015 INFO:teuthology.orchestra.run.smithi069.stdout:true 2024-04-14T14:29:09.347 INFO:teuthology.orchestra.run.smithi069.stdout:"Error: UPGRADE_REDEPLOY_DAEMON: Upgrading daemon iscsi.foo.smithi069.yuumtr on host smithi069 failed." 2024-04-14T14:29:10.066 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:29:09 smithi069 ceph-mon[83680]: pgmap v310: 161 pgs: 161 active+clean; 585 KiB data, 266 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-04-14T14:29:10.066 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:29:09 smithi069 ceph-mon[88636]: pgmap v310: 161 pgs: 161 active+clean; 585 KiB data, 266 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-04-14T14:29:10.094 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:29:09 smithi122 ceph-mon[67998]: pgmap v310: 161 pgs: 161 active+clean; 585 KiB data, 266 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-04-14T14:29:10.173 DEBUG:teuthology.orchestra.run.smithi069:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid dc326fc2-fa65-11ee-bc8e-c7b262605968 -e sha1=a971fd5f5719038c70b817bafacd6d5400b1d888 -- bash -c 'ceph orch ps' 2024-04-14T14:29:10.793 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:29:10 smithi069 ceph-mon[83680]: from='client.54838 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T14:29:10.793 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:29:10 smithi069 ceph-mon[83680]: from='client.54844 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T14:29:10.794 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:29:10 smithi069 ceph-mon[88636]: from='client.54838 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T14:29:10.794 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:29:10 smithi069 ceph-mon[88636]: from='client.54844 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T14:29:11.094 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:29:10 smithi122 ceph-mon[67998]: from='client.54838 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T14:29:11.094 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:29:10 smithi122 ceph-mon[67998]: from='client.54844 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T14:29:11.816 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:29:11 smithi069 ceph-mon[83680]: pgmap v311: 161 pgs: 161 active+clean; 585 KiB data, 266 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:29:11.816 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:29:11 smithi069 ceph-mon[83680]: from='client.44844 -' entity='client.iscsi.foo.smithi069.yuumtr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T14:29:11.816 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:29:11 smithi069 ceph-mon[88636]: pgmap v311: 161 pgs: 161 active+clean; 585 KiB data, 266 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:29:11.816 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:29:11 smithi069 ceph-mon[88636]: from='client.44844 -' entity='client.iscsi.foo.smithi069.yuumtr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T14:29:12.094 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:29:11 smithi122 ceph-mon[67998]: pgmap v311: 161 pgs: 161 active+clean; 585 KiB data, 266 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:29:12.094 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:29:11 smithi122 ceph-mon[67998]: from='client.44844 -' entity='client.iscsi.foo.smithi069.yuumtr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T14:29:12.298 INFO:teuthology.orchestra.run.smithi069.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-14T14:29:12.298 INFO:teuthology.orchestra.run.smithi069.stdout:alertmanager.a smithi069 *:9093,9094 running (10m) 74s ago 32m 25.9M - 0.25.0 c8568f914cd2 7524f106e7bd 2024-04-14T14:29:12.298 INFO:teuthology.orchestra.run.smithi069.stdout:grafana.a smithi122 *:3000 running (10m) 74s ago 32m 82.4M - 9.4.12 2bacad6d85d8 5685178e3eba 2024-04-14T14:29:12.298 INFO:teuthology.orchestra.run.smithi069.stdout:iscsi.foo.smithi069.yuumtr smithi069 running (12m) 74s ago 31m 62.9M - 3.5 e1d6a67b021e 38cc81ea8cfa 2024-04-14T14:29:12.298 INFO:teuthology.orchestra.run.smithi069.stdout:mgr.x smithi122 *:8443,9283,8765 running (11m) 74s ago 37m 455M - 19.0.0-2778-ga971fd5f b96c509eef84 e29977590b43 2024-04-14T14:29:12.298 INFO:teuthology.orchestra.run.smithi069.stdout:mgr.y smithi069 *:8443,9283,8765 running (21m) 74s ago 38m 513M - 19.0.0-2778-ga971fd5f b96c509eef84 7af3d29d0e5f 2024-04-14T14:29:12.298 INFO:teuthology.orchestra.run.smithi069.stdout:mon.a smithi069 running (8m) 74s ago 38m 63.2M 2048M 19.0.0-2778-ga971fd5f b96c509eef84 8bcd6b3d3846 2024-04-14T14:29:12.298 INFO:teuthology.orchestra.run.smithi069.stdout:mon.b smithi122 running (9m) 74s ago 37m 49.5M 2048M 19.0.0-2778-ga971fd5f b96c509eef84 05f437a1a9a1 2024-04-14T14:29:12.299 INFO:teuthology.orchestra.run.smithi069.stdout:mon.c smithi069 running (8m) 74s ago 37m 44.8M 2048M 19.0.0-2778-ga971fd5f b96c509eef84 84aa95f11d0c 2024-04-14T14:29:12.299 INFO:teuthology.orchestra.run.smithi069.stdout:node-exporter.a smithi069 *:9100 running (11m) 74s ago 33m 15.7M - 1.5.0 0da6a335fe13 0de0eb2171b9 2024-04-14T14:29:12.299 INFO:teuthology.orchestra.run.smithi069.stdout:node-exporter.b smithi122 *:9100 running (11m) 74s ago 33m 16.3M - 1.5.0 0da6a335fe13 67877bf02d8e 2024-04-14T14:29:12.299 INFO:teuthology.orchestra.run.smithi069.stdout:osd.0 smithi069 running (7m) 74s ago 36m 62.2M 2992M 19.0.0-2778-ga971fd5f b96c509eef84 a0e57e3fb494 2024-04-14T14:29:12.299 INFO:teuthology.orchestra.run.smithi069.stdout:osd.1 smithi069 running (6m) 74s ago 36m 62.6M 2992M 19.0.0-2778-ga971fd5f b96c509eef84 cf73d17919bb 2024-04-14T14:29:12.299 INFO:teuthology.orchestra.run.smithi069.stdout:osd.2 smithi069 running (5m) 74s ago 36m 49.7M 2992M 19.0.0-2778-ga971fd5f b96c509eef84 c0efba5ef451 2024-04-14T14:29:12.299 INFO:teuthology.orchestra.run.smithi069.stdout:osd.3 smithi069 running (4m) 74s ago 35m 60.6M 2992M 19.0.0-2778-ga971fd5f b96c509eef84 d13460da0a65 2024-04-14T14:29:12.299 INFO:teuthology.orchestra.run.smithi069.stdout:osd.4 smithi122 running (4m) 74s ago 35m 58.6M 3248M 19.0.0-2778-ga971fd5f b96c509eef84 7a87a024f984 2024-04-14T14:29:12.299 INFO:teuthology.orchestra.run.smithi069.stdout:osd.5 smithi122 running (3m) 74s ago 34m 56.7M 3248M 19.0.0-2778-ga971fd5f b96c509eef84 02b623d44db3 2024-04-14T14:29:12.299 INFO:teuthology.orchestra.run.smithi069.stdout:osd.6 smithi122 running (3m) 74s ago 34m 53.9M 3248M 19.0.0-2778-ga971fd5f b96c509eef84 72f85ece3e75 2024-04-14T14:29:12.299 INFO:teuthology.orchestra.run.smithi069.stdout:osd.7 smithi122 running (2m) 74s ago 34m 55.3M 3248M 19.0.0-2778-ga971fd5f b96c509eef84 0c53c3cd2bc0 2024-04-14T14:29:12.299 INFO:teuthology.orchestra.run.smithi069.stdout:prometheus.a smithi122 *:9095 running (11m) 74s ago 33m 71.1M - 2.43.0 a07b618ecd1d 5c1f41d3bbe6 2024-04-14T14:29:12.300 INFO:teuthology.orchestra.run.smithi069.stdout:rgw.foo.smithi069.wesedz smithi069 *:8000 running (84s) 74s ago 31m 77.2M - 19.0.0-2778-ga971fd5f b96c509eef84 f135fd86285d 2024-04-14T14:29:12.300 INFO:teuthology.orchestra.run.smithi069.stdout:rgw.foo.smithi122.jiobyv smithi122 *:8000 running (78s) 74s ago 31m 76.3M - 19.0.0-2778-ga971fd5f b96c509eef84 7d37b0b84199 2024-04-14T14:29:12.929 DEBUG:teuthology.orchestra.run.smithi069:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid dc326fc2-fa65-11ee-bc8e-c7b262605968 -e sha1=a971fd5f5719038c70b817bafacd6d5400b1d888 -- bash -c 'ceph versions' 2024-04-14T14:29:13.817 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:29:13 smithi069 ceph-mon[83680]: pgmap v312: 161 pgs: 161 active+clean; 585 KiB data, 266 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:29:13.818 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:29:13 smithi069 ceph-mon[83680]: from='client.44907 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T14:29:13.819 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:29:13 smithi069 ceph-mon[88636]: pgmap v312: 161 pgs: 161 active+clean; 585 KiB data, 266 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:29:13.819 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:29:13 smithi069 ceph-mon[88636]: from='client.44907 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T14:29:14.094 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:29:13 smithi122 ceph-mon[67998]: pgmap v312: 161 pgs: 161 active+clean; 585 KiB data, 266 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:29:14.094 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:29:13 smithi122 ceph-mon[67998]: from='client.44907 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T14:29:15.051 INFO:teuthology.orchestra.run.smithi069.stdout:{ 2024-04-14T14:29:15.051 INFO:teuthology.orchestra.run.smithi069.stdout: "mon": { 2024-04-14T14:29:15.051 INFO:teuthology.orchestra.run.smithi069.stdout: "ceph version 19.0.0-2778-ga971fd5f (a971fd5f5719038c70b817bafacd6d5400b1d888) squid (dev)": 3 2024-04-14T14:29:15.051 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-14T14:29:15.051 INFO:teuthology.orchestra.run.smithi069.stdout: "mgr": { 2024-04-14T14:29:15.051 INFO:teuthology.orchestra.run.smithi069.stdout: "ceph version 19.0.0-2778-ga971fd5f (a971fd5f5719038c70b817bafacd6d5400b1d888) squid (dev)": 2 2024-04-14T14:29:15.051 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-14T14:29:15.051 INFO:teuthology.orchestra.run.smithi069.stdout: "osd": { 2024-04-14T14:29:15.051 INFO:teuthology.orchestra.run.smithi069.stdout: "ceph version 19.0.0-2778-ga971fd5f (a971fd5f5719038c70b817bafacd6d5400b1d888) squid (dev)": 8 2024-04-14T14:29:15.051 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-14T14:29:15.051 INFO:teuthology.orchestra.run.smithi069.stdout: "rgw": { 2024-04-14T14:29:15.051 INFO:teuthology.orchestra.run.smithi069.stdout: "ceph version 19.0.0-2778-ga971fd5f (a971fd5f5719038c70b817bafacd6d5400b1d888) squid (dev)": 2 2024-04-14T14:29:15.052 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-14T14:29:15.052 INFO:teuthology.orchestra.run.smithi069.stdout: "overall": { 2024-04-14T14:29:15.052 INFO:teuthology.orchestra.run.smithi069.stdout: "ceph version 19.0.0-2778-ga971fd5f (a971fd5f5719038c70b817bafacd6d5400b1d888) squid (dev)": 15 2024-04-14T14:29:15.052 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-14T14:29:15.052 INFO:teuthology.orchestra.run.smithi069.stdout:} 2024-04-14T14:29:15.626 DEBUG:teuthology.orchestra.run.smithi069:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid dc326fc2-fa65-11ee-bc8e-c7b262605968 -e sha1=a971fd5f5719038c70b817bafacd6d5400b1d888 -- bash -c 'echo "wait for servicemap items w/ changing names to refresh"' 2024-04-14T14:29:15.933 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:29:15 smithi069 ceph-mon[83680]: pgmap v313: 161 pgs: 161 active+clean; 585 KiB data, 266 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:29:15.933 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:29:15 smithi069 ceph-mon[83680]: from='client.? 172.21.15.69:0/1977044987' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T14:29:15.933 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:29:15 smithi069 ceph-mon[88636]: pgmap v313: 161 pgs: 161 active+clean; 585 KiB data, 266 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:29:15.933 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:29:15 smithi069 ceph-mon[88636]: from='client.? 172.21.15.69:0/1977044987' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T14:29:16.094 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:29:15 smithi122 ceph-mon[67998]: pgmap v313: 161 pgs: 161 active+clean; 585 KiB data, 266 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:29:16.094 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:29:15 smithi122 ceph-mon[67998]: from='client.? 172.21.15.69:0/1977044987' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T14:29:16.816 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 14 14:29:16 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-y[63038]: ::ffff:172.21.15.122 - - [14/Apr/2024:14:29:16] "GET /metrics HTTP/1.1" 200 38487 "" "Prometheus/2.43.0" 2024-04-14T14:29:17.445 INFO:teuthology.orchestra.run.smithi069.stdout:wait for servicemap items w/ changing names to refresh 2024-04-14T14:29:18.066 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:29:17 smithi069 ceph-mon[88636]: pgmap v314: 161 pgs: 161 active+clean; 585 KiB data, 266 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:29:18.066 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:29:17 smithi069 ceph-mon[83680]: pgmap v314: 161 pgs: 161 active+clean; 585 KiB data, 266 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:29:18.094 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:29:17 smithi122 ceph-mon[67998]: pgmap v314: 161 pgs: 161 active+clean; 585 KiB data, 266 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:29:18.101 DEBUG:teuthology.orchestra.run.smithi069:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid dc326fc2-fa65-11ee-bc8e-c7b262605968 -e sha1=a971fd5f5719038c70b817bafacd6d5400b1d888 -- bash -c 'sleep 60' 2024-04-14T14:29:19.662 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:29:19 smithi069 ceph-mon[83680]: pgmap v315: 161 pgs: 161 active+clean; 585 KiB data, 266 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:29:20.027 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:29:19 smithi069 ceph-mon[88636]: pgmap v315: 161 pgs: 161 active+clean; 585 KiB data, 266 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:29:20.094 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:29:19 smithi122 ceph-mon[67998]: pgmap v315: 161 pgs: 161 active+clean; 585 KiB data, 266 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:29:22.066 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:29:21 smithi069 ceph-mon[83680]: pgmap v316: 161 pgs: 161 active+clean; 585 KiB data, 266 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:29:22.066 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:29:21 smithi069 ceph-mon[83680]: from='client.44844 -' entity='client.iscsi.foo.smithi069.yuumtr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T14:29:22.066 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:29:21 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T14:29:22.067 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:29:21 smithi069 ceph-mon[88636]: pgmap v316: 161 pgs: 161 active+clean; 585 KiB data, 266 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:29:22.067 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:29:21 smithi069 ceph-mon[88636]: from='client.44844 -' entity='client.iscsi.foo.smithi069.yuumtr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T14:29:22.067 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:29:21 smithi069 ceph-mon[88636]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T14:29:22.094 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:29:21 smithi122 ceph-mon[67998]: pgmap v316: 161 pgs: 161 active+clean; 585 KiB data, 266 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:29:22.094 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:29:21 smithi122 ceph-mon[67998]: from='client.44844 -' entity='client.iscsi.foo.smithi069.yuumtr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T14:29:22.094 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:29:21 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T14:29:24.066 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:29:23 smithi069 ceph-mon[83680]: pgmap v317: 161 pgs: 161 active+clean; 585 KiB data, 266 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:29:24.066 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:29:23 smithi069 ceph-mon[88636]: pgmap v317: 161 pgs: 161 active+clean; 585 KiB data, 266 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:29:24.094 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:29:23 smithi122 ceph-mon[67998]: pgmap v317: 161 pgs: 161 active+clean; 585 KiB data, 266 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:29:26.066 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:29:25 smithi069 ceph-mon[83680]: pgmap v318: 161 pgs: 161 active+clean; 585 KiB data, 266 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:29:26.066 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:29:25 smithi069 ceph-mon[88636]: pgmap v318: 161 pgs: 161 active+clean; 585 KiB data, 266 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:29:26.094 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:29:25 smithi122 ceph-mon[67998]: pgmap v318: 161 pgs: 161 active+clean; 585 KiB data, 266 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:29:26.816 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 14 14:29:26 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-y[63038]: ::ffff:172.21.15.122 - - [14/Apr/2024:14:29:26] "GET /metrics HTTP/1.1" 200 38483 "" "Prometheus/2.43.0" 2024-04-14T14:29:28.066 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:29:27 smithi069 ceph-mon[83680]: pgmap v319: 161 pgs: 161 active+clean; 585 KiB data, 266 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:29:28.066 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:29:27 smithi069 ceph-mon[88636]: pgmap v319: 161 pgs: 161 active+clean; 585 KiB data, 266 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:29:28.094 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:29:27 smithi122 ceph-mon[67998]: pgmap v319: 161 pgs: 161 active+clean; 585 KiB data, 266 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:29:30.068 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:29:29 smithi069 ceph-mon[83680]: pgmap v320: 161 pgs: 161 active+clean; 585 KiB data, 266 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:29:30.069 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:29:29 smithi069 ceph-mon[88636]: pgmap v320: 161 pgs: 161 active+clean; 585 KiB data, 266 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:29:30.094 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:29:29 smithi122 ceph-mon[67998]: pgmap v320: 161 pgs: 161 active+clean; 585 KiB data, 266 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:29:32.066 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:29:31 smithi069 ceph-mon[83680]: pgmap v321: 161 pgs: 161 active+clean; 585 KiB data, 266 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:29:32.066 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:29:31 smithi069 ceph-mon[83680]: from='client.44844 -' entity='client.iscsi.foo.smithi069.yuumtr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T14:29:32.066 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:29:31 smithi069 ceph-mon[88636]: pgmap v321: 161 pgs: 161 active+clean; 585 KiB data, 266 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:29:32.067 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:29:31 smithi069 ceph-mon[88636]: from='client.44844 -' entity='client.iscsi.foo.smithi069.yuumtr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T14:29:32.094 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:29:31 smithi122 ceph-mon[67998]: pgmap v321: 161 pgs: 161 active+clean; 585 KiB data, 266 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:29:32.094 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:29:31 smithi122 ceph-mon[67998]: from='client.44844 -' entity='client.iscsi.foo.smithi069.yuumtr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T14:29:34.066 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:29:33 smithi069 ceph-mon[83680]: pgmap v322: 161 pgs: 161 active+clean; 585 KiB data, 266 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:29:34.066 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:29:33 smithi069 ceph-mon[88636]: pgmap v322: 161 pgs: 161 active+clean; 585 KiB data, 266 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:29:34.094 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:29:33 smithi122 ceph-mon[67998]: pgmap v322: 161 pgs: 161 active+clean; 585 KiB data, 266 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:29:36.066 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:29:35 smithi069 ceph-mon[83680]: pgmap v323: 161 pgs: 161 active+clean; 585 KiB data, 266 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:29:36.066 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:29:35 smithi069 ceph-mon[88636]: pgmap v323: 161 pgs: 161 active+clean; 585 KiB data, 266 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:29:36.094 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:29:35 smithi122 ceph-mon[67998]: pgmap v323: 161 pgs: 161 active+clean; 585 KiB data, 266 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:29:36.816 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:29:36 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T14:29:36.816 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 14 14:29:36 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-y[63038]: ::ffff:172.21.15.122 - - [14/Apr/2024:14:29:36] "GET /metrics HTTP/1.1" 200 38483 "" "Prometheus/2.43.0" 2024-04-14T14:29:36.816 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:29:36 smithi069 ceph-mon[88636]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T14:29:37.094 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:29:36 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T14:29:38.066 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:29:37 smithi069 ceph-mon[83680]: pgmap v324: 161 pgs: 161 active+clean; 585 KiB data, 266 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:29:38.067 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:29:37 smithi069 ceph-mon[88636]: pgmap v324: 161 pgs: 161 active+clean; 585 KiB data, 266 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:29:38.094 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:29:37 smithi122 ceph-mon[67998]: pgmap v324: 161 pgs: 161 active+clean; 585 KiB data, 266 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:29:40.066 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:29:39 smithi069 ceph-mon[83680]: pgmap v325: 161 pgs: 161 active+clean; 585 KiB data, 266 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:29:40.066 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:29:39 smithi069 ceph-mon[88636]: pgmap v325: 161 pgs: 161 active+clean; 585 KiB data, 266 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:29:40.093 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:29:39 smithi122 ceph-mon[67998]: pgmap v325: 161 pgs: 161 active+clean; 585 KiB data, 266 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:29:41.066 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:29:40 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T14:29:41.066 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:29:40 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-14T14:29:41.066 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:29:40 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:29:41.067 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:29:40 smithi069 ceph-mon[88636]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T14:29:41.067 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:29:40 smithi069 ceph-mon[88636]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-14T14:29:41.067 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:29:40 smithi069 ceph-mon[88636]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:29:41.094 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:29:40 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T14:29:41.094 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:29:40 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-14T14:29:41.094 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:29:40 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:29:42.066 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:29:41 smithi069 ceph-mon[83680]: pgmap v326: 161 pgs: 161 active+clean; 585 KiB data, 266 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:29:42.066 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:29:41 smithi069 ceph-mon[83680]: from='client.44844 -' entity='client.iscsi.foo.smithi069.yuumtr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T14:29:42.066 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:29:41 smithi069 ceph-mon[88636]: pgmap v326: 161 pgs: 161 active+clean; 585 KiB data, 266 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:29:42.067 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:29:41 smithi069 ceph-mon[88636]: from='client.44844 -' entity='client.iscsi.foo.smithi069.yuumtr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T14:29:42.094 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:29:41 smithi122 ceph-mon[67998]: pgmap v326: 161 pgs: 161 active+clean; 585 KiB data, 266 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:29:42.094 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:29:41 smithi122 ceph-mon[67998]: from='client.44844 -' entity='client.iscsi.foo.smithi069.yuumtr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T14:29:44.066 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:29:43 smithi069 ceph-mon[83680]: pgmap v327: 161 pgs: 161 active+clean; 585 KiB data, 266 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:29:44.067 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:29:43 smithi069 ceph-mon[88636]: pgmap v327: 161 pgs: 161 active+clean; 585 KiB data, 266 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:29:44.094 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:29:43 smithi122 ceph-mon[67998]: pgmap v327: 161 pgs: 161 active+clean; 585 KiB data, 266 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:29:46.066 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:29:45 smithi069 ceph-mon[83680]: pgmap v328: 161 pgs: 161 active+clean; 585 KiB data, 266 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:29:46.066 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:29:45 smithi069 ceph-mon[88636]: pgmap v328: 161 pgs: 161 active+clean; 585 KiB data, 266 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:29:46.094 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:29:45 smithi122 ceph-mon[67998]: pgmap v328: 161 pgs: 161 active+clean; 585 KiB data, 266 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:29:46.816 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 14 14:29:46 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-y[63038]: ::ffff:172.21.15.122 - - [14/Apr/2024:14:29:46] "GET /metrics HTTP/1.1" 200 38490 "" "Prometheus/2.43.0" 2024-04-14T14:29:48.066 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:29:47 smithi069 ceph-mon[83680]: pgmap v329: 161 pgs: 161 active+clean; 585 KiB data, 266 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:29:48.067 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:29:47 smithi069 ceph-mon[88636]: pgmap v329: 161 pgs: 161 active+clean; 585 KiB data, 266 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:29:48.094 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:29:47 smithi122 ceph-mon[67998]: pgmap v329: 161 pgs: 161 active+clean; 585 KiB data, 266 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:29:49.066 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:29:48 smithi069 ceph-mon[83680]: pgmap v330: 161 pgs: 161 active+clean; 585 KiB data, 266 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:29:49.066 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:29:48 smithi069 ceph-mon[88636]: pgmap v330: 161 pgs: 161 active+clean; 585 KiB data, 266 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:29:49.094 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:29:48 smithi122 ceph-mon[67998]: pgmap v330: 161 pgs: 161 active+clean; 585 KiB data, 266 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:29:50.868 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:29:50 smithi069 ceph-mon[83680]: pgmap v331: 161 pgs: 161 active+clean; 585 KiB data, 266 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:29:50.869 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:29:50 smithi069 ceph-mon[83680]: from='client.44844 -' entity='client.iscsi.foo.smithi069.yuumtr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T14:29:50.869 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:29:50 smithi069 ceph-mon[88636]: pgmap v331: 161 pgs: 161 active+clean; 585 KiB data, 266 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:29:50.869 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:29:50 smithi069 ceph-mon[88636]: from='client.44844 -' entity='client.iscsi.foo.smithi069.yuumtr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T14:29:51.094 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:29:50 smithi122 ceph-mon[67998]: pgmap v331: 161 pgs: 161 active+clean; 585 KiB data, 266 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:29:51.094 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:29:50 smithi122 ceph-mon[67998]: from='client.44844 -' entity='client.iscsi.foo.smithi069.yuumtr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T14:29:52.066 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:29:51 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T14:29:52.066 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:29:51 smithi069 ceph-mon[88636]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T14:29:52.094 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:29:51 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T14:29:53.066 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:29:52 smithi069 ceph-mon[83680]: pgmap v332: 161 pgs: 161 active+clean; 585 KiB data, 266 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:29:53.067 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:29:52 smithi069 ceph-mon[88636]: pgmap v332: 161 pgs: 161 active+clean; 585 KiB data, 266 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:29:53.094 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:29:52 smithi122 ceph-mon[67998]: pgmap v332: 161 pgs: 161 active+clean; 585 KiB data, 266 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:29:55.066 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:29:54 smithi069 ceph-mon[83680]: pgmap v333: 161 pgs: 161 active+clean; 585 KiB data, 266 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-04-14T14:29:55.066 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:29:54 smithi069 ceph-mon[88636]: pgmap v333: 161 pgs: 161 active+clean; 585 KiB data, 266 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-04-14T14:29:55.094 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:29:54 smithi122 ceph-mon[67998]: pgmap v333: 161 pgs: 161 active+clean; 585 KiB data, 266 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-04-14T14:29:56.789 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 14 14:29:56 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-y[63038]: ::ffff:172.21.15.122 - - [14/Apr/2024:14:29:56] "GET /metrics HTTP/1.1" 200 38485 "" "Prometheus/2.43.0" 2024-04-14T14:29:57.066 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:29:56 smithi069 ceph-mon[88636]: pgmap v334: 161 pgs: 161 active+clean; 585 KiB data, 266 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:29:57.066 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:29:56 smithi069 ceph-mon[83680]: pgmap v334: 161 pgs: 161 active+clean; 585 KiB data, 266 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:29:57.094 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:29:56 smithi122 ceph-mon[67998]: pgmap v334: 161 pgs: 161 active+clean; 585 KiB data, 266 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:29:59.066 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:29:58 smithi069 ceph-mon[83680]: pgmap v335: 161 pgs: 161 active+clean; 585 KiB data, 266 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:29:59.066 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:29:58 smithi069 ceph-mon[88636]: pgmap v335: 161 pgs: 161 active+clean; 585 KiB data, 266 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:29:59.094 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:29:58 smithi122 ceph-mon[67998]: pgmap v335: 161 pgs: 161 active+clean; 585 KiB data, 266 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:30:01.066 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:30:00 smithi069 ceph-mon[83680]: pgmap v336: 161 pgs: 161 active+clean; 585 KiB data, 266 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:30:01.067 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:30:00 smithi069 ceph-mon[83680]: Health detail: HEALTH_WARN Upgrading daemon iscsi.foo.smithi069.yuumtr on host smithi069 failed. 2024-04-14T14:30:01.067 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:30:00 smithi069 ceph-mon[83680]: [WRN] UPGRADE_REDEPLOY_DAEMON: Upgrading daemon iscsi.foo.smithi069.yuumtr on host smithi069 failed. 2024-04-14T14:30:01.067 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:30:00 smithi069 ceph-mon[83680]: Upgrade daemon: iscsi.foo.smithi069.yuumtr: cephadm exited with an error code: 1, stderr: Redeploy daemon iscsi.foo.smithi069.yuumtr ... 2024-04-14T14:30:01.067 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:30:00 smithi069 ceph-mon[83680]: Creating ceph-iscsi config... 2024-04-14T14:30:01.067 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:30:00 smithi069 ceph-mon[83680]: Write file: /var/lib/ceph/dc326fc2-fa65-11ee-bc8e-c7b262605968/iscsi.foo.smithi069.yuumtr/iscsi-gateway.cfg 2024-04-14T14:30:01.067 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:30:00 smithi069 ceph-mon[83680]: Write file: /var/lib/ceph/dc326fc2-fa65-11ee-bc8e-c7b262605968/iscsi.foo.smithi069.yuumtr/tcmu-runner-entrypoint.sh 2024-04-14T14:30:01.067 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:30:00 smithi069 ceph-mon[83680]: Failed to trim old cgroups /sys/fs/cgroup/system.slice/system-ceph\x2ddc326fc2\x2dfa65\x2d11ee\x2dbc8e\x2dc7b262605968.slice/ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968@iscsi.foo.smithi069.yuumtr.service 2024-04-14T14:30:01.067 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:30:00 smithi069 ceph-mon[83680]: Non-zero exit code 1 from systemctl start ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968@iscsi.foo.smithi069.yuumtr 2024-04-14T14:30:01.067 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:30:00 smithi069 ceph-mon[83680]: systemctl: stderr Job for ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968@iscsi.foo.smithi069.yuumtr.service failed because the control process exited with error code. 2024-04-14T14:30:01.067 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:30:00 smithi069 ceph-mon[83680]: systemctl: stderr See "systemctl status ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968@iscsi.foo.smithi069.yuumtr.service" and "journalctl -xeu ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968@iscsi.foo.smithi069.yuumtr.service" for details. 2024-04-14T14:30:01.067 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:30:00 smithi069 ceph-mon[83680]: Traceback (most recent call last): 2024-04-14T14:30:01.067 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:30:00 smithi069 ceph-mon[83680]: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-04-14T14:30:01.067 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:30:00 smithi069 ceph-mon[83680]: return _run_code(code, main_globals, None, 2024-04-14T14:30:01.067 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:30:00 smithi069 ceph-mon[83680]: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-04-14T14:30:01.068 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:30:00 smithi069 ceph-mon[83680]: exec(code, run_globals) 2024-04-14T14:30:01.068 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:30:00 smithi069 ceph-mon[83680]: File "/var/lib/ceph/dc326fc2-fa65-11ee-bc8e-c7b262605968/cephadm.de2a8faf91349910526b51da93d8bd42d57efdd2e93c32cd09ae64ecc1748ae9/__main__.py", line 5608, in 2024-04-14T14:30:01.068 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:30:00 smithi069 ceph-mon[83680]: File "/var/lib/ceph/dc326fc2-fa65-11ee-bc8e-c7b262605968/cephadm.de2a8faf91349910526b51da93d8bd42d57efdd2e93c32cd09ae64ecc1748ae9/__main__.py", line 5596, in main 2024-04-14T14:30:01.068 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:30:00 smithi069 ceph-mon[83680]: File "/var/lib/ceph/dc326fc2-fa65-11ee-bc8e-c7b262605968/cephadm.de2a8faf91349910526b51da93d8bd42d57efdd2e93c32cd09ae64ecc1748ae9/__main__.py", line 3096, in command_deploy_from 2024-04-14T14:30:01.068 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:30:00 smithi069 ceph-mon[83680]: File "/var/lib/ceph/dc326fc2-fa65-11ee-bc8e-c7b262605968/cephadm.de2a8faf91349910526b51da93d8bd42d57efdd2e93c32cd09ae64ecc1748ae9/__main__.py", line 3131, in _common_deploy 2024-04-14T14:30:01.068 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:30:00 smithi069 ceph-mon[83680]: File "/var/lib/ceph/dc326fc2-fa65-11ee-bc8e-c7b262605968/cephadm.de2a8faf91349910526b51da93d8bd42d57efdd2e93c32cd09ae64ecc1748ae9/__main__.py", line 3151, in _deploy_daemon_container 2024-04-14T14:30:01.068 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:30:00 smithi069 ceph-mon[83680]: File "/var/lib/ceph/dc326fc2-fa65-11ee-bc8e-c7b262605968/cephadm.de2a8faf91349910526b51da93d8bd42d57efdd2e93c32cd09ae64ecc1748ae9/__main__.py", line 1087, in deploy_daemon 2024-04-14T14:30:01.068 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:30:00 smithi069 ceph-mon[83680]: File "/var/lib/ceph/dc326fc2-fa65-11ee-bc8e-c7b262605968/cephadm.de2a8faf91349910526b51da93d8bd42d57efdd2e93c32cd09ae64ecc1748ae9/__main__.py", line 1223, in deploy_daemon_units 2024-04-14T14:30:01.068 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:30:00 smithi069 ceph-mon[88636]: pgmap v336: 161 pgs: 161 active+clean; 585 KiB data, 266 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:30:01.068 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:30:00 smithi069 ceph-mon[88636]: Health detail: HEALTH_WARN Upgrading daemon iscsi.foo.smithi069.yuumtr on host smithi069 failed. 2024-04-14T14:30:01.069 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:30:00 smithi069 ceph-mon[88636]: [WRN] UPGRADE_REDEPLOY_DAEMON: Upgrading daemon iscsi.foo.smithi069.yuumtr on host smithi069 failed. 2024-04-14T14:30:01.069 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:30:00 smithi069 ceph-mon[88636]: Upgrade daemon: iscsi.foo.smithi069.yuumtr: cephadm exited with an error code: 1, stderr: Redeploy daemon iscsi.foo.smithi069.yuumtr ... 2024-04-14T14:30:01.069 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:30:00 smithi069 ceph-mon[88636]: Creating ceph-iscsi config... 2024-04-14T14:30:01.069 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:30:00 smithi069 ceph-mon[88636]: Write file: /var/lib/ceph/dc326fc2-fa65-11ee-bc8e-c7b262605968/iscsi.foo.smithi069.yuumtr/iscsi-gateway.cfg 2024-04-14T14:30:01.069 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:30:00 smithi069 ceph-mon[88636]: Write file: /var/lib/ceph/dc326fc2-fa65-11ee-bc8e-c7b262605968/iscsi.foo.smithi069.yuumtr/tcmu-runner-entrypoint.sh 2024-04-14T14:30:01.069 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:30:00 smithi069 ceph-mon[88636]: Failed to trim old cgroups /sys/fs/cgroup/system.slice/system-ceph\x2ddc326fc2\x2dfa65\x2d11ee\x2dbc8e\x2dc7b262605968.slice/ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968@iscsi.foo.smithi069.yuumtr.service 2024-04-14T14:30:01.069 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:30:00 smithi069 ceph-mon[88636]: Non-zero exit code 1 from systemctl start ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968@iscsi.foo.smithi069.yuumtr 2024-04-14T14:30:01.069 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:30:00 smithi069 ceph-mon[88636]: systemctl: stderr Job for ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968@iscsi.foo.smithi069.yuumtr.service failed because the control process exited with error code. 2024-04-14T14:30:01.069 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:30:00 smithi069 ceph-mon[88636]: systemctl: stderr See "systemctl status ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968@iscsi.foo.smithi069.yuumtr.service" and "journalctl -xeu ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968@iscsi.foo.smithi069.yuumtr.service" for details. 2024-04-14T14:30:01.069 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:30:00 smithi069 ceph-mon[88636]: Traceback (most recent call last): 2024-04-14T14:30:01.069 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:30:00 smithi069 ceph-mon[88636]: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-04-14T14:30:01.069 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:30:00 smithi069 ceph-mon[88636]: return _run_code(code, main_globals, None, 2024-04-14T14:30:01.069 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:30:00 smithi069 ceph-mon[88636]: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-04-14T14:30:01.069 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:30:00 smithi069 ceph-mon[88636]: exec(code, run_globals) 2024-04-14T14:30:01.069 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:30:00 smithi069 ceph-mon[88636]: File "/var/lib/ceph/dc326fc2-fa65-11ee-bc8e-c7b262605968/cephadm.de2a8faf91349910526b51da93d8bd42d57efdd2e93c32cd09ae64ecc1748ae9/__main__.py", line 5608, in 2024-04-14T14:30:01.070 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:30:00 smithi069 ceph-mon[88636]: File "/var/lib/ceph/dc326fc2-fa65-11ee-bc8e-c7b262605968/cephadm.de2a8faf91349910526b51da93d8bd42d57efdd2e93c32cd09ae64ecc1748ae9/__main__.py", line 5596, in main 2024-04-14T14:30:01.070 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:30:00 smithi069 ceph-mon[88636]: File "/var/lib/ceph/dc326fc2-fa65-11ee-bc8e-c7b262605968/cephadm.de2a8faf91349910526b51da93d8bd42d57efdd2e93c32cd09ae64ecc1748ae9/__main__.py", line 3096, in command_deploy_from 2024-04-14T14:30:01.070 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:30:00 smithi069 ceph-mon[88636]: File "/var/lib/ceph/dc326fc2-fa65-11ee-bc8e-c7b262605968/cephadm.de2a8faf91349910526b51da93d8bd42d57efdd2e93c32cd09ae64ecc1748ae9/__main__.py", line 3131, in _common_deploy 2024-04-14T14:30:01.070 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:30:00 smithi069 ceph-mon[88636]: File "/var/lib/ceph/dc326fc2-fa65-11ee-bc8e-c7b262605968/cephadm.de2a8faf91349910526b51da93d8bd42d57efdd2e93c32cd09ae64ecc1748ae9/__main__.py", line 3151, in _deploy_daemon_container 2024-04-14T14:30:01.070 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:30:00 smithi069 ceph-mon[88636]: File "/var/lib/ceph/dc326fc2-fa65-11ee-bc8e-c7b262605968/cephadm.de2a8faf91349910526b51da93d8bd42d57efdd2e93c32cd09ae64ecc1748ae9/__main__.py", line 1087, in deploy_daemon 2024-04-14T14:30:01.070 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:30:00 smithi069 ceph-mon[88636]: File "/var/lib/ceph/dc326fc2-fa65-11ee-bc8e-c7b262605968/cephadm.de2a8faf91349910526b51da93d8bd42d57efdd2e93c32cd09ae64ecc1748ae9/__main__.py", line 1223, in deploy_daemon_units 2024-04-14T14:30:01.070 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:30:00 smithi069 ceph-mon[83680]: File "/var/lib/ceph/dc326fc2-fa65-11ee-bc8e-c7b262605968/cephadm.de2a8faf91349910526b51da93d8bd42d57efdd2e93c32cd09ae64ecc1748ae9/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-04-14T14:30:01.070 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:30:00 smithi069 ceph-mon[83680]: RuntimeError: Failed command: systemctl start ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968@iscsi.foo.smithi069.yuumtr: Job for ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968@iscsi.foo.smithi069.yuumtr.service failed because the control process exited with error code. 2024-04-14T14:30:01.071 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:30:00 smithi069 ceph-mon[83680]: See "systemctl status ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968@iscsi.foo.smithi069.yuumtr.service" and "journalctl -xeu ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968@iscsi.foo.smithi069.yuumtr.service" for details. 2024-04-14T14:30:01.071 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:30:00 smithi069 ceph-mon[83680]: from='client.44844 -' entity='client.iscsi.foo.smithi069.yuumtr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T14:30:01.071 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:30:00 smithi069 ceph-mon[88636]: File "/var/lib/ceph/dc326fc2-fa65-11ee-bc8e-c7b262605968/cephadm.de2a8faf91349910526b51da93d8bd42d57efdd2e93c32cd09ae64ecc1748ae9/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-04-14T14:30:01.071 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:30:00 smithi069 ceph-mon[88636]: RuntimeError: Failed command: systemctl start ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968@iscsi.foo.smithi069.yuumtr: Job for ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968@iscsi.foo.smithi069.yuumtr.service failed because the control process exited with error code. 2024-04-14T14:30:01.071 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:30:00 smithi069 ceph-mon[88636]: See "systemctl status ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968@iscsi.foo.smithi069.yuumtr.service" and "journalctl -xeu ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968@iscsi.foo.smithi069.yuumtr.service" for details. 2024-04-14T14:30:01.071 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:30:00 smithi069 ceph-mon[88636]: from='client.44844 -' entity='client.iscsi.foo.smithi069.yuumtr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T14:30:01.094 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:30:00 smithi122 ceph-mon[67998]: pgmap v336: 161 pgs: 161 active+clean; 585 KiB data, 266 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:30:01.094 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:30:00 smithi122 ceph-mon[67998]: Health detail: HEALTH_WARN Upgrading daemon iscsi.foo.smithi069.yuumtr on host smithi069 failed. 2024-04-14T14:30:01.094 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:30:00 smithi122 ceph-mon[67998]: [WRN] UPGRADE_REDEPLOY_DAEMON: Upgrading daemon iscsi.foo.smithi069.yuumtr on host smithi069 failed. 2024-04-14T14:30:01.094 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:30:00 smithi122 ceph-mon[67998]: Upgrade daemon: iscsi.foo.smithi069.yuumtr: cephadm exited with an error code: 1, stderr: Redeploy daemon iscsi.foo.smithi069.yuumtr ... 2024-04-14T14:30:01.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:30:00 smithi122 ceph-mon[67998]: Creating ceph-iscsi config... 2024-04-14T14:30:01.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:30:00 smithi122 ceph-mon[67998]: Write file: /var/lib/ceph/dc326fc2-fa65-11ee-bc8e-c7b262605968/iscsi.foo.smithi069.yuumtr/iscsi-gateway.cfg 2024-04-14T14:30:01.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:30:00 smithi122 ceph-mon[67998]: Write file: /var/lib/ceph/dc326fc2-fa65-11ee-bc8e-c7b262605968/iscsi.foo.smithi069.yuumtr/tcmu-runner-entrypoint.sh 2024-04-14T14:30:01.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:30:00 smithi122 ceph-mon[67998]: Failed to trim old cgroups /sys/fs/cgroup/system.slice/system-ceph\x2ddc326fc2\x2dfa65\x2d11ee\x2dbc8e\x2dc7b262605968.slice/ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968@iscsi.foo.smithi069.yuumtr.service 2024-04-14T14:30:01.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:30:00 smithi122 ceph-mon[67998]: Non-zero exit code 1 from systemctl start ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968@iscsi.foo.smithi069.yuumtr 2024-04-14T14:30:01.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:30:00 smithi122 ceph-mon[67998]: systemctl: stderr Job for ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968@iscsi.foo.smithi069.yuumtr.service failed because the control process exited with error code. 2024-04-14T14:30:01.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:30:00 smithi122 ceph-mon[67998]: systemctl: stderr See "systemctl status ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968@iscsi.foo.smithi069.yuumtr.service" and "journalctl -xeu ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968@iscsi.foo.smithi069.yuumtr.service" for details. 2024-04-14T14:30:01.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:30:00 smithi122 ceph-mon[67998]: Traceback (most recent call last): 2024-04-14T14:30:01.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:30:00 smithi122 ceph-mon[67998]: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-04-14T14:30:01.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:30:00 smithi122 ceph-mon[67998]: return _run_code(code, main_globals, None, 2024-04-14T14:30:01.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:30:00 smithi122 ceph-mon[67998]: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-04-14T14:30:01.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:30:00 smithi122 ceph-mon[67998]: exec(code, run_globals) 2024-04-14T14:30:01.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:30:00 smithi122 ceph-mon[67998]: File "/var/lib/ceph/dc326fc2-fa65-11ee-bc8e-c7b262605968/cephadm.de2a8faf91349910526b51da93d8bd42d57efdd2e93c32cd09ae64ecc1748ae9/__main__.py", line 5608, in 2024-04-14T14:30:01.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:30:00 smithi122 ceph-mon[67998]: File "/var/lib/ceph/dc326fc2-fa65-11ee-bc8e-c7b262605968/cephadm.de2a8faf91349910526b51da93d8bd42d57efdd2e93c32cd09ae64ecc1748ae9/__main__.py", line 5596, in main 2024-04-14T14:30:01.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:30:00 smithi122 ceph-mon[67998]: File "/var/lib/ceph/dc326fc2-fa65-11ee-bc8e-c7b262605968/cephadm.de2a8faf91349910526b51da93d8bd42d57efdd2e93c32cd09ae64ecc1748ae9/__main__.py", line 3096, in command_deploy_from 2024-04-14T14:30:01.096 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:30:00 smithi122 ceph-mon[67998]: File "/var/lib/ceph/dc326fc2-fa65-11ee-bc8e-c7b262605968/cephadm.de2a8faf91349910526b51da93d8bd42d57efdd2e93c32cd09ae64ecc1748ae9/__main__.py", line 3131, in _common_deploy 2024-04-14T14:30:01.096 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:30:00 smithi122 ceph-mon[67998]: File "/var/lib/ceph/dc326fc2-fa65-11ee-bc8e-c7b262605968/cephadm.de2a8faf91349910526b51da93d8bd42d57efdd2e93c32cd09ae64ecc1748ae9/__main__.py", line 3151, in _deploy_daemon_container 2024-04-14T14:30:01.096 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:30:00 smithi122 ceph-mon[67998]: File "/var/lib/ceph/dc326fc2-fa65-11ee-bc8e-c7b262605968/cephadm.de2a8faf91349910526b51da93d8bd42d57efdd2e93c32cd09ae64ecc1748ae9/__main__.py", line 1087, in deploy_daemon 2024-04-14T14:30:01.096 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:30:00 smithi122 ceph-mon[67998]: File "/var/lib/ceph/dc326fc2-fa65-11ee-bc8e-c7b262605968/cephadm.de2a8faf91349910526b51da93d8bd42d57efdd2e93c32cd09ae64ecc1748ae9/__main__.py", line 1223, in deploy_daemon_units 2024-04-14T14:30:01.096 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:30:00 smithi122 ceph-mon[67998]: File "/var/lib/ceph/dc326fc2-fa65-11ee-bc8e-c7b262605968/cephadm.de2a8faf91349910526b51da93d8bd42d57efdd2e93c32cd09ae64ecc1748ae9/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-04-14T14:30:01.096 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:30:00 smithi122 ceph-mon[67998]: RuntimeError: Failed command: systemctl start ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968@iscsi.foo.smithi069.yuumtr: Job for ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968@iscsi.foo.smithi069.yuumtr.service failed because the control process exited with error code. 2024-04-14T14:30:01.096 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:30:00 smithi122 ceph-mon[67998]: See "systemctl status ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968@iscsi.foo.smithi069.yuumtr.service" and "journalctl -xeu ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968@iscsi.foo.smithi069.yuumtr.service" for details. 2024-04-14T14:30:01.096 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:30:00 smithi122 ceph-mon[67998]: from='client.44844 -' entity='client.iscsi.foo.smithi069.yuumtr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T14:30:03.066 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:30:02 smithi069 ceph-mon[83680]: pgmap v337: 161 pgs: 161 active+clean; 585 KiB data, 266 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:30:03.067 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:30:02 smithi069 ceph-mon[88636]: pgmap v337: 161 pgs: 161 active+clean; 585 KiB data, 266 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:30:03.094 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:30:02 smithi122 ceph-mon[67998]: pgmap v337: 161 pgs: 161 active+clean; 585 KiB data, 266 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:30:05.066 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:30:04 smithi069 ceph-mon[83680]: pgmap v338: 161 pgs: 161 active+clean; 585 KiB data, 266 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:30:05.066 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:30:04 smithi069 ceph-mon[88636]: pgmap v338: 161 pgs: 161 active+clean; 585 KiB data, 266 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:30:05.094 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:30:04 smithi122 ceph-mon[67998]: pgmap v338: 161 pgs: 161 active+clean; 585 KiB data, 266 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:30:06.802 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 14 14:30:06 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-y[63038]: ::ffff:172.21.15.122 - - [14/Apr/2024:14:30:06] "GET /metrics HTTP/1.1" 200 38485 "" "Prometheus/2.43.0" 2024-04-14T14:30:07.066 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:30:06 smithi069 ceph-mon[83680]: pgmap v339: 161 pgs: 161 active+clean; 585 KiB data, 266 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:30:07.066 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:30:06 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T14:30:07.066 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:30:06 smithi069 ceph-mon[88636]: pgmap v339: 161 pgs: 161 active+clean; 585 KiB data, 266 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:30:07.066 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:30:06 smithi069 ceph-mon[88636]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T14:30:07.094 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:30:06 smithi122 ceph-mon[67998]: pgmap v339: 161 pgs: 161 active+clean; 585 KiB data, 266 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:30:07.094 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:30:06 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T14:30:09.066 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:30:08 smithi069 ceph-mon[83680]: pgmap v340: 161 pgs: 161 active+clean; 585 KiB data, 266 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:30:09.066 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:30:08 smithi069 ceph-mon[88636]: pgmap v340: 161 pgs: 161 active+clean; 585 KiB data, 266 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:30:09.094 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:30:08 smithi122 ceph-mon[67998]: pgmap v340: 161 pgs: 161 active+clean; 585 KiB data, 266 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:30:11.066 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:30:10 smithi069 ceph-mon[83680]: pgmap v341: 161 pgs: 161 active+clean; 585 KiB data, 266 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-04-14T14:30:11.066 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:30:10 smithi069 ceph-mon[83680]: from='client.44844 -' entity='client.iscsi.foo.smithi069.yuumtr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T14:30:11.066 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:30:10 smithi069 ceph-mon[88636]: pgmap v341: 161 pgs: 161 active+clean; 585 KiB data, 266 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-04-14T14:30:11.066 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:30:10 smithi069 ceph-mon[88636]: from='client.44844 -' entity='client.iscsi.foo.smithi069.yuumtr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T14:30:11.094 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:30:10 smithi122 ceph-mon[67998]: pgmap v341: 161 pgs: 161 active+clean; 585 KiB data, 266 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-04-14T14:30:11.094 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:30:10 smithi122 ceph-mon[67998]: from='client.44844 -' entity='client.iscsi.foo.smithi069.yuumtr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T14:30:13.066 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:30:12 smithi069 ceph-mon[83680]: pgmap v342: 161 pgs: 161 active+clean; 585 KiB data, 266 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:30:13.069 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:30:12 smithi069 ceph-mon[88636]: pgmap v342: 161 pgs: 161 active+clean; 585 KiB data, 266 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:30:13.094 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:30:12 smithi122 ceph-mon[67998]: pgmap v342: 161 pgs: 161 active+clean; 585 KiB data, 266 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:30:15.066 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:30:14 smithi069 ceph-mon[83680]: pgmap v343: 161 pgs: 161 active+clean; 585 KiB data, 266 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-04-14T14:30:15.066 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:30:14 smithi069 ceph-mon[88636]: pgmap v343: 161 pgs: 161 active+clean; 585 KiB data, 266 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-04-14T14:30:15.094 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:30:14 smithi122 ceph-mon[67998]: pgmap v343: 161 pgs: 161 active+clean; 585 KiB data, 266 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-04-14T14:30:16.816 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 14 14:30:16 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-y[63038]: ::ffff:172.21.15.122 - - [14/Apr/2024:14:30:16] "GET /metrics HTTP/1.1" 200 38484 "" "Prometheus/2.43.0" 2024-04-14T14:30:17.094 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:30:16 smithi122 ceph-mon[67998]: pgmap v344: 161 pgs: 161 active+clean; 585 KiB data, 266 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-04-14T14:30:17.316 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:30:16 smithi069 ceph-mon[83680]: pgmap v344: 161 pgs: 161 active+clean; 585 KiB data, 266 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-04-14T14:30:17.316 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:30:16 smithi069 ceph-mon[88636]: pgmap v344: 161 pgs: 161 active+clean; 585 KiB data, 266 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-04-14T14:30:19.094 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:30:18 smithi122 ceph-mon[67998]: pgmap v345: 161 pgs: 161 active+clean; 585 KiB data, 266 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:30:19.316 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:30:18 smithi069 ceph-mon[83680]: pgmap v345: 161 pgs: 161 active+clean; 585 KiB data, 266 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:30:19.316 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:30:18 smithi069 ceph-mon[88636]: pgmap v345: 161 pgs: 161 active+clean; 585 KiB data, 266 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:30:20.516 DEBUG:teuthology.orchestra.run.smithi069:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid dc326fc2-fa65-11ee-bc8e-c7b262605968 -e sha1=a971fd5f5719038c70b817bafacd6d5400b1d888 -- bash -c 'ceph orch ps' 2024-04-14T14:30:21.094 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:30:20 smithi122 ceph-mon[67998]: pgmap v346: 161 pgs: 161 active+clean; 585 KiB data, 266 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-04-14T14:30:21.094 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:30:20 smithi122 ceph-mon[67998]: from='client.44844 -' entity='client.iscsi.foo.smithi069.yuumtr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T14:30:21.131 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:30:20 smithi069 ceph-mon[83680]: pgmap v346: 161 pgs: 161 active+clean; 585 KiB data, 266 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-04-14T14:30:21.131 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:30:20 smithi069 ceph-mon[83680]: from='client.44844 -' entity='client.iscsi.foo.smithi069.yuumtr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T14:30:21.131 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:30:20 smithi069 ceph-mon[88636]: pgmap v346: 161 pgs: 161 active+clean; 585 KiB data, 266 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-04-14T14:30:21.131 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:30:20 smithi069 ceph-mon[88636]: from='client.44844 -' entity='client.iscsi.foo.smithi069.yuumtr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T14:30:22.049 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:30:21 smithi069 ceph-mon[88636]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T14:30:22.050 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:30:21 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T14:30:22.094 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:30:21 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T14:30:22.484 INFO:teuthology.orchestra.run.smithi069.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-14T14:30:22.484 INFO:teuthology.orchestra.run.smithi069.stdout:alertmanager.a smithi069 *:9093,9094 running (12m) 2m ago 33m 25.9M - 0.25.0 c8568f914cd2 7524f106e7bd 2024-04-14T14:30:22.484 INFO:teuthology.orchestra.run.smithi069.stdout:grafana.a smithi122 *:3000 running (11m) 2m ago 33m 82.4M - 9.4.12 2bacad6d85d8 5685178e3eba 2024-04-14T14:30:22.485 INFO:teuthology.orchestra.run.smithi069.stdout:iscsi.foo.smithi069.yuumtr smithi069 running (13m) 2m ago 32m 62.9M - 3.5 e1d6a67b021e 38cc81ea8cfa 2024-04-14T14:30:22.485 INFO:teuthology.orchestra.run.smithi069.stdout:mgr.x smithi122 *:8443,9283,8765 running (13m) 2m ago 38m 455M - 19.0.0-2778-ga971fd5f b96c509eef84 e29977590b43 2024-04-14T14:30:22.485 INFO:teuthology.orchestra.run.smithi069.stdout:mgr.y smithi069 *:8443,9283,8765 running (22m) 2m ago 39m 513M - 19.0.0-2778-ga971fd5f b96c509eef84 7af3d29d0e5f 2024-04-14T14:30:22.485 INFO:teuthology.orchestra.run.smithi069.stdout:mon.a smithi069 running (9m) 2m ago 40m 63.2M 2048M 19.0.0-2778-ga971fd5f b96c509eef84 8bcd6b3d3846 2024-04-14T14:30:22.485 INFO:teuthology.orchestra.run.smithi069.stdout:mon.b smithi122 running (10m) 2m ago 38m 49.5M 2048M 19.0.0-2778-ga971fd5f b96c509eef84 05f437a1a9a1 2024-04-14T14:30:22.485 INFO:teuthology.orchestra.run.smithi069.stdout:mon.c smithi069 running (9m) 2m ago 38m 44.8M 2048M 19.0.0-2778-ga971fd5f b96c509eef84 84aa95f11d0c 2024-04-14T14:30:22.485 INFO:teuthology.orchestra.run.smithi069.stdout:node-exporter.a smithi069 *:9100 running (12m) 2m ago 34m 15.7M - 1.5.0 0da6a335fe13 0de0eb2171b9 2024-04-14T14:30:22.485 INFO:teuthology.orchestra.run.smithi069.stdout:node-exporter.b smithi122 *:9100 running (12m) 2m ago 34m 16.3M - 1.5.0 0da6a335fe13 67877bf02d8e 2024-04-14T14:30:22.485 INFO:teuthology.orchestra.run.smithi069.stdout:osd.0 smithi069 running (8m) 2m ago 37m 62.2M 2992M 19.0.0-2778-ga971fd5f b96c509eef84 a0e57e3fb494 2024-04-14T14:30:22.485 INFO:teuthology.orchestra.run.smithi069.stdout:osd.1 smithi069 running (7m) 2m ago 37m 62.6M 2992M 19.0.0-2778-ga971fd5f b96c509eef84 cf73d17919bb 2024-04-14T14:30:22.485 INFO:teuthology.orchestra.run.smithi069.stdout:osd.2 smithi069 running (6m) 2m ago 37m 49.7M 2992M 19.0.0-2778-ga971fd5f b96c509eef84 c0efba5ef451 2024-04-14T14:30:22.485 INFO:teuthology.orchestra.run.smithi069.stdout:osd.3 smithi069 running (6m) 2m ago 36m 60.6M 2992M 19.0.0-2778-ga971fd5f b96c509eef84 d13460da0a65 2024-04-14T14:30:22.485 INFO:teuthology.orchestra.run.smithi069.stdout:osd.4 smithi122 running (5m) 2m ago 36m 58.6M 3248M 19.0.0-2778-ga971fd5f b96c509eef84 7a87a024f984 2024-04-14T14:30:22.485 INFO:teuthology.orchestra.run.smithi069.stdout:osd.5 smithi122 running (4m) 2m ago 36m 56.7M 3248M 19.0.0-2778-ga971fd5f b96c509eef84 02b623d44db3 2024-04-14T14:30:22.486 INFO:teuthology.orchestra.run.smithi069.stdout:osd.6 smithi122 running (4m) 2m ago 35m 53.9M 3248M 19.0.0-2778-ga971fd5f b96c509eef84 72f85ece3e75 2024-04-14T14:30:22.486 INFO:teuthology.orchestra.run.smithi069.stdout:osd.7 smithi122 running (3m) 2m ago 35m 55.3M 3248M 19.0.0-2778-ga971fd5f b96c509eef84 0c53c3cd2bc0 2024-04-14T14:30:22.486 INFO:teuthology.orchestra.run.smithi069.stdout:prometheus.a smithi122 *:9095 running (12m) 2m ago 34m 71.1M - 2.43.0 a07b618ecd1d 5c1f41d3bbe6 2024-04-14T14:30:22.486 INFO:teuthology.orchestra.run.smithi069.stdout:rgw.foo.smithi069.wesedz smithi069 *:8000 running (2m) 2m ago 33m 77.2M - 19.0.0-2778-ga971fd5f b96c509eef84 f135fd86285d 2024-04-14T14:30:22.486 INFO:teuthology.orchestra.run.smithi069.stdout:rgw.foo.smithi122.jiobyv smithi122 *:8000 running (2m) 2m ago 33m 76.3M - 19.0.0-2778-ga971fd5f b96c509eef84 7d37b0b84199 2024-04-14T14:30:23.066 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:30:22 smithi069 ceph-mon[83680]: pgmap v347: 161 pgs: 161 active+clean; 585 KiB data, 266 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:30:23.066 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:30:22 smithi069 ceph-mon[83680]: from='client.44919 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T14:30:23.066 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:30:22 smithi069 ceph-mon[88636]: pgmap v347: 161 pgs: 161 active+clean; 585 KiB data, 266 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:30:23.066 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:30:22 smithi069 ceph-mon[88636]: from='client.44919 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T14:30:23.094 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:30:22 smithi122 ceph-mon[67998]: pgmap v347: 161 pgs: 161 active+clean; 585 KiB data, 266 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:30:23.094 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:30:22 smithi122 ceph-mon[67998]: from='client.44919 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T14:30:23.160 DEBUG:teuthology.orchestra.run.smithi069:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid dc326fc2-fa65-11ee-bc8e-c7b262605968 -e sha1=a971fd5f5719038c70b817bafacd6d5400b1d888 -- bash -c 'ceph versions' 2024-04-14T14:30:25.066 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:30:24 smithi069 ceph-mon[83680]: pgmap v348: 161 pgs: 161 active+clean; 585 KiB data, 266 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:30:25.066 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:30:24 smithi069 ceph-mon[88636]: pgmap v348: 161 pgs: 161 active+clean; 585 KiB data, 266 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:30:25.094 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:30:24 smithi122 ceph-mon[67998]: pgmap v348: 161 pgs: 161 active+clean; 585 KiB data, 266 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:30:25.190 INFO:teuthology.orchestra.run.smithi069.stdout:{ 2024-04-14T14:30:25.190 INFO:teuthology.orchestra.run.smithi069.stdout: "mon": { 2024-04-14T14:30:25.190 INFO:teuthology.orchestra.run.smithi069.stdout: "ceph version 19.0.0-2778-ga971fd5f (a971fd5f5719038c70b817bafacd6d5400b1d888) squid (dev)": 3 2024-04-14T14:30:25.190 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-14T14:30:25.190 INFO:teuthology.orchestra.run.smithi069.stdout: "mgr": { 2024-04-14T14:30:25.190 INFO:teuthology.orchestra.run.smithi069.stdout: "ceph version 19.0.0-2778-ga971fd5f (a971fd5f5719038c70b817bafacd6d5400b1d888) squid (dev)": 2 2024-04-14T14:30:25.191 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-14T14:30:25.191 INFO:teuthology.orchestra.run.smithi069.stdout: "osd": { 2024-04-14T14:30:25.191 INFO:teuthology.orchestra.run.smithi069.stdout: "ceph version 19.0.0-2778-ga971fd5f (a971fd5f5719038c70b817bafacd6d5400b1d888) squid (dev)": 8 2024-04-14T14:30:25.191 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-14T14:30:25.191 INFO:teuthology.orchestra.run.smithi069.stdout: "rgw": { 2024-04-14T14:30:25.191 INFO:teuthology.orchestra.run.smithi069.stdout: "ceph version 19.0.0-2778-ga971fd5f (a971fd5f5719038c70b817bafacd6d5400b1d888) squid (dev)": 2 2024-04-14T14:30:25.191 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-14T14:30:25.191 INFO:teuthology.orchestra.run.smithi069.stdout: "overall": { 2024-04-14T14:30:25.191 INFO:teuthology.orchestra.run.smithi069.stdout: "ceph version 19.0.0-2778-ga971fd5f (a971fd5f5719038c70b817bafacd6d5400b1d888) squid (dev)": 15 2024-04-14T14:30:25.191 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-14T14:30:25.191 INFO:teuthology.orchestra.run.smithi069.stdout:} 2024-04-14T14:30:25.820 DEBUG:teuthology.orchestra.run.smithi069:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid dc326fc2-fa65-11ee-bc8e-c7b262605968 -e sha1=a971fd5f5719038c70b817bafacd6d5400b1d888 -- bash -c 'ceph orch upgrade status' 2024-04-14T14:30:26.094 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:30:25 smithi122 ceph-mon[67998]: from='client.? 172.21.15.69:0/2148296559' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T14:30:26.123 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:30:25 smithi069 ceph-mon[83680]: from='client.? 172.21.15.69:0/2148296559' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T14:30:26.123 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:30:25 smithi069 ceph-mon[88636]: from='client.? 172.21.15.69:0/2148296559' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T14:30:26.816 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 14 14:30:26 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-y[63038]: ::ffff:172.21.15.122 - - [14/Apr/2024:14:30:26] "GET /metrics HTTP/1.1" 200 38489 "" "Prometheus/2.43.0" 2024-04-14T14:30:27.094 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:30:26 smithi122 ceph-mon[67998]: pgmap v349: 161 pgs: 161 active+clean; 585 KiB data, 266 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:30:27.128 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:30:26 smithi069 ceph-mon[83680]: pgmap v349: 161 pgs: 161 active+clean; 585 KiB data, 266 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:30:27.128 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:30:26 smithi069 ceph-mon[88636]: pgmap v349: 161 pgs: 161 active+clean; 585 KiB data, 266 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:30:27.849 INFO:teuthology.orchestra.run.smithi069.stdout:{ 2024-04-14T14:30:27.850 INFO:teuthology.orchestra.run.smithi069.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888", 2024-04-14T14:30:27.850 INFO:teuthology.orchestra.run.smithi069.stdout: "in_progress": true, 2024-04-14T14:30:27.850 INFO:teuthology.orchestra.run.smithi069.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-14T14:30:27.850 INFO:teuthology.orchestra.run.smithi069.stdout: "services_complete": [ 2024-04-14T14:30:27.850 INFO:teuthology.orchestra.run.smithi069.stdout: "rgw", 2024-04-14T14:30:27.850 INFO:teuthology.orchestra.run.smithi069.stdout: "mon", 2024-04-14T14:30:27.850 INFO:teuthology.orchestra.run.smithi069.stdout: "osd", 2024-04-14T14:30:27.850 INFO:teuthology.orchestra.run.smithi069.stdout: "mgr" 2024-04-14T14:30:27.850 INFO:teuthology.orchestra.run.smithi069.stdout: ], 2024-04-14T14:30:27.850 INFO:teuthology.orchestra.run.smithi069.stdout: "progress": "15/21 daemons upgraded", 2024-04-14T14:30:27.850 INFO:teuthology.orchestra.run.smithi069.stdout: "message": "Error: UPGRADE_REDEPLOY_DAEMON: Upgrading daemon iscsi.foo.smithi069.yuumtr on host smithi069 failed.", 2024-04-14T14:30:27.850 INFO:teuthology.orchestra.run.smithi069.stdout: "is_paused": true 2024-04-14T14:30:27.851 INFO:teuthology.orchestra.run.smithi069.stdout:} 2024-04-14T14:30:28.467 DEBUG:teuthology.orchestra.run.smithi069:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid dc326fc2-fa65-11ee-bc8e-c7b262605968 -e sha1=a971fd5f5719038c70b817bafacd6d5400b1d888 -- bash -c 'ceph health detail' 2024-04-14T14:30:29.316 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:30:29 smithi069 ceph-mon[83680]: pgmap v350: 161 pgs: 161 active+clean; 585 KiB data, 266 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:30:29.316 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:30:29 smithi069 ceph-mon[83680]: from='client.44931 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T14:30:29.316 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:30:29 smithi069 ceph-mon[88636]: pgmap v350: 161 pgs: 161 active+clean; 585 KiB data, 266 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:30:29.316 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:30:29 smithi069 ceph-mon[88636]: from='client.44931 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T14:30:29.594 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:30:29 smithi122 ceph-mon[67998]: pgmap v350: 161 pgs: 161 active+clean; 585 KiB data, 266 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:30:29.594 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:30:29 smithi122 ceph-mon[67998]: from='client.44931 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T14:30:30.508 INFO:teuthology.orchestra.run.smithi069.stdout:HEALTH_WARN Upgrading daemon iscsi.foo.smithi069.yuumtr on host smithi069 failed. 2024-04-14T14:30:30.508 INFO:teuthology.orchestra.run.smithi069.stdout:[WRN] UPGRADE_REDEPLOY_DAEMON: Upgrading daemon iscsi.foo.smithi069.yuumtr on host smithi069 failed. 2024-04-14T14:30:30.508 INFO:teuthology.orchestra.run.smithi069.stdout: Upgrade daemon: iscsi.foo.smithi069.yuumtr: cephadm exited with an error code: 1, stderr: Redeploy daemon iscsi.foo.smithi069.yuumtr ... 2024-04-14T14:30:30.508 INFO:teuthology.orchestra.run.smithi069.stdout:Creating ceph-iscsi config... 2024-04-14T14:30:30.508 INFO:teuthology.orchestra.run.smithi069.stdout:Write file: /var/lib/ceph/dc326fc2-fa65-11ee-bc8e-c7b262605968/iscsi.foo.smithi069.yuumtr/iscsi-gateway.cfg 2024-04-14T14:30:30.508 INFO:teuthology.orchestra.run.smithi069.stdout:Write file: /var/lib/ceph/dc326fc2-fa65-11ee-bc8e-c7b262605968/iscsi.foo.smithi069.yuumtr/tcmu-runner-entrypoint.sh 2024-04-14T14:30:30.508 INFO:teuthology.orchestra.run.smithi069.stdout:Failed to trim old cgroups /sys/fs/cgroup/system.slice/system-ceph\x2ddc326fc2\x2dfa65\x2d11ee\x2dbc8e\x2dc7b262605968.slice/ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968@iscsi.foo.smithi069.yuumtr.service 2024-04-14T14:30:30.508 INFO:teuthology.orchestra.run.smithi069.stdout:Non-zero exit code 1 from systemctl start ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968@iscsi.foo.smithi069.yuumtr 2024-04-14T14:30:30.508 INFO:teuthology.orchestra.run.smithi069.stdout:systemctl: stderr Job for ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968@iscsi.foo.smithi069.yuumtr.service failed because the control process exited with error code. 2024-04-14T14:30:30.508 INFO:teuthology.orchestra.run.smithi069.stdout:systemctl: stderr See "systemctl status ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968@iscsi.foo.smithi069.yuumtr.service" and "journalctl -xeu ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968@iscsi.foo.smithi069.yuumtr.service" for details. 2024-04-14T14:30:30.509 INFO:teuthology.orchestra.run.smithi069.stdout:Traceback (most recent call last): 2024-04-14T14:30:30.509 INFO:teuthology.orchestra.run.smithi069.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-04-14T14:30:30.509 INFO:teuthology.orchestra.run.smithi069.stdout: return _run_code(code, main_globals, None, 2024-04-14T14:30:30.509 INFO:teuthology.orchestra.run.smithi069.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-04-14T14:30:30.509 INFO:teuthology.orchestra.run.smithi069.stdout: exec(code, run_globals) 2024-04-14T14:30:30.509 INFO:teuthology.orchestra.run.smithi069.stdout: File "/var/lib/ceph/dc326fc2-fa65-11ee-bc8e-c7b262605968/cephadm.de2a8faf91349910526b51da93d8bd42d57efdd2e93c32cd09ae64ecc1748ae9/__main__.py", line 5608, in 2024-04-14T14:30:30.509 INFO:teuthology.orchestra.run.smithi069.stdout: File "/var/lib/ceph/dc326fc2-fa65-11ee-bc8e-c7b262605968/cephadm.de2a8faf91349910526b51da93d8bd42d57efdd2e93c32cd09ae64ecc1748ae9/__main__.py", line 5596, in main 2024-04-14T14:30:30.509 INFO:teuthology.orchestra.run.smithi069.stdout: File "/var/lib/ceph/dc326fc2-fa65-11ee-bc8e-c7b262605968/cephadm.de2a8faf91349910526b51da93d8bd42d57efdd2e93c32cd09ae64ecc1748ae9/__main__.py", line 3096, in command_deploy_from 2024-04-14T14:30:30.509 INFO:teuthology.orchestra.run.smithi069.stdout: File "/var/lib/ceph/dc326fc2-fa65-11ee-bc8e-c7b262605968/cephadm.de2a8faf91349910526b51da93d8bd42d57efdd2e93c32cd09ae64ecc1748ae9/__main__.py", line 3131, in _common_deploy 2024-04-14T14:30:30.509 INFO:teuthology.orchestra.run.smithi069.stdout: File "/var/lib/ceph/dc326fc2-fa65-11ee-bc8e-c7b262605968/cephadm.de2a8faf91349910526b51da93d8bd42d57efdd2e93c32cd09ae64ecc1748ae9/__main__.py", line 3151, in _deploy_daemon_container 2024-04-14T14:30:30.509 INFO:teuthology.orchestra.run.smithi069.stdout: File "/var/lib/ceph/dc326fc2-fa65-11ee-bc8e-c7b262605968/cephadm.de2a8faf91349910526b51da93d8bd42d57efdd2e93c32cd09ae64ecc1748ae9/__main__.py", line 1087, in deploy_daemon 2024-04-14T14:30:30.509 INFO:teuthology.orchestra.run.smithi069.stdout: File "/var/lib/ceph/dc326fc2-fa65-11ee-bc8e-c7b262605968/cephadm.de2a8faf91349910526b51da93d8bd42d57efdd2e93c32cd09ae64ecc1748ae9/__main__.py", line 1223, in deploy_daemon_units 2024-04-14T14:30:30.509 INFO:teuthology.orchestra.run.smithi069.stdout: File "/var/lib/ceph/dc326fc2-fa65-11ee-bc8e-c7b262605968/cephadm.de2a8faf91349910526b51da93d8bd42d57efdd2e93c32cd09ae64ecc1748ae9/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-04-14T14:30:30.509 INFO:teuthology.orchestra.run.smithi069.stdout:RuntimeError: Failed command: systemctl start ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968@iscsi.foo.smithi069.yuumtr: Job for ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968@iscsi.foo.smithi069.yuumtr.service failed because the control process exited with error code. 2024-04-14T14:30:30.510 INFO:teuthology.orchestra.run.smithi069.stdout:See "systemctl status ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968@iscsi.foo.smithi069.yuumtr.service" and "journalctl -xeu ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968@iscsi.foo.smithi069.yuumtr.service" for details. 2024-04-14T14:30:31.175 DEBUG:teuthology.orchestra.run.smithi069:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid dc326fc2-fa65-11ee-bc8e-c7b262605968 -e sha1=a971fd5f5719038c70b817bafacd6d5400b1d888 -- bash -c 'ceph versions | jq -e '"'"'.overall | length == 1'"'"'' 2024-04-14T14:30:31.482 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:30:31 smithi069 ceph-mon[83680]: pgmap v351: 161 pgs: 161 active+clean; 585 KiB data, 266 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:30:31.482 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:30:31 smithi069 ceph-mon[83680]: from='client.? 172.21.15.69:0/4127592777' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-14T14:30:31.482 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:30:31 smithi069 ceph-mon[83680]: from='client.44844 -' entity='client.iscsi.foo.smithi069.yuumtr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T14:30:31.483 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:30:31 smithi069 ceph-mon[88636]: pgmap v351: 161 pgs: 161 active+clean; 585 KiB data, 266 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:30:31.483 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:30:31 smithi069 ceph-mon[88636]: from='client.? 172.21.15.69:0/4127592777' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-14T14:30:31.483 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:30:31 smithi069 ceph-mon[88636]: from='client.44844 -' entity='client.iscsi.foo.smithi069.yuumtr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T14:30:31.594 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:30:31 smithi122 ceph-mon[67998]: pgmap v351: 161 pgs: 161 active+clean; 585 KiB data, 266 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:30:31.594 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:30:31 smithi122 ceph-mon[67998]: from='client.? 172.21.15.69:0/4127592777' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-14T14:30:31.594 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:30:31 smithi122 ceph-mon[67998]: from='client.44844 -' entity='client.iscsi.foo.smithi069.yuumtr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T14:30:33.184 INFO:teuthology.orchestra.run.smithi069.stdout:true 2024-04-14T14:30:33.480 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:30:33 smithi069 ceph-mon[83680]: pgmap v352: 161 pgs: 161 active+clean; 585 KiB data, 266 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:30:33.480 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:30:33 smithi069 ceph-mon[83680]: from='client.? 172.21.15.69:0/3173120175' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T14:30:33.480 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:30:33 smithi069 ceph-mon[88636]: pgmap v352: 161 pgs: 161 active+clean; 585 KiB data, 266 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:30:33.480 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:30:33 smithi069 ceph-mon[88636]: from='client.? 172.21.15.69:0/3173120175' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T14:30:33.594 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:30:33 smithi122 ceph-mon[67998]: pgmap v352: 161 pgs: 161 active+clean; 585 KiB data, 266 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:30:33.594 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:30:33 smithi122 ceph-mon[67998]: from='client.? 172.21.15.69:0/3173120175' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T14:30:33.778 DEBUG:teuthology.orchestra.run.smithi069:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid dc326fc2-fa65-11ee-bc8e-c7b262605968 -e sha1=a971fd5f5719038c70b817bafacd6d5400b1d888 -- bash -c 'ceph versions | jq -e '"'"'.overall | keys'"'"' | grep $sha1' 2024-04-14T14:30:35.306 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:30:35 smithi069 ceph-mon[83680]: pgmap v353: 161 pgs: 161 active+clean; 585 KiB data, 266 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:30:35.307 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:30:35 smithi069 ceph-mon[88636]: pgmap v353: 161 pgs: 161 active+clean; 585 KiB data, 266 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:30:35.594 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:30:35 smithi122 ceph-mon[67998]: pgmap v353: 161 pgs: 161 active+clean; 585 KiB data, 266 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:30:35.803 INFO:teuthology.orchestra.run.smithi069.stdout: "ceph version 19.0.0-2778-ga971fd5f (a971fd5f5719038c70b817bafacd6d5400b1d888) squid (dev)" 2024-04-14T14:30:36.368 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:30:36 smithi069 ceph-mon[83680]: from='client.? 172.21.15.69:0/3955759950' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T14:30:36.369 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:30:36 smithi069 ceph-mon[88636]: from='client.? 172.21.15.69:0/3955759950' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T14:30:36.392 DEBUG:teuthology.orchestra.run.smithi069:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid dc326fc2-fa65-11ee-bc8e-c7b262605968 -e sha1=a971fd5f5719038c70b817bafacd6d5400b1d888 -- bash -c 'ceph orch ls | grep '"'"'^osd '"'"'' 2024-04-14T14:30:36.594 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:30:36 smithi122 ceph-mon[67998]: from='client.? 172.21.15.69:0/3955759950' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-14T14:30:36.703 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 14 14:30:36 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-y[63038]: ::ffff:172.21.15.122 - - [14/Apr/2024:14:30:36] "GET /metrics HTTP/1.1" 200 38489 "" "Prometheus/2.43.0" 2024-04-14T14:30:37.566 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:30:37 smithi069 ceph-mon[83680]: pgmap v354: 161 pgs: 161 active+clean; 585 KiB data, 266 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:30:37.567 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:30:37 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T14:30:37.567 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:30:37 smithi069 ceph-mon[88636]: pgmap v354: 161 pgs: 161 active+clean; 585 KiB data, 266 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:30:37.567 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:30:37 smithi069 ceph-mon[88636]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T14:30:37.594 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:30:37 smithi122 ceph-mon[67998]: pgmap v354: 161 pgs: 161 active+clean; 585 KiB data, 266 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:30:37.594 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:30:37 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-14T14:30:38.318 INFO:teuthology.orchestra.run.smithi069.stdout:osd 8 2m ago - 2024-04-14T14:30:38.908 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-04-14T14:30:38.919 INFO:tasks.cephadm:Running commands on role mon.a host ubuntu@smithi069.front.sepia.ceph.com 2024-04-14T14:30:38.919 DEBUG:teuthology.orchestra.run.smithi069:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid dc326fc2-fa65-11ee-bc8e-c7b262605968 -- bash -c 'ceph orch upgrade ls' 2024-04-14T14:30:39.552 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:30:39 smithi069 ceph-mon[83680]: pgmap v355: 161 pgs: 161 active+clean; 585 KiB data, 266 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:30:39.553 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:30:39 smithi069 ceph-mon[83680]: from='client.44955 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T14:30:39.553 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:30:39 smithi069 ceph-mon[88636]: pgmap v355: 161 pgs: 161 active+clean; 585 KiB data, 266 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:30:39.553 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:30:39 smithi069 ceph-mon[88636]: from='client.44955 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T14:30:39.594 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:30:39 smithi122 ceph-mon[67998]: pgmap v355: 161 pgs: 161 active+clean; 585 KiB data, 266 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:30:39.594 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:30:39 smithi122 ceph-mon[67998]: from='client.44955 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T14:30:41.528 INFO:teuthology.orchestra.run.smithi069.stdout:{ 2024-04-14T14:30:41.528 INFO:teuthology.orchestra.run.smithi069.stdout: "image": "quay.io/ceph/ceph", 2024-04-14T14:30:41.528 INFO:teuthology.orchestra.run.smithi069.stdout: "registry": "quay.io", 2024-04-14T14:30:41.528 INFO:teuthology.orchestra.run.smithi069.stdout: "bare_image": "ceph/ceph", 2024-04-14T14:30:41.528 INFO:teuthology.orchestra.run.smithi069.stdout: "versions": [] 2024-04-14T14:30:41.528 INFO:teuthology.orchestra.run.smithi069.stdout:} 2024-04-14T14:30:41.541 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:30:41 smithi069 ceph-mon[88636]: pgmap v356: 161 pgs: 161 active+clean; 585 KiB data, 266 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:30:41.542 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:30:41 smithi069 ceph-mon[88636]: from='client.44844 -' entity='client.iscsi.foo.smithi069.yuumtr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T14:30:41.542 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:30:41 smithi069 ceph-mon[88636]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T14:30:41.542 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:30:41 smithi069 ceph-mon[88636]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-14T14:30:41.542 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:30:41 smithi069 ceph-mon[88636]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:30:41.542 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:30:41 smithi069 ceph-mon[83680]: pgmap v356: 161 pgs: 161 active+clean; 585 KiB data, 266 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:30:41.542 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:30:41 smithi069 ceph-mon[83680]: from='client.44844 -' entity='client.iscsi.foo.smithi069.yuumtr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T14:30:41.542 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:30:41 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T14:30:41.542 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:30:41 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-14T14:30:41.542 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:30:41 smithi069 ceph-mon[83680]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:30:41.594 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:30:41 smithi122 ceph-mon[67998]: pgmap v356: 161 pgs: 161 active+clean; 585 KiB data, 266 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:30:41.594 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:30:41 smithi122 ceph-mon[67998]: from='client.44844 -' entity='client.iscsi.foo.smithi069.yuumtr' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-14T14:30:41.594 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:30:41 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-14T14:30:41.594 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:30:41 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-14T14:30:41.594 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:30:41 smithi122 ceph-mon[67998]: from='mgr.44103 172.21.15.69:0/3940961253' entity='mgr.y' 2024-04-14T14:30:42.120 DEBUG:teuthology.orchestra.run.smithi069:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid dc326fc2-fa65-11ee-bc8e-c7b262605968 -- bash -c 'ceph orch upgrade ls --image quay.io/ceph/ceph --show-all-versions | grep 16.2.0' 2024-04-14T14:30:42.426 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:30:42 smithi069 ceph-mon[83680]: from='client.54904 -' entity='client.admin' cmd=[{"prefix": "orch upgrade ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T14:30:42.426 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:30:42 smithi069 ceph-mon[88636]: from='client.54904 -' entity='client.admin' cmd=[{"prefix": "orch upgrade ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T14:30:42.594 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:30:42 smithi122 ceph-mon[67998]: from='client.54904 -' entity='client.admin' cmd=[{"prefix": "orch upgrade ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T14:30:43.452 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:30:43 smithi069 ceph-mon[83680]: pgmap v357: 161 pgs: 161 active+clean; 585 KiB data, 266 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:30:43.452 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:30:43 smithi069 ceph-mon[88636]: pgmap v357: 161 pgs: 161 active+clean; 585 KiB data, 266 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:30:43.594 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:30:43 smithi122 ceph-mon[67998]: pgmap v357: 161 pgs: 161 active+clean; 585 KiB data, 266 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-14T14:30:44.604 INFO:teuthology.orchestra.run.smithi069.stdout: "16.2.0", 2024-04-14T14:30:45.217 DEBUG:teuthology.orchestra.run.smithi069:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid dc326fc2-fa65-11ee-bc8e-c7b262605968 -- bash -c 'ceph orch upgrade ls --image quay.io/ceph/ceph --tags | grep v16.2.2' 2024-04-14T14:30:45.523 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:30:45 smithi069 ceph-mon[83680]: pgmap v358: 161 pgs: 161 active+clean; 585 KiB data, 266 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:30:45.523 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:30:45 smithi069 ceph-mon[83680]: from='client.44967 -' entity='client.admin' cmd=[{"prefix": "orch upgrade ls", "image": "quay.io/ceph/ceph", "show_all_versions": true, "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T14:30:45.523 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:30:45 smithi069 ceph-mon[88636]: pgmap v358: 161 pgs: 161 active+clean; 585 KiB data, 266 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:30:45.523 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:30:45 smithi069 ceph-mon[88636]: from='client.44967 -' entity='client.admin' cmd=[{"prefix": "orch upgrade ls", "image": "quay.io/ceph/ceph", "show_all_versions": true, "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T14:30:45.594 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:30:45 smithi122 ceph-mon[67998]: pgmap v358: 161 pgs: 161 active+clean; 585 KiB data, 266 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:30:45.594 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:30:45 smithi122 ceph-mon[67998]: from='client.44967 -' entity='client.admin' cmd=[{"prefix": "orch upgrade ls", "image": "quay.io/ceph/ceph", "show_all_versions": true, "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T14:30:46.712 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 14 14:30:46 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-y[63038]: ::ffff:172.21.15.122 - - [14/Apr/2024:14:30:46] "GET /metrics HTTP/1.1" 200 38484 "" "Prometheus/2.43.0" 2024-04-14T14:30:47.566 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:30:47 smithi069 ceph-mon[83680]: pgmap v359: 161 pgs: 161 active+clean; 585 KiB data, 266 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:30:47.566 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:30:47 smithi069 ceph-mon[88636]: pgmap v359: 161 pgs: 161 active+clean; 585 KiB data, 266 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:30:47.594 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:30:47 smithi122 ceph-mon[67998]: pgmap v359: 161 pgs: 161 active+clean; 585 KiB data, 266 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-14T14:30:47.728 INFO:teuthology.orchestra.run.smithi069.stdout: "v16.2.2", 2024-04-14T14:30:47.728 INFO:teuthology.orchestra.run.smithi069.stdout: "v16.2.2-20210505", 2024-04-14T14:30:48.340 DEBUG:teuthology.run_tasks:Unwinding manager cephadm 2024-04-14T14:30:48.350 INFO:tasks.cephadm:Teardown begin 2024-04-14T14:30:48.350 DEBUG:teuthology.orchestra.run.smithi069:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-04-14T14:30:48.380 DEBUG:teuthology.orchestra.run.smithi122:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-04-14T14:30:48.415 INFO:tasks.cephadm:Cleaning up testdir ceph.* files... 2024-04-14T14:30:48.415 DEBUG:teuthology.orchestra.run.smithi069:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-04-14T14:30:48.434 DEBUG:teuthology.orchestra.run.smithi122:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-04-14T14:30:48.473 INFO:tasks.cephadm:Stopping all daemons... 2024-04-14T14:30:48.474 INFO:tasks.cephadm.mon.a:Stopping mon.a... 2024-04-14T14:30:48.474 DEBUG:teuthology.orchestra.run.smithi069:> sudo systemctl stop ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968@mon.a 2024-04-14T14:30:48.566 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:30:48 smithi069 ceph-mon[83680]: from='client.44973 -' entity='client.admin' cmd=[{"prefix": "orch upgrade ls", "image": "quay.io/ceph/ceph", "tags": true, "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T14:30:48.570 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:30:48 smithi069 systemd[1]: Stopping Ceph mon.a for dc326fc2-fa65-11ee-bc8e-c7b262605968... 2024-04-14T14:30:48.570 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:30:48 smithi069 ceph-mon[88636]: from='client.44973 -' entity='client.admin' cmd=[{"prefix": "orch upgrade ls", "image": "quay.io/ceph/ceph", "tags": true, "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T14:30:48.594 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:30:48 smithi122 ceph-mon[67998]: from='client.44973 -' entity='client.admin' cmd=[{"prefix": "orch upgrade ls", "image": "quay.io/ceph/ceph", "tags": true, "target": ["mon-mgr", ""]}]: dispatch 2024-04-14T14:30:49.066 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:30:48 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mon-a[83676]: 2024-04-14T14:30:48.787+0000 7f37c9dc1700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-04-14T14:30:49.066 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:30:48 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mon-a[83676]: 2024-04-14T14:30:48.787+0000 7f37c9dc1700 -1 mon.a@0(leader) e4 *** Got Signal Terminated *** 2024-04-14T14:30:49.066 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:30:48 smithi069 podman[130438]: 2024-04-14 14:30:48.818512336 +0000 UTC m=+0.160695487 container died 8bcd6b3d3846c97cfe8e4e3c5925c23bb2c673c01b9102dced6517351af8bc80 (image=quay.ceph.io/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888, name=ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mon-a, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, ceph=True, org.label-schema.build-date=20240403, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_CLEAN=True, RELEASE=main-a971fd5) 2024-04-14T14:30:49.066 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 14 14:30:48 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-y[63038]: [14/Apr/2024:14:30:48] ENGINE Bus STOPPING 2024-04-14T14:30:49.066 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 14 14:30:48 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-y[63038]: [14/Apr/2024:14:30:48] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-04-14T14:30:49.066 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 14 14:30:48 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-y[63038]: [14/Apr/2024:14:30:48] ENGINE Bus STOPPED 2024-04-14T14:30:49.067 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 14 14:30:48 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-y[63038]: [14/Apr/2024:14:30:48] ENGINE Bus STARTING 2024-04-14T14:30:49.067 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 14 14:30:48 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-y[63038]: CherryPy Checker: 2024-04-14T14:30:49.067 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 14 14:30:48 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-y[63038]: The Application mounted at '' has an empty config. 2024-04-14T14:30:49.067 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 14 14:30:48 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-y[63038]: 2024-04-14T14:30:49.067 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 14 14:30:48 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-y[63038]: [14/Apr/2024:14:30:48] ENGINE Serving on http://:::9283 2024-04-14T14:30:49.067 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 14 14:30:48 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-y[63038]: [14/Apr/2024:14:30:48] ENGINE Bus STARTED 2024-04-14T14:30:49.465 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:30:49 smithi069 podman[130438]: 2024-04-14 14:30:49.229811291 +0000 UTC m=+0.571994452 container cleanup 8bcd6b3d3846c97cfe8e4e3c5925c23bb2c673c01b9102dced6517351af8bc80 (image=quay.ceph.io/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888, name=ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mon-a, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, GIT_CLEAN=True, RELEASE=main-a971fd5, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, ceph=True, org.label-schema.build-date=20240403, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-14T14:30:49.465 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:30:49 smithi069 bash[130438]: ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mon-a 2024-04-14T14:30:49.704 DEBUG:teuthology.orchestra.run.smithi069:> sudo pkill -f 'journalctl -f -n 0 -u ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968@mon.a.service' 2024-04-14T14:30:49.728 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:30:49 smithi069 podman[130463]: 2024-04-14 14:30:49.464829673 +0000 UTC m=+0.647900552 container remove 8bcd6b3d3846c97cfe8e4e3c5925c23bb2c673c01b9102dced6517351af8bc80 (image=quay.ceph.io/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888, name=ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mon-a, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=main-a971fd5, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240403, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.schema-version=1.0) 2024-04-14T14:30:49.729 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:30:49 smithi069 systemd[1]: ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968@mon.a.service: Deactivated successfully. 2024-04-14T14:30:49.729 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:30:49 smithi069 systemd[1]: Stopped Ceph mon.a for dc326fc2-fa65-11ee-bc8e-c7b262605968. 2024-04-14T14:30:49.729 INFO:journalctl@ceph.mon.a.smithi069.stdout:Apr 14 14:30:49 smithi069 systemd[1]: ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968@mon.a.service: Consumed 9.794s CPU time. 2024-04-14T14:30:49.750 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-14T14:30:49.750 INFO:tasks.cephadm.mon.a:Stopped mon.a 2024-04-14T14:30:49.750 INFO:tasks.cephadm.mon.b:Stopping mon.c... 2024-04-14T14:30:49.750 DEBUG:teuthology.orchestra.run.smithi069:> sudo systemctl stop ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968@mon.c 2024-04-14T14:30:50.060 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:30:49 smithi069 systemd[1]: Stopping Ceph mon.c for dc326fc2-fa65-11ee-bc8e-c7b262605968... 2024-04-14T14:30:50.316 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:30:50 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mon-c[88632]: 2024-04-14T14:30:50.059+0000 7f948c416700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.c -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-04-14T14:30:50.316 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:30:50 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mon-c[88632]: 2024-04-14T14:30:50.059+0000 7f948c416700 -1 mon.c@1(peon) e4 *** Got Signal Terminated *** 2024-04-14T14:30:50.316 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:30:50 smithi069 podman[130568]: 2024-04-14 14:30:50.233357023 +0000 UTC m=+0.266365190 container died 84aa95f11d0c392c7b8479408817b8e8804c7d09554d5f4083aadc86bb18d9b7 (image=quay.ceph.io/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888, name=ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mon-c, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, ceph=True, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, RELEASE=main-a971fd5, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.build-date=20240403, org.label-schema.vendor=CentOS, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0) 2024-04-14T14:30:50.317 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 14 14:30:50 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-y[63038]: [14/Apr/2024:14:30:50] ENGINE Bus STOPPING 2024-04-14T14:30:50.317 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 14 14:30:50 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-y[63038]: [14/Apr/2024:14:30:50] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-04-14T14:30:50.317 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 14 14:30:50 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-y[63038]: [14/Apr/2024:14:30:50] ENGINE Bus STOPPED 2024-04-14T14:30:50.317 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 14 14:30:50 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-y[63038]: [14/Apr/2024:14:30:50] ENGINE Bus STARTING 2024-04-14T14:30:50.317 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 14 14:30:50 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-y[63038]: [14/Apr/2024:14:30:50] ENGINE Serving on http://:::9283 2024-04-14T14:30:50.317 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 14 14:30:50 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-y[63038]: [14/Apr/2024:14:30:50] ENGINE Bus STARTED 2024-04-14T14:30:50.816 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:30:50 smithi069 podman[130568]: 2024-04-14 14:30:50.601663075 +0000 UTC m=+0.634671242 container cleanup 84aa95f11d0c392c7b8479408817b8e8804c7d09554d5f4083aadc86bb18d9b7 (image=quay.ceph.io/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888, name=ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mon-c, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240403, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, RELEASE=main-a971fd5, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0) 2024-04-14T14:30:50.816 INFO:journalctl@ceph.mon.c.smithi069.stdout:Apr 14 14:30:50 smithi069 bash[130568]: ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mon-c 2024-04-14T14:30:51.069 DEBUG:teuthology.orchestra.run.smithi069:> sudo pkill -f 'journalctl -f -n 0 -u ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968@mon.c.service' 2024-04-14T14:30:51.107 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-14T14:30:51.107 INFO:tasks.cephadm.mon.b:Stopped mon.c 2024-04-14T14:30:51.107 INFO:tasks.cephadm.mon.b:Stopping mon.b... 2024-04-14T14:30:51.107 DEBUG:teuthology.orchestra.run.smithi122:> sudo systemctl stop ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968@mon.b 2024-04-14T14:30:51.460 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:30:51 smithi122 systemd[1]: Stopping Ceph mon.b for dc326fc2-fa65-11ee-bc8e-c7b262605968... 2024-04-14T14:30:51.844 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:30:51 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mon-b[67994]: 2024-04-14T14:30:51.458+0000 7f3624824700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.b -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-04-14T14:30:51.844 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:30:51 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mon-b[67994]: 2024-04-14T14:30:51.458+0000 7f3624824700 -1 mon.b@2(peon) e4 *** Got Signal Terminated *** 2024-04-14T14:30:51.844 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:30:51 smithi122 podman[101039]: 2024-04-14 14:30:51.653581927 +0000 UTC m=+0.351749761 container died 05f437a1a9a1d666fc75595e6dd186557ac67a8fcc56c8a07ac28426fa996267 (image=quay.ceph.io/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888, name=ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mon-b, RELEASE=main-a971fd5, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, io.buildah.version=1.33.5, org.label-schema.build-date=20240403, org.label-schema.license=GPLv2) 2024-04-14T14:30:52.344 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:30:52 smithi122 podman[101039]: 2024-04-14 14:30:52.126264855 +0000 UTC m=+0.824432669 container cleanup 05f437a1a9a1d666fc75595e6dd186557ac67a8fcc56c8a07ac28426fa996267 (image=quay.ceph.io/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888, name=ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mon-b, org.label-schema.build-date=20240403, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , RELEASE=main-a971fd5, ceph=True, org.label-schema.schema-version=1.0, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.vendor=CentOS) 2024-04-14T14:30:52.344 INFO:journalctl@ceph.mon.b.smithi122.stdout:Apr 14 14:30:52 smithi122 bash[101039]: ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mon-b 2024-04-14T14:30:52.497 DEBUG:teuthology.orchestra.run.smithi122:> sudo pkill -f 'journalctl -f -n 0 -u ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968@mon.b.service' 2024-04-14T14:30:52.543 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-14T14:30:52.543 INFO:tasks.cephadm.mon.b:Stopped mon.b 2024-04-14T14:30:52.543 INFO:tasks.cephadm.mgr.y:Stopping mgr.y... 2024-04-14T14:30:52.543 DEBUG:teuthology.orchestra.run.smithi069:> sudo systemctl stop ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968@mgr.y 2024-04-14T14:30:52.840 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 14 14:30:52 smithi069 systemd[1]: Stopping Ceph mgr.y for dc326fc2-fa65-11ee-bc8e-c7b262605968... 2024-04-14T14:30:53.157 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 14 14:30:52 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-y[63038]: 2024-04-14T14:30:52.838+0000 7fdd76ccd700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mgr -n mgr.y -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-14T14:30:53.157 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 14 14:30:52 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-y[63038]: 2024-04-14T14:30:52.838+0000 7fdd76ccd700 -1 mgr handle_mgr_signal *** Got signal Terminated *** 2024-04-14T14:30:53.157 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 14 14:30:52 smithi069 podman[130713]: 2024-04-14 14:30:52.871450494 +0000 UTC m=+0.143321871 container died 7af3d29d0e5fda9c03e65cecd2e5360e8fdf5b95688bc0c3f2871d3c90b65f16 (image=quay.ceph.io/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888, name=ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-y, org.label-schema.build-date=20240403, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, RELEASE=main-a971fd5, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.schema-version=1.0) 2024-04-14T14:30:53.482 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 14 14:30:53 smithi069 podman[130713]: 2024-04-14 14:30:53.231151613 +0000 UTC m=+0.503022988 container cleanup 7af3d29d0e5fda9c03e65cecd2e5360e8fdf5b95688bc0c3f2871d3c90b65f16 (image=quay.ceph.io/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888, name=ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-y, maintainer=Guillaume Abrioux , GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.schema-version=1.0, RELEASE=main-a971fd5, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240403, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, ceph=True, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.vendor=CentOS) 2024-04-14T14:30:53.483 INFO:journalctl@ceph.mgr.y.smithi069.stdout:Apr 14 14:30:53 smithi069 bash[130713]: ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-y 2024-04-14T14:30:53.650 DEBUG:teuthology.orchestra.run.smithi069:> sudo pkill -f 'journalctl -f -n 0 -u ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968@mgr.y.service' 2024-04-14T14:30:53.690 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-14T14:30:53.690 INFO:tasks.cephadm.mgr.y:Stopped mgr.y 2024-04-14T14:30:53.690 INFO:tasks.cephadm.mgr.x:Stopping mgr.x... 2024-04-14T14:30:53.690 DEBUG:teuthology.orchestra.run.smithi122:> sudo systemctl stop ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968@mgr.x 2024-04-14T14:30:54.036 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Apr 14 14:30:53 smithi122 systemd[1]: Stopping Ceph mgr.x for dc326fc2-fa65-11ee-bc8e-c7b262605968... 2024-04-14T14:30:54.344 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Apr 14 14:30:54 smithi122 podman[101161]: 2024-04-14 14:30:54.036615761 +0000 UTC m=+0.155857600 container died e29977590b43ad1ad13ba05c4608ae8f9bc96b9ee58f26504f2954a6a4a038e4 (image=quay.ceph.io/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888, name=ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-x, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.build-date=20240403, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, CEPH_POINT_RELEASE=, ceph=True, RELEASE=main-a971fd5) 2024-04-14T14:30:54.699 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Apr 14 14:30:54 smithi122 podman[101161]: 2024-04-14 14:30:54.369452054 +0000 UTC m=+0.488693943 container cleanup e29977590b43ad1ad13ba05c4608ae8f9bc96b9ee58f26504f2954a6a4a038e4 (image=quay.ceph.io/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888, name=ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-x, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, RELEASE=main-a971fd5, GIT_CLEAN=True, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.build-date=20240403, org.label-schema.schema-version=1.0) 2024-04-14T14:30:54.699 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Apr 14 14:30:54 smithi122 bash[101161]: ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-x 2024-04-14T14:30:54.945 DEBUG:teuthology.orchestra.run.smithi122:> sudo pkill -f 'journalctl -f -n 0 -u ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968@mgr.x.service' 2024-04-14T14:30:54.969 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Apr 14 14:30:54 smithi122 podman[101175]: 2024-04-14 14:30:54.699025164 +0000 UTC m=+0.665955266 container remove e29977590b43ad1ad13ba05c4608ae8f9bc96b9ee58f26504f2954a6a4a038e4 (image=quay.ceph.io/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888, name=ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-mgr-x, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240403, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_BRANCH=HEAD, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, RELEASE=main-a971fd5, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=) 2024-04-14T14:30:54.969 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Apr 14 14:30:54 smithi122 systemd[1]: ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968@mgr.x.service: Main process exited, code=exited, status=143/n/a 2024-04-14T14:30:54.969 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Apr 14 14:30:54 smithi122 systemd[1]: ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968@mgr.x.service: Failed with result 'exit-code'. 2024-04-14T14:30:54.969 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Apr 14 14:30:54 smithi122 systemd[1]: Stopped Ceph mgr.x for dc326fc2-fa65-11ee-bc8e-c7b262605968. 2024-04-14T14:30:54.969 INFO:journalctl@ceph.mgr.x.smithi122.stdout:Apr 14 14:30:54 smithi122 systemd[1]: ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968@mgr.x.service: Consumed 22.978s CPU time. 2024-04-14T14:30:54.981 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-14T14:30:54.981 INFO:tasks.cephadm.mgr.x:Stopped mgr.x 2024-04-14T14:30:54.982 INFO:tasks.cephadm.osd.0:Stopping osd.0... 2024-04-14T14:30:54.982 DEBUG:teuthology.orchestra.run.smithi069:> sudo systemctl stop ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968@osd.0 2024-04-14T14:30:55.286 INFO:journalctl@ceph.osd.0.smithi069.stdout:Apr 14 14:30:55 smithi069 systemd[1]: Stopping Ceph osd.0 for dc326fc2-fa65-11ee-bc8e-c7b262605968... 2024-04-14T14:30:55.566 INFO:journalctl@ceph.osd.0.smithi069.stdout:Apr 14 14:30:55 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-osd-0[95733]: 2024-04-14T14:30:55.285+0000 7feefb60b700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.0 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-14T14:30:55.566 INFO:journalctl@ceph.osd.0.smithi069.stdout:Apr 14 14:30:55 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-osd-0[95733]: 2024-04-14T14:30:55.285+0000 7feefb60b700 -1 osd.0 162 *** Got signal Terminated *** 2024-04-14T14:30:55.566 INFO:journalctl@ceph.osd.0.smithi069.stdout:Apr 14 14:30:55 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-osd-0[95733]: 2024-04-14T14:30:55.285+0000 7feefb60b700 -1 osd.0 162 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-14T14:30:59.844 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Apr 14 14:30:59 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-prometheus-a[64422]: ts=2024-04-14T14:30:59.583Z caller=refresh.go:99 level=error component="discovery manager notify" discovery=http config=config-0 msg="Unable to refresh target groups" err="Get \"http://172.21.15.69:8765/sd/prometheus/sd-config?service=alertmanager\": dial tcp 172.21.15.69:8765: connect: connection refused" 2024-04-14T14:30:59.845 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Apr 14 14:30:59 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-prometheus-a[64422]: ts=2024-04-14T14:30:59.583Z caller=refresh.go:99 level=error component="discovery manager scrape" discovery=http config=node msg="Unable to refresh target groups" err="Get \"http://172.21.15.69:8765/sd/prometheus/sd-config?service=node-exporter\": dial tcp 172.21.15.69:8765: connect: connection refused" 2024-04-14T14:30:59.845 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Apr 14 14:30:59 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-prometheus-a[64422]: ts=2024-04-14T14:30:59.585Z caller=refresh.go:99 level=error component="discovery manager scrape" discovery=http config=ceph-exporter msg="Unable to refresh target groups" err="Get \"http://172.21.15.69:8765/sd/prometheus/sd-config?service=ceph-exporter\": dial tcp 172.21.15.69:8765: connect: connection refused" 2024-04-14T14:30:59.845 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Apr 14 14:30:59 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-prometheus-a[64422]: ts=2024-04-14T14:30:59.587Z caller=refresh.go:99 level=error component="discovery manager scrape" discovery=http config=nvmeof msg="Unable to refresh target groups" err="Get \"http://172.21.15.69:8765/sd/prometheus/sd-config?service=nvmeof\": dial tcp 172.21.15.69:8765: connect: connection refused" 2024-04-14T14:30:59.845 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Apr 14 14:30:59 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-prometheus-a[64422]: ts=2024-04-14T14:30:59.587Z caller=refresh.go:99 level=error component="discovery manager scrape" discovery=http config=ceph msg="Unable to refresh target groups" err="Get \"http://172.21.15.69:8765/sd/prometheus/sd-config?service=mgr-prometheus\": dial tcp 172.21.15.69:8765: connect: connection refused" 2024-04-14T14:31:01.003 INFO:journalctl@ceph.osd.0.smithi069.stdout:Apr 14 14:31:00 smithi069 podman[130835]: 2024-04-14 14:31:00.727660826 +0000 UTC m=+5.554218867 container died a0e57e3fb494794d3a44191703521befbaa302aea1f47acc3e8bf914481c9af9 (image=quay.ceph.io/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888, name=ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-osd-0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.build-date=20240403, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, ceph=True, GIT_CLEAN=True, RELEASE=main-a971fd5, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0) 2024-04-14T14:31:01.316 INFO:journalctl@ceph.osd.0.smithi069.stdout:Apr 14 14:31:01 smithi069 podman[130835]: 2024-04-14 14:31:01.10180915 +0000 UTC m=+5.928367191 container cleanup a0e57e3fb494794d3a44191703521befbaa302aea1f47acc3e8bf914481c9af9 (image=quay.ceph.io/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888, name=ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-osd-0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_BRANCH=HEAD, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240403, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, RELEASE=main-a971fd5, maintainer=Guillaume Abrioux ) 2024-04-14T14:31:01.316 INFO:journalctl@ceph.osd.0.smithi069.stdout:Apr 14 14:31:01 smithi069 bash[130835]: ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-osd-0 2024-04-14T14:31:01.703 INFO:journalctl@ceph.osd.0.smithi069.stdout:Apr 14 14:31:01 smithi069 podman[130852]: 2024-04-14 14:31:01.411553734 +0000 UTC m=+0.684620772 container remove a0e57e3fb494794d3a44191703521befbaa302aea1f47acc3e8bf914481c9af9 (image=quay.ceph.io/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888, name=ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-osd-0, org.label-schema.license=GPLv2, org.label-schema.build-date=20240403, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, RELEASE=main-a971fd5, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0) 2024-04-14T14:31:01.973 INFO:journalctl@ceph.osd.0.smithi069.stdout:Apr 14 14:31:01 smithi069 podman[130944]: 2024-04-14 14:31:01.678913201 +0000 UTC m=+0.020092639 image pull b96c509eef841460b3271f5fdd331860da87b004524f4ed59414aa3680183b53 quay.ceph.io/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888 2024-04-14T14:31:01.974 INFO:journalctl@ceph.osd.0.smithi069.stdout:Apr 14 14:31:01 smithi069 podman[130944]: 2024-04-14 14:31:01.79599201 +0000 UTC m=+0.137166560 container create eb17e248253a020a18eb40b843376585e5cdd063c8efbb7cfa26eb10f1edd5cb (image=quay.ceph.io/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888, name=ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-osd-0-deactivate, RELEASE=main-a971fd5, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.build-date=20240403, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_CLEAN=True) 2024-04-14T14:31:02.276 INFO:journalctl@ceph.osd.0.smithi069.stdout:Apr 14 14:31:01 smithi069 podman[130944]: 2024-04-14 14:31:01.973597091 +0000 UTC m=+0.314771654 container init eb17e248253a020a18eb40b843376585e5cdd063c8efbb7cfa26eb10f1edd5cb (image=quay.ceph.io/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888, name=ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-osd-0-deactivate, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240403, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_BRANCH=HEAD, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, RELEASE=main-a971fd5, org.label-schema.vendor=CentOS, ceph=True) 2024-04-14T14:31:02.276 INFO:journalctl@ceph.osd.0.smithi069.stdout:Apr 14 14:31:01 smithi069 podman[130944]: 2024-04-14 14:31:01.976959639 +0000 UTC m=+0.318134198 container start eb17e248253a020a18eb40b843376585e5cdd063c8efbb7cfa26eb10f1edd5cb (image=quay.ceph.io/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888, name=ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-osd-0-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240403, io.buildah.version=1.33.5, RELEASE=main-a971fd5, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_CLEAN=True, maintainer=Guillaume Abrioux ) 2024-04-14T14:31:02.276 INFO:journalctl@ceph.osd.0.smithi069.stdout:Apr 14 14:31:02 smithi069 podman[130944]: 2024-04-14 14:31:02.057077204 +0000 UTC m=+0.398251763 container attach eb17e248253a020a18eb40b843376585e5cdd063c8efbb7cfa26eb10f1edd5cb (image=quay.ceph.io/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888, name=ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-osd-0-deactivate, org.label-schema.build-date=20240403, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, RELEASE=main-a971fd5, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_CLEAN=True) 2024-04-14T14:31:02.566 INFO:journalctl@ceph.osd.0.smithi069.stdout:Apr 14 14:31:02 smithi069 podman[130944]: 2024-04-14 14:31:02.276085766 +0000 UTC m=+0.617260321 container died eb17e248253a020a18eb40b843376585e5cdd063c8efbb7cfa26eb10f1edd5cb (image=quay.ceph.io/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888, name=ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-osd-0-deactivate, maintainer=Guillaume Abrioux , GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, RELEASE=main-a971fd5, GIT_BRANCH=HEAD, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240403, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.license=GPLv2) 2024-04-14T14:31:03.018 DEBUG:teuthology.orchestra.run.smithi069:> sudo pkill -f 'journalctl -f -n 0 -u ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968@osd.0.service' 2024-04-14T14:31:03.064 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-14T14:31:03.065 INFO:tasks.cephadm.osd.0:Stopped osd.0 2024-04-14T14:31:03.065 INFO:tasks.cephadm.osd.1:Stopping osd.1... 2024-04-14T14:31:03.065 DEBUG:teuthology.orchestra.run.smithi069:> sudo systemctl stop ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968@osd.1 2024-04-14T14:31:03.539 INFO:journalctl@ceph.osd.1.smithi069.stdout:Apr 14 14:31:03 smithi069 systemd[1]: Stopping Ceph osd.1 for dc326fc2-fa65-11ee-bc8e-c7b262605968... 2024-04-14T14:31:03.816 INFO:journalctl@ceph.osd.1.smithi069.stdout:Apr 14 14:31:03 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-osd-1[100957]: 2024-04-14T14:31:03.538+0000 7f450df27700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.1 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-14T14:31:03.816 INFO:journalctl@ceph.osd.1.smithi069.stdout:Apr 14 14:31:03 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-osd-1[100957]: 2024-04-14T14:31:03.538+0000 7f450df27700 -1 osd.1 162 *** Got signal Terminated *** 2024-04-14T14:31:03.816 INFO:journalctl@ceph.osd.1.smithi069.stdout:Apr 14 14:31:03 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-osd-1[100957]: 2024-04-14T14:31:03.538+0000 7f450df27700 -1 osd.1 162 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-14T14:31:09.181 INFO:journalctl@ceph.osd.1.smithi069.stdout:Apr 14 14:31:08 smithi069 podman[131049]: 2024-04-14 14:31:08.920730806 +0000 UTC m=+5.483550803 container died cf73d17919bb431b2654fadb0d2d50ef708ede157cd76d8d28d56fed6c7affdc (image=quay.ceph.io/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888, name=ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-osd-1, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_CLEAN=True, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, RELEASE=main-a971fd5, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240403, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, maintainer=Guillaume Abrioux ) 2024-04-14T14:31:09.556 INFO:journalctl@ceph.osd.1.smithi069.stdout:Apr 14 14:31:09 smithi069 podman[131049]: 2024-04-14 14:31:09.280197793 +0000 UTC m=+5.843017789 container cleanup cf73d17919bb431b2654fadb0d2d50ef708ede157cd76d8d28d56fed6c7affdc (image=quay.ceph.io/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888, name=ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-osd-1, GIT_BRANCH=HEAD, ceph=True, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.license=GPLv2, org.label-schema.build-date=20240403, GIT_CLEAN=True, RELEASE=main-a971fd5, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-14T14:31:09.556 INFO:journalctl@ceph.osd.1.smithi069.stdout:Apr 14 14:31:09 smithi069 bash[131049]: ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-osd-1 2024-04-14T14:31:09.816 INFO:journalctl@ceph.osd.1.smithi069.stdout:Apr 14 14:31:09 smithi069 podman[131067]: 2024-04-14 14:31:09.556289442 +0000 UTC m=+0.633856531 container remove cf73d17919bb431b2654fadb0d2d50ef708ede157cd76d8d28d56fed6c7affdc (image=quay.ceph.io/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888, name=ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-osd-1, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.build-date=20240403, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=main-a971fd5, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_CLEAN=True, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=) 2024-04-14T14:31:10.211 INFO:journalctl@ceph.osd.1.smithi069.stdout:Apr 14 14:31:09 smithi069 podman[131160]: 2024-04-14 14:31:09.789528451 +0000 UTC m=+0.017793227 image pull b96c509eef841460b3271f5fdd331860da87b004524f4ed59414aa3680183b53 quay.ceph.io/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888 2024-04-14T14:31:10.211 INFO:journalctl@ceph.osd.1.smithi069.stdout:Apr 14 14:31:09 smithi069 podman[131160]: 2024-04-14 14:31:09.890602022 +0000 UTC m=+0.118866840 container create cbf3fca60a0bfc66d3ad3ad31130b941ae186404587e438194f9ae6c39748eb4 (image=quay.ceph.io/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888, name=ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-osd-1-deactivate, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, RELEASE=main-a971fd5, org.label-schema.build-date=20240403, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0) 2024-04-14T14:31:10.211 INFO:journalctl@ceph.osd.1.smithi069.stdout:Apr 14 14:31:10 smithi069 podman[131160]: 2024-04-14 14:31:10.118206637 +0000 UTC m=+0.346471403 container init cbf3fca60a0bfc66d3ad3ad31130b941ae186404587e438194f9ae6c39748eb4 (image=quay.ceph.io/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888, name=ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-osd-1-deactivate, org.label-schema.build-date=20240403, GIT_BRANCH=HEAD, GIT_CLEAN=True, RELEASE=main-a971fd5, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, io.buildah.version=1.33.5, ceph=True, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-14T14:31:10.211 INFO:journalctl@ceph.osd.1.smithi069.stdout:Apr 14 14:31:10 smithi069 podman[131160]: 2024-04-14 14:31:10.121610397 +0000 UTC m=+0.349875163 container start cbf3fca60a0bfc66d3ad3ad31130b941ae186404587e438194f9ae6c39748eb4 (image=quay.ceph.io/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888, name=ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-osd-1-deactivate, RELEASE=main-a971fd5, ceph=True, org.label-schema.build-date=20240403, maintainer=Guillaume Abrioux , GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_CLEAN=True) 2024-04-14T14:31:10.211 INFO:journalctl@ceph.osd.1.smithi069.stdout:Apr 14 14:31:10 smithi069 podman[131160]: 2024-04-14 14:31:10.209701135 +0000 UTC m=+0.437965948 container attach cbf3fca60a0bfc66d3ad3ad31130b941ae186404587e438194f9ae6c39748eb4 (image=quay.ceph.io/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888, name=ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-osd-1-deactivate, org.label-schema.build-date=20240403, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, RELEASE=main-a971fd5, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD) 2024-04-14T14:31:10.566 INFO:journalctl@ceph.osd.1.smithi069.stdout:Apr 14 14:31:10 smithi069 podman[131160]: 2024-04-14 14:31:10.410320578 +0000 UTC m=+0.638585349 container died cbf3fca60a0bfc66d3ad3ad31130b941ae186404587e438194f9ae6c39748eb4 (image=quay.ceph.io/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888, name=ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-osd-1-deactivate, org.label-schema.license=GPLv2, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, ceph=True, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.build-date=20240403, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=main-a971fd5, org.label-schema.schema-version=1.0) 2024-04-14T14:31:11.020 DEBUG:teuthology.orchestra.run.smithi069:> sudo pkill -f 'journalctl -f -n 0 -u ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968@osd.1.service' 2024-04-14T14:31:11.062 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-14T14:31:11.063 INFO:tasks.cephadm.osd.1:Stopped osd.1 2024-04-14T14:31:11.063 INFO:tasks.cephadm.osd.2:Stopping osd.2... 2024-04-14T14:31:11.063 DEBUG:teuthology.orchestra.run.smithi069:> sudo systemctl stop ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968@osd.2 2024-04-14T14:31:11.316 INFO:journalctl@ceph.osd.2.smithi069.stdout:Apr 14 14:31:11 smithi069 systemd[1]: Stopping Ceph osd.2 for dc326fc2-fa65-11ee-bc8e-c7b262605968... 2024-04-14T14:31:11.816 INFO:journalctl@ceph.osd.2.smithi069.stdout:Apr 14 14:31:11 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-osd-2[107592]: 2024-04-14T14:31:11.416+0000 7f5bff1b4700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.2 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-14T14:31:11.816 INFO:journalctl@ceph.osd.2.smithi069.stdout:Apr 14 14:31:11 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-osd-2[107592]: 2024-04-14T14:31:11.416+0000 7f5bff1b4700 -1 osd.2 162 *** Got signal Terminated *** 2024-04-14T14:31:11.816 INFO:journalctl@ceph.osd.2.smithi069.stdout:Apr 14 14:31:11 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-osd-2[107592]: 2024-04-14T14:31:11.416+0000 7f5bff1b4700 -1 osd.2 162 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-14T14:31:17.126 INFO:journalctl@ceph.osd.2.smithi069.stdout:Apr 14 14:31:16 smithi069 podman[131265]: 2024-04-14 14:31:16.850449073 +0000 UTC m=+5.563330867 container died c0efba5ef45111ecb47224b1910bd2a0d09803805741e3787a63114789d33889 (image=quay.ceph.io/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888, name=ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-osd-2, maintainer=Guillaume Abrioux , ceph=True, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.build-date=20240403, org.label-schema.vendor=CentOS, RELEASE=main-a971fd5, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-14T14:31:17.459 INFO:journalctl@ceph.osd.2.smithi069.stdout:Apr 14 14:31:17 smithi069 podman[131265]: 2024-04-14 14:31:17.224971399 +0000 UTC m=+5.937853190 container cleanup c0efba5ef45111ecb47224b1910bd2a0d09803805741e3787a63114789d33889 (image=quay.ceph.io/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888, name=ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-osd-2, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, RELEASE=main-a971fd5, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240403, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-14T14:31:17.459 INFO:journalctl@ceph.osd.2.smithi069.stdout:Apr 14 14:31:17 smithi069 bash[131265]: ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-osd-2 2024-04-14T14:31:17.794 INFO:journalctl@ceph.osd.2.smithi069.stdout:Apr 14 14:31:17 smithi069 podman[131282]: 2024-04-14 14:31:17.459346732 +0000 UTC m=+0.611857767 container remove c0efba5ef45111ecb47224b1910bd2a0d09803805741e3787a63114789d33889 (image=quay.ceph.io/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888, name=ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-osd-2, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240403, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.license=GPLv2, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=main-a971fd5) 2024-04-14T14:31:18.066 INFO:journalctl@ceph.osd.2.smithi069.stdout:Apr 14 14:31:17 smithi069 podman[131375]: 2024-04-14 14:31:17.793618228 +0000 UTC m=+0.114682457 container create cdae0f3f77247fa78983a51db0eb70966bcbd05d359c5bfb87e06713dbe28c66 (image=quay.ceph.io/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888, name=ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-osd-2-deactivate, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=main-a971fd5, ceph=True, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240403, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS) 2024-04-14T14:31:18.066 INFO:journalctl@ceph.osd.2.smithi069.stdout:Apr 14 14:31:17 smithi069 podman[131375]: 2024-04-14 14:31:17.696405101 +0000 UTC m=+0.017469302 image pull b96c509eef841460b3271f5fdd331860da87b004524f4ed59414aa3680183b53 quay.ceph.io/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888 2024-04-14T14:31:18.066 INFO:journalctl@ceph.osd.2.smithi069.stdout:Apr 14 14:31:18 smithi069 podman[131375]: 2024-04-14 14:31:18.004743587 +0000 UTC m=+0.325807780 container init cdae0f3f77247fa78983a51db0eb70966bcbd05d359c5bfb87e06713dbe28c66 (image=quay.ceph.io/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888, name=ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-osd-2-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, RELEASE=main-a971fd5, org.label-schema.license=GPLv2, org.label-schema.build-date=20240403, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, ceph=True, maintainer=Guillaume Abrioux ) 2024-04-14T14:31:18.066 INFO:journalctl@ceph.osd.2.smithi069.stdout:Apr 14 14:31:18 smithi069 podman[131375]: 2024-04-14 14:31:18.008416728 +0000 UTC m=+0.329480919 container start cdae0f3f77247fa78983a51db0eb70966bcbd05d359c5bfb87e06713dbe28c66 (image=quay.ceph.io/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888, name=ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-osd-2-deactivate, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.build-date=20240403, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=main-a971fd5, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , ceph=True) 2024-04-14T14:31:18.566 INFO:journalctl@ceph.osd.2.smithi069.stdout:Apr 14 14:31:18 smithi069 podman[131375]: 2024-04-14 14:31:18.121317948 +0000 UTC m=+0.442382135 container attach cdae0f3f77247fa78983a51db0eb70966bcbd05d359c5bfb87e06713dbe28c66 (image=quay.ceph.io/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888, name=ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-osd-2-deactivate, GIT_CLEAN=True, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, ceph=True, RELEASE=main-a971fd5, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240403, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-14T14:31:18.566 INFO:journalctl@ceph.osd.2.smithi069.stdout:Apr 14 14:31:18 smithi069 podman[131375]: 2024-04-14 14:31:18.3174901 +0000 UTC m=+0.638554277 container died cdae0f3f77247fa78983a51db0eb70966bcbd05d359c5bfb87e06713dbe28c66 (image=quay.ceph.io/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888, name=ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-osd-2-deactivate, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.build-date=20240403, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, RELEASE=main-a971fd5, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0) 2024-04-14T14:31:18.830 DEBUG:teuthology.orchestra.run.smithi069:> sudo pkill -f 'journalctl -f -n 0 -u ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968@osd.2.service' 2024-04-14T14:31:18.856 INFO:journalctl@ceph.osd.2.smithi069.stdout:Apr 14 14:31:18 smithi069 podman[131401]: 2024-04-14 14:31:18.808032685 +0000 UTC m=+0.484661954 container remove cdae0f3f77247fa78983a51db0eb70966bcbd05d359c5bfb87e06713dbe28c66 (image=quay.ceph.io/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888, name=ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-osd-2-deactivate, GIT_CLEAN=True, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, GIT_BRANCH=HEAD, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, RELEASE=main-a971fd5, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.build-date=20240403, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-14T14:31:18.856 INFO:journalctl@ceph.osd.2.smithi069.stdout:Apr 14 14:31:18 smithi069 systemd[1]: ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968@osd.2.service: Deactivated successfully. 2024-04-14T14:31:18.856 INFO:journalctl@ceph.osd.2.smithi069.stdout:Apr 14 14:31:18 smithi069 systemd[1]: Stopped Ceph osd.2 for dc326fc2-fa65-11ee-bc8e-c7b262605968. 2024-04-14T14:31:18.856 INFO:journalctl@ceph.osd.2.smithi069.stdout:Apr 14 14:31:18 smithi069 systemd[1]: ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968@osd.2.service: Consumed 5.068s CPU time. 2024-04-14T14:31:18.872 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-14T14:31:18.872 INFO:tasks.cephadm.osd.2:Stopped osd.2 2024-04-14T14:31:18.872 INFO:tasks.cephadm.osd.3:Stopping osd.3... 2024-04-14T14:31:18.873 DEBUG:teuthology.orchestra.run.smithi069:> sudo systemctl stop ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968@osd.3 2024-04-14T14:31:19.198 INFO:journalctl@ceph.osd.3.smithi069.stdout:Apr 14 14:31:18 smithi069 systemd[1]: Stopping Ceph osd.3 for dc326fc2-fa65-11ee-bc8e-c7b262605968... 2024-04-14T14:31:19.566 INFO:journalctl@ceph.osd.3.smithi069.stdout:Apr 14 14:31:19 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-osd-3[114221]: 2024-04-14T14:31:19.193+0000 7f7fd778e700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.3 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-14T14:31:19.566 INFO:journalctl@ceph.osd.3.smithi069.stdout:Apr 14 14:31:19 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-osd-3[114221]: 2024-04-14T14:31:19.193+0000 7f7fd778e700 -1 osd.3 162 *** Got signal Terminated *** 2024-04-14T14:31:19.566 INFO:journalctl@ceph.osd.3.smithi069.stdout:Apr 14 14:31:19 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-osd-3[114221]: 2024-04-14T14:31:19.193+0000 7f7fd778e700 -1 osd.3 162 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-14T14:31:23.594 INFO:journalctl@ceph.osd.4.smithi122.stdout:Apr 14 14:31:23 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-osd-4[78175]: 2024-04-14T14:31:23.258+0000 7f1cb8732700 -1 osd.4 162 heartbeat_check: no reply from 172.21.15.69:6806 osd.0 since back 2024-04-14T14:30:59.296475+0000 front 2024-04-14T14:30:59.296489+0000 (oldest deadline 2024-04-14T14:31:22.796312+0000) 2024-04-14T14:31:24.094 INFO:journalctl@ceph.osd.5.smithi122.stdout:Apr 14 14:31:23 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-osd-5[82734]: 2024-04-14T14:31:23.605+0000 7eff07d49700 -1 osd.5 162 heartbeat_check: no reply from 172.21.15.69:6806 osd.0 since back 2024-04-14T14:30:57.287308+0000 front 2024-04-14T14:30:57.287435+0000 (oldest deadline 2024-04-14T14:31:23.187249+0000) 2024-04-14T14:31:24.094 INFO:journalctl@ceph.osd.6.smithi122.stdout:Apr 14 14:31:23 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-osd-6[87451]: 2024-04-14T14:31:23.748+0000 7ff647f73700 -1 osd.6 162 heartbeat_check: no reply from 172.21.15.69:6806 osd.0 since back 2024-04-14T14:30:57.614597+0000 front 2024-04-14T14:30:57.614586+0000 (oldest deadline 2024-04-14T14:31:23.514362+0000) 2024-04-14T14:31:24.542 INFO:journalctl@ceph.osd.3.smithi069.stdout:Apr 14 14:31:24 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-osd-3[114221]: 2024-04-14T14:31:24.142+0000 7f7fcfd7a700 -1 osd.3 162 heartbeat_check: no reply from 172.21.15.69:6806 osd.0 since back 2024-04-14T14:30:59.117968+0000 front 2024-04-14T14:30:59.117992+0000 (oldest deadline 2024-04-14T14:31:23.817998+0000) 2024-04-14T14:31:24.594 INFO:journalctl@ceph.osd.4.smithi122.stdout:Apr 14 14:31:24 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-osd-4[78175]: 2024-04-14T14:31:24.246+0000 7f1cb8732700 -1 osd.4 162 heartbeat_check: no reply from 172.21.15.69:6806 osd.0 since back 2024-04-14T14:30:59.296475+0000 front 2024-04-14T14:30:59.296489+0000 (oldest deadline 2024-04-14T14:31:22.796312+0000) 2024-04-14T14:31:24.816 INFO:journalctl@ceph.osd.3.smithi069.stdout:Apr 14 14:31:24 smithi069 podman[131480]: 2024-04-14 14:31:24.541666595 +0000 UTC m=+5.453956459 container died d13460da0a65c304df961bf789bfab5641d98dd1203ab79193a183d03a7e0f46 (image=quay.ceph.io/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888, name=ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-osd-3, org.label-schema.build-date=20240403, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, RELEASE=main-a971fd5, ceph=True, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5) 2024-04-14T14:31:25.094 INFO:journalctl@ceph.osd.5.smithi122.stdout:Apr 14 14:31:24 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-osd-5[82734]: 2024-04-14T14:31:24.607+0000 7eff07d49700 -1 osd.5 162 heartbeat_check: no reply from 172.21.15.69:6806 osd.0 since back 2024-04-14T14:30:57.287308+0000 front 2024-04-14T14:30:57.287435+0000 (oldest deadline 2024-04-14T14:31:23.187249+0000) 2024-04-14T14:31:25.094 INFO:journalctl@ceph.osd.6.smithi122.stdout:Apr 14 14:31:24 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-osd-6[87451]: 2024-04-14T14:31:24.788+0000 7ff647f73700 -1 osd.6 162 heartbeat_check: no reply from 172.21.15.69:6806 osd.0 since back 2024-04-14T14:30:57.614597+0000 front 2024-04-14T14:30:57.614586+0000 (oldest deadline 2024-04-14T14:31:23.514362+0000) 2024-04-14T14:31:25.145 INFO:journalctl@ceph.osd.3.smithi069.stdout:Apr 14 14:31:24 smithi069 podman[131480]: 2024-04-14 14:31:24.918530399 +0000 UTC m=+5.830820251 container cleanup d13460da0a65c304df961bf789bfab5641d98dd1203ab79193a183d03a7e0f46 (image=quay.ceph.io/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888, name=ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-osd-3, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.build-date=20240403, org.label-schema.schema-version=1.0, RELEASE=main-a971fd5, GIT_CLEAN=True, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=) 2024-04-14T14:31:25.145 INFO:journalctl@ceph.osd.3.smithi069.stdout:Apr 14 14:31:24 smithi069 bash[131480]: ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-osd-3 2024-04-14T14:31:25.454 INFO:journalctl@ceph.osd.3.smithi069.stdout:Apr 14 14:31:25 smithi069 podman[131497]: 2024-04-14 14:31:25.145003145 +0000 UTC m=+0.601878480 container remove d13460da0a65c304df961bf789bfab5641d98dd1203ab79193a183d03a7e0f46 (image=quay.ceph.io/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888, name=ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-osd-3, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.build-date=20240403, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , RELEASE=main-a971fd5, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, ceph=True, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_CLEAN=True) 2024-04-14T14:31:25.581 INFO:journalctl@ceph.osd.4.smithi122.stdout:Apr 14 14:31:25 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-osd-4[78175]: 2024-04-14T14:31:25.243+0000 7f1cb8732700 -1 osd.4 162 heartbeat_check: no reply from 172.21.15.69:6806 osd.0 since back 2024-04-14T14:30:59.296475+0000 front 2024-04-14T14:30:59.296489+0000 (oldest deadline 2024-04-14T14:31:22.796312+0000) 2024-04-14T14:31:25.729 INFO:journalctl@ceph.osd.3.smithi069.stdout:Apr 14 14:31:25 smithi069 podman[131603]: 2024-04-14 14:31:25.357720796 +0000 UTC m=+0.018070623 image pull b96c509eef841460b3271f5fdd331860da87b004524f4ed59414aa3680183b53 quay.ceph.io/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888 2024-04-14T14:31:25.729 INFO:journalctl@ceph.osd.3.smithi069.stdout:Apr 14 14:31:25 smithi069 podman[131603]: 2024-04-14 14:31:25.46254022 +0000 UTC m=+0.122890037 container create 0db00c614650c84925ba8ae95d799c08a9d470b0bb73b964bb89734ecd4e8346 (image=quay.ceph.io/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888, name=ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-osd-3-deactivate, io.buildah.version=1.33.5, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240403, maintainer=Guillaume Abrioux , RELEASE=main-a971fd5, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.license=GPLv2) 2024-04-14T14:31:25.729 INFO:journalctl@ceph.osd.3.smithi069.stdout:Apr 14 14:31:25 smithi069 podman[131603]: 2024-04-14 14:31:25.664885388 +0000 UTC m=+0.325235205 container init 0db00c614650c84925ba8ae95d799c08a9d470b0bb73b964bb89734ecd4e8346 (image=quay.ceph.io/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888, name=ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-osd-3-deactivate, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, RELEASE=main-a971fd5, org.label-schema.build-date=20240403, io.buildah.version=1.33.5, ceph=True, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2) 2024-04-14T14:31:25.729 INFO:journalctl@ceph.osd.3.smithi069.stdout:Apr 14 14:31:25 smithi069 podman[131603]: 2024-04-14 14:31:25.668012792 +0000 UTC m=+0.328362610 container start 0db00c614650c84925ba8ae95d799c08a9d470b0bb73b964bb89734ecd4e8346 (image=quay.ceph.io/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888, name=ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-osd-3-deactivate, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_CLEAN=True, org.label-schema.build-date=20240403, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , RELEASE=main-a971fd5, org.label-schema.vendor=CentOS) 2024-04-14T14:31:25.844 INFO:journalctl@ceph.osd.5.smithi122.stdout:Apr 14 14:31:25 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-osd-5[82734]: 2024-04-14T14:31:25.608+0000 7eff07d49700 -1 osd.5 162 heartbeat_check: no reply from 172.21.15.69:6806 osd.0 since back 2024-04-14T14:30:57.287308+0000 front 2024-04-14T14:30:57.287435+0000 (oldest deadline 2024-04-14T14:31:23.187249+0000) 2024-04-14T14:31:25.844 INFO:journalctl@ceph.osd.6.smithi122.stdout:Apr 14 14:31:25 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-osd-6[87451]: 2024-04-14T14:31:25.774+0000 7ff647f73700 -1 osd.6 162 heartbeat_check: no reply from 172.21.15.69:6806 osd.0 since back 2024-04-14T14:30:57.614597+0000 front 2024-04-14T14:30:57.614586+0000 (oldest deadline 2024-04-14T14:31:23.514362+0000) 2024-04-14T14:31:26.066 INFO:journalctl@ceph.osd.3.smithi069.stdout:Apr 14 14:31:25 smithi069 podman[131603]: 2024-04-14 14:31:25.759798134 +0000 UTC m=+0.420147961 container attach 0db00c614650c84925ba8ae95d799c08a9d470b0bb73b964bb89734ecd4e8346 (image=quay.ceph.io/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888, name=ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-osd-3-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240403, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, RELEASE=main-a971fd5, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux ) 2024-04-14T14:31:26.066 INFO:journalctl@ceph.osd.3.smithi069.stdout:Apr 14 14:31:25 smithi069 podman[131603]: 2024-04-14 14:31:25.95256865 +0000 UTC m=+0.612918460 container died 0db00c614650c84925ba8ae95d799c08a9d470b0bb73b964bb89734ecd4e8346 (image=quay.ceph.io/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888, name=ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-osd-3-deactivate, org.label-schema.license=GPLv2, RELEASE=main-a971fd5, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.build-date=20240403, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, ceph=True) 2024-04-14T14:31:26.534 DEBUG:teuthology.orchestra.run.smithi069:> sudo pkill -f 'journalctl -f -n 0 -u ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968@osd.3.service' 2024-04-14T14:31:26.560 INFO:journalctl@ceph.osd.3.smithi069.stdout:Apr 14 14:31:26 smithi069 podman[131631]: 2024-04-14 14:31:26.518205767 +0000 UTC m=+0.555719704 container remove 0db00c614650c84925ba8ae95d799c08a9d470b0bb73b964bb89734ecd4e8346 (image=quay.ceph.io/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888, name=ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-osd-3-deactivate, io.buildah.version=1.33.5, GIT_CLEAN=True, RELEASE=main-a971fd5, org.label-schema.schema-version=1.0, ceph=True, maintainer=Guillaume Abrioux , GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.build-date=20240403, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-14T14:31:26.561 INFO:journalctl@ceph.osd.3.smithi069.stdout:Apr 14 14:31:26 smithi069 systemd[1]: ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968@osd.3.service: Deactivated successfully. 2024-04-14T14:31:26.561 INFO:journalctl@ceph.osd.3.smithi069.stdout:Apr 14 14:31:26 smithi069 systemd[1]: Stopped Ceph osd.3 for dc326fc2-fa65-11ee-bc8e-c7b262605968. 2024-04-14T14:31:26.561 INFO:journalctl@ceph.osd.3.smithi069.stdout:Apr 14 14:31:26 smithi069 systemd[1]: ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968@osd.3.service: Consumed 6.042s CPU time. 2024-04-14T14:31:26.576 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-14T14:31:26.577 INFO:tasks.cephadm.osd.3:Stopped osd.3 2024-04-14T14:31:26.577 INFO:tasks.cephadm.osd.4:Stopping osd.4... 2024-04-14T14:31:26.577 DEBUG:teuthology.orchestra.run.smithi122:> sudo systemctl stop ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968@osd.4 2024-04-14T14:31:26.594 INFO:journalctl@ceph.osd.4.smithi122.stdout:Apr 14 14:31:26 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-osd-4[78175]: 2024-04-14T14:31:26.270+0000 7f1cb8732700 -1 osd.4 162 heartbeat_check: no reply from 172.21.15.69:6806 osd.0 since back 2024-04-14T14:30:59.296475+0000 front 2024-04-14T14:30:59.296489+0000 (oldest deadline 2024-04-14T14:31:22.796312+0000) 2024-04-14T14:31:26.594 INFO:journalctl@ceph.osd.7.smithi122.stdout:Apr 14 14:31:26 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-osd-7[92447]: 2024-04-14T14:31:26.132+0000 7f97c2275700 -1 osd.7 162 heartbeat_check: no reply from 172.21.15.69:6806 osd.0 since back 2024-04-14T14:30:59.635081+0000 front 2024-04-14T14:30:59.635113+0000 (oldest deadline 2024-04-14T14:31:25.535039+0000) 2024-04-14T14:31:26.911 INFO:journalctl@ceph.osd.4.smithi122.stdout:Apr 14 14:31:26 smithi122 systemd[1]: Stopping Ceph osd.4 for dc326fc2-fa65-11ee-bc8e-c7b262605968... 2024-04-14T14:31:26.911 INFO:journalctl@ceph.osd.5.smithi122.stdout:Apr 14 14:31:26 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-osd-5[82734]: 2024-04-14T14:31:26.616+0000 7eff07d49700 -1 osd.5 162 heartbeat_check: no reply from 172.21.15.69:6806 osd.0 since back 2024-04-14T14:30:57.287308+0000 front 2024-04-14T14:30:57.287435+0000 (oldest deadline 2024-04-14T14:31:23.187249+0000) 2024-04-14T14:31:26.911 INFO:journalctl@ceph.osd.6.smithi122.stdout:Apr 14 14:31:26 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-osd-6[87451]: 2024-04-14T14:31:26.807+0000 7ff647f73700 -1 osd.6 162 heartbeat_check: no reply from 172.21.15.69:6806 osd.0 since back 2024-04-14T14:30:57.614597+0000 front 2024-04-14T14:30:57.614586+0000 (oldest deadline 2024-04-14T14:31:23.514362+0000) 2024-04-14T14:31:27.239 INFO:journalctl@ceph.osd.7.smithi122.stdout:Apr 14 14:31:27 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-osd-7[92447]: 2024-04-14T14:31:27.112+0000 7f97c2275700 -1 osd.7 162 heartbeat_check: no reply from 172.21.15.69:6806 osd.0 since back 2024-04-14T14:30:59.635081+0000 front 2024-04-14T14:30:59.635113+0000 (oldest deadline 2024-04-14T14:31:25.535039+0000) 2024-04-14T14:31:27.240 INFO:journalctl@ceph.osd.4.smithi122.stdout:Apr 14 14:31:26 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-osd-4[78175]: 2024-04-14T14:31:26.909+0000 7f1cc0146700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.4 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-14T14:31:27.240 INFO:journalctl@ceph.osd.4.smithi122.stdout:Apr 14 14:31:26 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-osd-4[78175]: 2024-04-14T14:31:26.909+0000 7f1cc0146700 -1 osd.4 162 *** Got signal Terminated *** 2024-04-14T14:31:27.240 INFO:journalctl@ceph.osd.4.smithi122.stdout:Apr 14 14:31:26 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-osd-4[78175]: 2024-04-14T14:31:26.909+0000 7f1cc0146700 -1 osd.4 162 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-14T14:31:27.240 INFO:journalctl@ceph.osd.4.smithi122.stdout:Apr 14 14:31:27 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-osd-4[78175]: 2024-04-14T14:31:27.238+0000 7f1cb8732700 -1 osd.4 162 heartbeat_check: no reply from 172.21.15.69:6806 osd.0 since back 2024-04-14T14:30:59.296475+0000 front 2024-04-14T14:30:59.296489+0000 (oldest deadline 2024-04-14T14:31:22.796312+0000) 2024-04-14T14:31:28.094 INFO:journalctl@ceph.osd.5.smithi122.stdout:Apr 14 14:31:27 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-osd-5[82734]: 2024-04-14T14:31:27.617+0000 7eff07d49700 -1 osd.5 162 heartbeat_check: no reply from 172.21.15.69:6806 osd.0 since back 2024-04-14T14:30:57.287308+0000 front 2024-04-14T14:30:57.287435+0000 (oldest deadline 2024-04-14T14:31:23.187249+0000) 2024-04-14T14:31:28.094 INFO:journalctl@ceph.osd.6.smithi122.stdout:Apr 14 14:31:27 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-osd-6[87451]: 2024-04-14T14:31:27.789+0000 7ff647f73700 -1 osd.6 162 heartbeat_check: no reply from 172.21.15.69:6806 osd.0 since back 2024-04-14T14:30:57.614597+0000 front 2024-04-14T14:30:57.614586+0000 (oldest deadline 2024-04-14T14:31:23.514362+0000) 2024-04-14T14:31:28.594 INFO:journalctl@ceph.osd.4.smithi122.stdout:Apr 14 14:31:28 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-osd-4[78175]: 2024-04-14T14:31:28.270+0000 7f1cb8732700 -1 osd.4 162 heartbeat_check: no reply from 172.21.15.69:6806 osd.0 since back 2024-04-14T14:30:59.296475+0000 front 2024-04-14T14:30:59.296489+0000 (oldest deadline 2024-04-14T14:31:22.796312+0000) 2024-04-14T14:31:28.594 INFO:journalctl@ceph.osd.7.smithi122.stdout:Apr 14 14:31:28 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-osd-7[92447]: 2024-04-14T14:31:28.094+0000 7f97c2275700 -1 osd.7 162 heartbeat_check: no reply from 172.21.15.69:6806 osd.0 since back 2024-04-14T14:30:59.635081+0000 front 2024-04-14T14:30:59.635113+0000 (oldest deadline 2024-04-14T14:31:25.535039+0000) 2024-04-14T14:31:29.051 INFO:journalctl@ceph.osd.6.smithi122.stdout:Apr 14 14:31:28 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-osd-6[87451]: 2024-04-14T14:31:28.747+0000 7ff647f73700 -1 osd.6 162 heartbeat_check: no reply from 172.21.15.69:6806 osd.0 since back 2024-04-14T14:30:57.614597+0000 front 2024-04-14T14:30:57.614586+0000 (oldest deadline 2024-04-14T14:31:23.514362+0000) 2024-04-14T14:31:29.051 INFO:journalctl@ceph.osd.5.smithi122.stdout:Apr 14 14:31:28 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-osd-5[82734]: 2024-04-14T14:31:28.617+0000 7eff07d49700 -1 osd.5 162 heartbeat_check: no reply from 172.21.15.69:6806 osd.0 since back 2024-04-14T14:30:57.287308+0000 front 2024-04-14T14:30:57.287435+0000 (oldest deadline 2024-04-14T14:31:23.187249+0000) 2024-04-14T14:31:29.344 INFO:journalctl@ceph.osd.4.smithi122.stdout:Apr 14 14:31:29 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-osd-4[78175]: 2024-04-14T14:31:29.247+0000 7f1cb8732700 -1 osd.4 162 heartbeat_check: no reply from 172.21.15.69:6806 osd.0 since back 2024-04-14T14:30:59.296475+0000 front 2024-04-14T14:30:59.296489+0000 (oldest deadline 2024-04-14T14:31:22.796312+0000) 2024-04-14T14:31:29.344 INFO:journalctl@ceph.osd.7.smithi122.stdout:Apr 14 14:31:29 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-osd-7[92447]: 2024-04-14T14:31:29.050+0000 7f97c2275700 -1 osd.7 162 heartbeat_check: no reply from 172.21.15.69:6806 osd.0 since back 2024-04-14T14:30:59.635081+0000 front 2024-04-14T14:30:59.635113+0000 (oldest deadline 2024-04-14T14:31:25.535039+0000) 2024-04-14T14:31:29.844 INFO:journalctl@ceph.osd.5.smithi122.stdout:Apr 14 14:31:29 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-osd-5[82734]: 2024-04-14T14:31:29.590+0000 7eff07d49700 -1 osd.5 162 heartbeat_check: no reply from 172.21.15.69:6806 osd.0 since back 2024-04-14T14:30:57.287308+0000 front 2024-04-14T14:30:57.287435+0000 (oldest deadline 2024-04-14T14:31:23.187249+0000) 2024-04-14T14:31:29.844 INFO:journalctl@ceph.osd.6.smithi122.stdout:Apr 14 14:31:29 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-osd-6[87451]: 2024-04-14T14:31:29.778+0000 7ff647f73700 -1 osd.6 162 heartbeat_check: no reply from 172.21.15.69:6806 osd.0 since back 2024-04-14T14:30:57.614597+0000 front 2024-04-14T14:30:57.614586+0000 (oldest deadline 2024-04-14T14:31:23.514362+0000) 2024-04-14T14:31:30.291 INFO:journalctl@ceph.osd.7.smithi122.stdout:Apr 14 14:31:30 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-osd-7[92447]: 2024-04-14T14:31:30.001+0000 7f97c2275700 -1 osd.7 162 heartbeat_check: no reply from 172.21.15.69:6806 osd.0 since back 2024-04-14T14:30:59.635081+0000 front 2024-04-14T14:30:59.635113+0000 (oldest deadline 2024-04-14T14:31:25.535039+0000) 2024-04-14T14:31:30.545 INFO:journalctl@ceph.osd.4.smithi122.stdout:Apr 14 14:31:30 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-osd-4[78175]: 2024-04-14T14:31:30.291+0000 7f1cb8732700 -1 osd.4 162 heartbeat_check: no reply from 172.21.15.69:6806 osd.0 since back 2024-04-14T14:30:59.296475+0000 front 2024-04-14T14:30:59.296489+0000 (oldest deadline 2024-04-14T14:31:22.796312+0000) 2024-04-14T14:31:30.819 INFO:journalctl@ceph.osd.5.smithi122.stdout:Apr 14 14:31:30 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-osd-5[82734]: 2024-04-14T14:31:30.544+0000 7eff07d49700 -1 osd.5 162 heartbeat_check: no reply from 172.21.15.69:6806 osd.0 since back 2024-04-14T14:30:57.287308+0000 front 2024-04-14T14:30:57.287435+0000 (oldest deadline 2024-04-14T14:31:23.187249+0000) 2024-04-14T14:31:31.094 INFO:journalctl@ceph.osd.6.smithi122.stdout:Apr 14 14:31:30 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-osd-6[87451]: 2024-04-14T14:31:30.817+0000 7ff647f73700 -1 osd.6 162 heartbeat_check: no reply from 172.21.15.69:6806 osd.0 since back 2024-04-14T14:30:57.614597+0000 front 2024-04-14T14:30:57.614586+0000 (oldest deadline 2024-04-14T14:31:23.514362+0000) 2024-04-14T14:31:31.094 INFO:journalctl@ceph.osd.6.smithi122.stdout:Apr 14 14:31:30 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-osd-6[87451]: 2024-04-14T14:31:30.817+0000 7ff647f73700 -1 osd.6 162 heartbeat_check: no reply from 172.21.15.69:6814 osd.1 since back 2024-04-14T14:31:05.815477+0000 front 2024-04-14T14:31:05.815418+0000 (oldest deadline 2024-04-14T14:31:30.515245+0000) 2024-04-14T14:31:31.094 INFO:journalctl@ceph.osd.7.smithi122.stdout:Apr 14 14:31:31 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-osd-7[92447]: 2024-04-14T14:31:31.019+0000 7f97c2275700 -1 osd.7 162 heartbeat_check: no reply from 172.21.15.69:6806 osd.0 since back 2024-04-14T14:30:59.635081+0000 front 2024-04-14T14:30:59.635113+0000 (oldest deadline 2024-04-14T14:31:25.535039+0000) 2024-04-14T14:31:31.094 INFO:journalctl@ceph.osd.7.smithi122.stdout:Apr 14 14:31:31 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-osd-7[92447]: 2024-04-14T14:31:31.019+0000 7f97c2275700 -1 osd.7 162 heartbeat_check: no reply from 172.21.15.69:6814 osd.1 since back 2024-04-14T14:31:06.635906+0000 front 2024-04-14T14:31:06.635997+0000 (oldest deadline 2024-04-14T14:31:30.135764+0000) 2024-04-14T14:31:31.594 INFO:journalctl@ceph.osd.4.smithi122.stdout:Apr 14 14:31:31 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-osd-4[78175]: 2024-04-14T14:31:31.304+0000 7f1cb8732700 -1 osd.4 162 heartbeat_check: no reply from 172.21.15.69:6806 osd.0 since back 2024-04-14T14:30:59.296475+0000 front 2024-04-14T14:30:59.296489+0000 (oldest deadline 2024-04-14T14:31:22.796312+0000) 2024-04-14T14:31:31.594 INFO:journalctl@ceph.osd.4.smithi122.stdout:Apr 14 14:31:31 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-osd-4[78175]: 2024-04-14T14:31:31.304+0000 7f1cb8732700 -1 osd.4 162 heartbeat_check: no reply from 172.21.15.69:6814 osd.1 since back 2024-04-14T14:31:07.699367+0000 front 2024-04-14T14:31:07.699339+0000 (oldest deadline 2024-04-14T14:31:31.199219+0000) 2024-04-14T14:31:31.594 INFO:journalctl@ceph.osd.5.smithi122.stdout:Apr 14 14:31:31 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-osd-5[82734]: 2024-04-14T14:31:31.498+0000 7eff07d49700 -1 osd.5 162 heartbeat_check: no reply from 172.21.15.69:6806 osd.0 since back 2024-04-14T14:30:57.287308+0000 front 2024-04-14T14:30:57.287435+0000 (oldest deadline 2024-04-14T14:31:23.187249+0000) 2024-04-14T14:31:31.594 INFO:journalctl@ceph.osd.5.smithi122.stdout:Apr 14 14:31:31 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-osd-5[82734]: 2024-04-14T14:31:31.498+0000 7eff07d49700 -1 osd.5 162 heartbeat_check: no reply from 172.21.15.69:6814 osd.1 since back 2024-04-14T14:31:07.888129+0000 front 2024-04-14T14:31:07.888233+0000 (oldest deadline 2024-04-14T14:31:30.788034+0000) 2024-04-14T14:31:32.094 INFO:journalctl@ceph.osd.6.smithi122.stdout:Apr 14 14:31:31 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-osd-6[87451]: 2024-04-14T14:31:31.807+0000 7ff647f73700 -1 osd.6 162 heartbeat_check: no reply from 172.21.15.69:6806 osd.0 since back 2024-04-14T14:30:57.614597+0000 front 2024-04-14T14:30:57.614586+0000 (oldest deadline 2024-04-14T14:31:23.514362+0000) 2024-04-14T14:31:32.094 INFO:journalctl@ceph.osd.6.smithi122.stdout:Apr 14 14:31:31 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-osd-6[87451]: 2024-04-14T14:31:31.807+0000 7ff647f73700 -1 osd.6 162 heartbeat_check: no reply from 172.21.15.69:6814 osd.1 since back 2024-04-14T14:31:05.815477+0000 front 2024-04-14T14:31:05.815418+0000 (oldest deadline 2024-04-14T14:31:30.515245+0000) 2024-04-14T14:31:32.094 INFO:journalctl@ceph.osd.7.smithi122.stdout:Apr 14 14:31:32 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-osd-7[92447]: 2024-04-14T14:31:32.004+0000 7f97c2275700 -1 osd.7 162 heartbeat_check: no reply from 172.21.15.69:6806 osd.0 since back 2024-04-14T14:30:59.635081+0000 front 2024-04-14T14:30:59.635113+0000 (oldest deadline 2024-04-14T14:31:25.535039+0000) 2024-04-14T14:31:32.095 INFO:journalctl@ceph.osd.7.smithi122.stdout:Apr 14 14:31:32 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-osd-7[92447]: 2024-04-14T14:31:32.004+0000 7f97c2275700 -1 osd.7 162 heartbeat_check: no reply from 172.21.15.69:6814 osd.1 since back 2024-04-14T14:31:06.635906+0000 front 2024-04-14T14:31:06.635997+0000 (oldest deadline 2024-04-14T14:31:30.135764+0000) 2024-04-14T14:31:32.694 INFO:journalctl@ceph.osd.4.smithi122.stdout:Apr 14 14:31:32 smithi122 podman[101297]: 2024-04-14 14:31:32.422272867 +0000 UTC m=+5.645546745 container died 7a87a024f984e181c2e3243a4f3d5f6097efb32288914d71fbafb502229e5d07 (image=quay.ceph.io/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888, name=ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-osd-4, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_CLEAN=True, RELEASE=main-a971fd5, io.buildah.version=1.33.5, org.label-schema.build-date=20240403, maintainer=Guillaume Abrioux , GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS) 2024-04-14T14:31:32.695 INFO:journalctl@ceph.osd.5.smithi122.stdout:Apr 14 14:31:32 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-osd-5[82734]: 2024-04-14T14:31:32.546+0000 7eff07d49700 -1 osd.5 162 heartbeat_check: no reply from 172.21.15.69:6806 osd.0 since back 2024-04-14T14:30:57.287308+0000 front 2024-04-14T14:30:57.287435+0000 (oldest deadline 2024-04-14T14:31:23.187249+0000) 2024-04-14T14:31:32.695 INFO:journalctl@ceph.osd.5.smithi122.stdout:Apr 14 14:31:32 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-osd-5[82734]: 2024-04-14T14:31:32.546+0000 7eff07d49700 -1 osd.5 162 heartbeat_check: no reply from 172.21.15.69:6814 osd.1 since back 2024-04-14T14:31:07.888129+0000 front 2024-04-14T14:31:07.888233+0000 (oldest deadline 2024-04-14T14:31:30.788034+0000) 2024-04-14T14:31:32.989 INFO:journalctl@ceph.osd.4.smithi122.stdout:Apr 14 14:31:32 smithi122 podman[101297]: 2024-04-14 14:31:32.910261206 +0000 UTC m=+6.133535094 container cleanup 7a87a024f984e181c2e3243a4f3d5f6097efb32288914d71fbafb502229e5d07 (image=quay.ceph.io/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888, name=ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-osd-4, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, ceph=True, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, RELEASE=main-a971fd5, org.label-schema.build-date=20240403, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-14T14:31:32.989 INFO:journalctl@ceph.osd.4.smithi122.stdout:Apr 14 14:31:32 smithi122 bash[101297]: ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-osd-4 2024-04-14T14:31:32.989 INFO:journalctl@ceph.osd.6.smithi122.stdout:Apr 14 14:31:32 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-osd-6[87451]: 2024-04-14T14:31:32.853+0000 7ff647f73700 -1 osd.6 162 heartbeat_check: no reply from 172.21.15.69:6806 osd.0 since back 2024-04-14T14:30:57.614597+0000 front 2024-04-14T14:30:57.614586+0000 (oldest deadline 2024-04-14T14:31:23.514362+0000) 2024-04-14T14:31:32.989 INFO:journalctl@ceph.osd.6.smithi122.stdout:Apr 14 14:31:32 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-osd-6[87451]: 2024-04-14T14:31:32.853+0000 7ff647f73700 -1 osd.6 162 heartbeat_check: no reply from 172.21.15.69:6814 osd.1 since back 2024-04-14T14:31:05.815477+0000 front 2024-04-14T14:31:05.815418+0000 (oldest deadline 2024-04-14T14:31:30.515245+0000) 2024-04-14T14:31:33.344 INFO:journalctl@ceph.osd.4.smithi122.stdout:Apr 14 14:31:33 smithi122 podman[101314]: 2024-04-14 14:31:33.170411217 +0000 UTC m=+0.738865737 container remove 7a87a024f984e181c2e3243a4f3d5f6097efb32288914d71fbafb502229e5d07 (image=quay.ceph.io/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888, name=ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-osd-4, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, RELEASE=main-a971fd5, GIT_CLEAN=True, org.label-schema.build-date=20240403, org.label-schema.license=GPLv2, ceph=True, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0) 2024-04-14T14:31:33.344 INFO:journalctl@ceph.osd.7.smithi122.stdout:Apr 14 14:31:32 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-osd-7[92447]: 2024-04-14T14:31:32.988+0000 7f97c2275700 -1 osd.7 162 heartbeat_check: no reply from 172.21.15.69:6806 osd.0 since back 2024-04-14T14:30:59.635081+0000 front 2024-04-14T14:30:59.635113+0000 (oldest deadline 2024-04-14T14:31:25.535039+0000) 2024-04-14T14:31:33.344 INFO:journalctl@ceph.osd.7.smithi122.stdout:Apr 14 14:31:32 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-osd-7[92447]: 2024-04-14T14:31:32.988+0000 7f97c2275700 -1 osd.7 162 heartbeat_check: no reply from 172.21.15.69:6814 osd.1 since back 2024-04-14T14:31:06.635906+0000 front 2024-04-14T14:31:06.635997+0000 (oldest deadline 2024-04-14T14:31:30.135764+0000) 2024-04-14T14:31:33.765 INFO:journalctl@ceph.osd.5.smithi122.stdout:Apr 14 14:31:33 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-osd-5[82734]: 2024-04-14T14:31:33.535+0000 7eff07d49700 -1 osd.5 162 heartbeat_check: no reply from 172.21.15.69:6806 osd.0 since back 2024-04-14T14:30:57.287308+0000 front 2024-04-14T14:30:57.287435+0000 (oldest deadline 2024-04-14T14:31:23.187249+0000) 2024-04-14T14:31:33.765 INFO:journalctl@ceph.osd.5.smithi122.stdout:Apr 14 14:31:33 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-osd-5[82734]: 2024-04-14T14:31:33.535+0000 7eff07d49700 -1 osd.5 162 heartbeat_check: no reply from 172.21.15.69:6814 osd.1 since back 2024-04-14T14:31:07.888129+0000 front 2024-04-14T14:31:07.888233+0000 (oldest deadline 2024-04-14T14:31:30.788034+0000) 2024-04-14T14:31:33.766 INFO:journalctl@ceph.osd.4.smithi122.stdout:Apr 14 14:31:33 smithi122 podman[101408]: 2024-04-14 14:31:33.379105132 +0000 UTC m=+0.017060510 image pull b96c509eef841460b3271f5fdd331860da87b004524f4ed59414aa3680183b53 quay.ceph.io/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888 2024-04-14T14:31:33.766 INFO:journalctl@ceph.osd.4.smithi122.stdout:Apr 14 14:31:33 smithi122 podman[101408]: 2024-04-14 14:31:33.479654674 +0000 UTC m=+0.117610079 container create 91a0027500e01794a91de228f5be23fced366d1fa7675c19700adc8ebb437bcd (image=quay.ceph.io/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888, name=ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-osd-4-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, org.label-schema.build-date=20240403, RELEASE=main-a971fd5, org.label-schema.schema-version=1.0, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, GIT_CLEAN=True) 2024-04-14T14:31:33.766 INFO:journalctl@ceph.osd.4.smithi122.stdout:Apr 14 14:31:33 smithi122 podman[101408]: 2024-04-14 14:31:33.699185154 +0000 UTC m=+0.337140524 container init 91a0027500e01794a91de228f5be23fced366d1fa7675c19700adc8ebb437bcd (image=quay.ceph.io/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888, name=ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-osd-4-deactivate, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240403, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, ceph=True, maintainer=Guillaume Abrioux , GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, RELEASE=main-a971fd5, org.label-schema.license=GPLv2) 2024-04-14T14:31:33.766 INFO:journalctl@ceph.osd.4.smithi122.stdout:Apr 14 14:31:33 smithi122 podman[101408]: 2024-04-14 14:31:33.70257959 +0000 UTC m=+0.340534956 container start 91a0027500e01794a91de228f5be23fced366d1fa7675c19700adc8ebb437bcd (image=quay.ceph.io/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888, name=ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-osd-4-deactivate, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, RELEASE=main-a971fd5, io.buildah.version=1.33.5, org.label-schema.build-date=20240403, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_CLEAN=True) 2024-04-14T14:31:33.766 INFO:journalctl@ceph.osd.4.smithi122.stdout:Apr 14 14:31:33 smithi122 podman[101408]: 2024-04-14 14:31:33.765422328 +0000 UTC m=+0.403377697 container attach 91a0027500e01794a91de228f5be23fced366d1fa7675c19700adc8ebb437bcd (image=quay.ceph.io/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888, name=ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-osd-4-deactivate, org.label-schema.build-date=20240403, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, io.buildah.version=1.33.5, GIT_CLEAN=True, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, RELEASE=main-a971fd5, org.label-schema.license=GPLv2) 2024-04-14T14:31:34.024 INFO:journalctl@ceph.osd.4.smithi122.stdout:Apr 14 14:31:34 smithi122 podman[101408]: 2024-04-14 14:31:34.005288663 +0000 UTC m=+0.643244027 container died 91a0027500e01794a91de228f5be23fced366d1fa7675c19700adc8ebb437bcd (image=quay.ceph.io/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888, name=ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-osd-4-deactivate, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240403, maintainer=Guillaume Abrioux , GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=main-a971fd5, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=) 2024-04-14T14:31:34.024 INFO:journalctl@ceph.osd.6.smithi122.stdout:Apr 14 14:31:33 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-osd-6[87451]: 2024-04-14T14:31:33.825+0000 7ff647f73700 -1 osd.6 162 heartbeat_check: no reply from 172.21.15.69:6806 osd.0 since back 2024-04-14T14:30:57.614597+0000 front 2024-04-14T14:30:57.614586+0000 (oldest deadline 2024-04-14T14:31:23.514362+0000) 2024-04-14T14:31:34.024 INFO:journalctl@ceph.osd.6.smithi122.stdout:Apr 14 14:31:33 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-osd-6[87451]: 2024-04-14T14:31:33.825+0000 7ff647f73700 -1 osd.6 162 heartbeat_check: no reply from 172.21.15.69:6814 osd.1 since back 2024-04-14T14:31:05.815477+0000 front 2024-04-14T14:31:05.815418+0000 (oldest deadline 2024-04-14T14:31:30.515245+0000) 2024-04-14T14:31:34.024 INFO:journalctl@ceph.osd.7.smithi122.stdout:Apr 14 14:31:33 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-osd-7[92447]: 2024-04-14T14:31:33.938+0000 7f97c2275700 -1 osd.7 162 heartbeat_check: no reply from 172.21.15.69:6806 osd.0 since back 2024-04-14T14:30:59.635081+0000 front 2024-04-14T14:30:59.635113+0000 (oldest deadline 2024-04-14T14:31:25.535039+0000) 2024-04-14T14:31:34.025 INFO:journalctl@ceph.osd.7.smithi122.stdout:Apr 14 14:31:33 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-osd-7[92447]: 2024-04-14T14:31:33.938+0000 7f97c2275700 -1 osd.7 162 heartbeat_check: no reply from 172.21.15.69:6814 osd.1 since back 2024-04-14T14:31:06.635906+0000 front 2024-04-14T14:31:06.635997+0000 (oldest deadline 2024-04-14T14:31:30.135764+0000) 2024-04-14T14:31:34.594 INFO:journalctl@ceph.osd.5.smithi122.stdout:Apr 14 14:31:34 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-osd-5[82734]: 2024-04-14T14:31:34.556+0000 7eff07d49700 -1 osd.5 162 heartbeat_check: no reply from 172.21.15.69:6806 osd.0 since back 2024-04-14T14:30:57.287308+0000 front 2024-04-14T14:30:57.287435+0000 (oldest deadline 2024-04-14T14:31:23.187249+0000) 2024-04-14T14:31:34.594 INFO:journalctl@ceph.osd.5.smithi122.stdout:Apr 14 14:31:34 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-osd-5[82734]: 2024-04-14T14:31:34.556+0000 7eff07d49700 -1 osd.5 162 heartbeat_check: no reply from 172.21.15.69:6814 osd.1 since back 2024-04-14T14:31:07.888129+0000 front 2024-04-14T14:31:07.888233+0000 (oldest deadline 2024-04-14T14:31:30.788034+0000) 2024-04-14T14:31:34.705 DEBUG:teuthology.orchestra.run.smithi122:> sudo pkill -f 'journalctl -f -n 0 -u ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968@osd.4.service' 2024-04-14T14:31:34.788 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-14T14:31:34.788 INFO:tasks.cephadm.osd.4:Stopped osd.4 2024-04-14T14:31:34.788 INFO:tasks.cephadm.osd.5:Stopping osd.5... 2024-04-14T14:31:34.788 DEBUG:teuthology.orchestra.run.smithi122:> sudo systemctl stop ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968@osd.5 2024-04-14T14:31:35.094 INFO:journalctl@ceph.osd.5.smithi122.stdout:Apr 14 14:31:34 smithi122 systemd[1]: Stopping Ceph osd.5 for dc326fc2-fa65-11ee-bc8e-c7b262605968... 2024-04-14T14:31:35.094 INFO:journalctl@ceph.osd.6.smithi122.stdout:Apr 14 14:31:34 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-osd-6[87451]: 2024-04-14T14:31:34.855+0000 7ff647f73700 -1 osd.6 162 heartbeat_check: no reply from 172.21.15.69:6806 osd.0 since back 2024-04-14T14:30:57.614597+0000 front 2024-04-14T14:30:57.614586+0000 (oldest deadline 2024-04-14T14:31:23.514362+0000) 2024-04-14T14:31:35.094 INFO:journalctl@ceph.osd.6.smithi122.stdout:Apr 14 14:31:34 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-osd-6[87451]: 2024-04-14T14:31:34.855+0000 7ff647f73700 -1 osd.6 162 heartbeat_check: no reply from 172.21.15.69:6814 osd.1 since back 2024-04-14T14:31:05.815477+0000 front 2024-04-14T14:31:05.815418+0000 (oldest deadline 2024-04-14T14:31:30.515245+0000) 2024-04-14T14:31:35.095 INFO:journalctl@ceph.osd.7.smithi122.stdout:Apr 14 14:31:34 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-osd-7[92447]: 2024-04-14T14:31:34.905+0000 7f97c2275700 -1 osd.7 162 heartbeat_check: no reply from 172.21.15.69:6806 osd.0 since back 2024-04-14T14:30:59.635081+0000 front 2024-04-14T14:30:59.635113+0000 (oldest deadline 2024-04-14T14:31:25.535039+0000) 2024-04-14T14:31:35.095 INFO:journalctl@ceph.osd.7.smithi122.stdout:Apr 14 14:31:34 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-osd-7[92447]: 2024-04-14T14:31:34.905+0000 7f97c2275700 -1 osd.7 162 heartbeat_check: no reply from 172.21.15.69:6814 osd.1 since back 2024-04-14T14:31:06.635906+0000 front 2024-04-14T14:31:06.635997+0000 (oldest deadline 2024-04-14T14:31:30.135764+0000) 2024-04-14T14:31:35.550 INFO:journalctl@ceph.osd.5.smithi122.stdout:Apr 14 14:31:35 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-osd-5[82734]: 2024-04-14T14:31:35.120+0000 7eff0f75d700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.5 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-14T14:31:35.550 INFO:journalctl@ceph.osd.5.smithi122.stdout:Apr 14 14:31:35 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-osd-5[82734]: 2024-04-14T14:31:35.120+0000 7eff0f75d700 -1 osd.5 162 *** Got signal Terminated *** 2024-04-14T14:31:35.550 INFO:journalctl@ceph.osd.5.smithi122.stdout:Apr 14 14:31:35 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-osd-5[82734]: 2024-04-14T14:31:35.120+0000 7eff0f75d700 -1 osd.5 162 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-14T14:31:35.550 INFO:journalctl@ceph.osd.5.smithi122.stdout:Apr 14 14:31:35 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-osd-5[82734]: 2024-04-14T14:31:35.548+0000 7eff07d49700 -1 osd.5 162 heartbeat_check: no reply from 172.21.15.69:6806 osd.0 since back 2024-04-14T14:30:57.287308+0000 front 2024-04-14T14:30:57.287435+0000 (oldest deadline 2024-04-14T14:31:23.187249+0000) 2024-04-14T14:31:35.550 INFO:journalctl@ceph.osd.5.smithi122.stdout:Apr 14 14:31:35 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-osd-5[82734]: 2024-04-14T14:31:35.548+0000 7eff07d49700 -1 osd.5 162 heartbeat_check: no reply from 172.21.15.69:6814 osd.1 since back 2024-04-14T14:31:07.888129+0000 front 2024-04-14T14:31:07.888233+0000 (oldest deadline 2024-04-14T14:31:30.788034+0000) 2024-04-14T14:31:36.344 INFO:journalctl@ceph.osd.6.smithi122.stdout:Apr 14 14:31:35 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-osd-6[87451]: 2024-04-14T14:31:35.865+0000 7ff647f73700 -1 osd.6 162 heartbeat_check: no reply from 172.21.15.69:6806 osd.0 since back 2024-04-14T14:30:57.614597+0000 front 2024-04-14T14:30:57.614586+0000 (oldest deadline 2024-04-14T14:31:23.514362+0000) 2024-04-14T14:31:36.345 INFO:journalctl@ceph.osd.6.smithi122.stdout:Apr 14 14:31:35 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-osd-6[87451]: 2024-04-14T14:31:35.865+0000 7ff647f73700 -1 osd.6 162 heartbeat_check: no reply from 172.21.15.69:6814 osd.1 since back 2024-04-14T14:31:05.815477+0000 front 2024-04-14T14:31:05.815418+0000 (oldest deadline 2024-04-14T14:31:30.515245+0000) 2024-04-14T14:31:36.345 INFO:journalctl@ceph.osd.7.smithi122.stdout:Apr 14 14:31:35 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-osd-7[92447]: 2024-04-14T14:31:35.890+0000 7f97c2275700 -1 osd.7 162 heartbeat_check: no reply from 172.21.15.69:6806 osd.0 since back 2024-04-14T14:30:59.635081+0000 front 2024-04-14T14:30:59.635113+0000 (oldest deadline 2024-04-14T14:31:25.535039+0000) 2024-04-14T14:31:36.345 INFO:journalctl@ceph.osd.7.smithi122.stdout:Apr 14 14:31:35 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-osd-7[92447]: 2024-04-14T14:31:35.890+0000 7f97c2275700 -1 osd.7 162 heartbeat_check: no reply from 172.21.15.69:6814 osd.1 since back 2024-04-14T14:31:06.635906+0000 front 2024-04-14T14:31:06.635997+0000 (oldest deadline 2024-04-14T14:31:30.135764+0000) 2024-04-14T14:31:36.844 INFO:journalctl@ceph.osd.5.smithi122.stdout:Apr 14 14:31:36 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-osd-5[82734]: 2024-04-14T14:31:36.540+0000 7eff07d49700 -1 osd.5 162 heartbeat_check: no reply from 172.21.15.69:6806 osd.0 since back 2024-04-14T14:30:57.287308+0000 front 2024-04-14T14:30:57.287435+0000 (oldest deadline 2024-04-14T14:31:23.187249+0000) 2024-04-14T14:31:36.844 INFO:journalctl@ceph.osd.5.smithi122.stdout:Apr 14 14:31:36 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-osd-5[82734]: 2024-04-14T14:31:36.540+0000 7eff07d49700 -1 osd.5 162 heartbeat_check: no reply from 172.21.15.69:6814 osd.1 since back 2024-04-14T14:31:07.888129+0000 front 2024-04-14T14:31:07.888233+0000 (oldest deadline 2024-04-14T14:31:30.788034+0000) 2024-04-14T14:31:37.344 INFO:journalctl@ceph.osd.6.smithi122.stdout:Apr 14 14:31:36 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-osd-6[87451]: 2024-04-14T14:31:36.850+0000 7ff647f73700 -1 osd.6 162 heartbeat_check: no reply from 172.21.15.69:6806 osd.0 since back 2024-04-14T14:30:57.614597+0000 front 2024-04-14T14:30:57.614586+0000 (oldest deadline 2024-04-14T14:31:23.514362+0000) 2024-04-14T14:31:37.344 INFO:journalctl@ceph.osd.6.smithi122.stdout:Apr 14 14:31:36 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-osd-6[87451]: 2024-04-14T14:31:36.850+0000 7ff647f73700 -1 osd.6 162 heartbeat_check: no reply from 172.21.15.69:6814 osd.1 since back 2024-04-14T14:31:05.815477+0000 front 2024-04-14T14:31:05.815418+0000 (oldest deadline 2024-04-14T14:31:30.515245+0000) 2024-04-14T14:31:37.344 INFO:journalctl@ceph.osd.7.smithi122.stdout:Apr 14 14:31:36 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-osd-7[92447]: 2024-04-14T14:31:36.892+0000 7f97c2275700 -1 osd.7 162 heartbeat_check: no reply from 172.21.15.69:6806 osd.0 since back 2024-04-14T14:30:59.635081+0000 front 2024-04-14T14:30:59.635113+0000 (oldest deadline 2024-04-14T14:31:25.535039+0000) 2024-04-14T14:31:37.344 INFO:journalctl@ceph.osd.7.smithi122.stdout:Apr 14 14:31:36 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-osd-7[92447]: 2024-04-14T14:31:36.892+0000 7f97c2275700 -1 osd.7 162 heartbeat_check: no reply from 172.21.15.69:6814 osd.1 since back 2024-04-14T14:31:06.635906+0000 front 2024-04-14T14:31:06.635997+0000 (oldest deadline 2024-04-14T14:31:30.135764+0000) 2024-04-14T14:31:37.836 INFO:journalctl@ceph.osd.5.smithi122.stdout:Apr 14 14:31:37 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-osd-5[82734]: 2024-04-14T14:31:37.580+0000 7eff07d49700 -1 osd.5 162 heartbeat_check: no reply from 172.21.15.69:6806 osd.0 since back 2024-04-14T14:30:57.287308+0000 front 2024-04-14T14:30:57.287435+0000 (oldest deadline 2024-04-14T14:31:23.187249+0000) 2024-04-14T14:31:37.837 INFO:journalctl@ceph.osd.5.smithi122.stdout:Apr 14 14:31:37 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-osd-5[82734]: 2024-04-14T14:31:37.580+0000 7eff07d49700 -1 osd.5 162 heartbeat_check: no reply from 172.21.15.69:6814 osd.1 since back 2024-04-14T14:31:07.888129+0000 front 2024-04-14T14:31:07.888233+0000 (oldest deadline 2024-04-14T14:31:30.788034+0000) 2024-04-14T14:31:37.837 INFO:journalctl@ceph.osd.5.smithi122.stdout:Apr 14 14:31:37 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-osd-5[82734]: 2024-04-14T14:31:37.580+0000 7eff07d49700 -1 osd.5 162 heartbeat_check: no reply from 172.21.15.69:6822 osd.2 since back 2024-04-14T14:31:11.889113+0000 front 2024-04-14T14:31:11.888947+0000 (oldest deadline 2024-04-14T14:31:37.189036+0000) 2024-04-14T14:31:38.094 INFO:journalctl@ceph.osd.6.smithi122.stdout:Apr 14 14:31:37 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-osd-6[87451]: 2024-04-14T14:31:37.836+0000 7ff647f73700 -1 osd.6 162 heartbeat_check: no reply from 172.21.15.69:6806 osd.0 since back 2024-04-14T14:30:57.614597+0000 front 2024-04-14T14:30:57.614586+0000 (oldest deadline 2024-04-14T14:31:23.514362+0000) 2024-04-14T14:31:38.094 INFO:journalctl@ceph.osd.6.smithi122.stdout:Apr 14 14:31:37 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-osd-6[87451]: 2024-04-14T14:31:37.836+0000 7ff647f73700 -1 osd.6 162 heartbeat_check: no reply from 172.21.15.69:6814 osd.1 since back 2024-04-14T14:31:05.815477+0000 front 2024-04-14T14:31:05.815418+0000 (oldest deadline 2024-04-14T14:31:30.515245+0000) 2024-04-14T14:31:38.094 INFO:journalctl@ceph.osd.7.smithi122.stdout:Apr 14 14:31:37 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-osd-7[92447]: 2024-04-14T14:31:37.900+0000 7f97c2275700 -1 osd.7 162 heartbeat_check: no reply from 172.21.15.69:6806 osd.0 since back 2024-04-14T14:30:59.635081+0000 front 2024-04-14T14:30:59.635113+0000 (oldest deadline 2024-04-14T14:31:25.535039+0000) 2024-04-14T14:31:38.094 INFO:journalctl@ceph.osd.7.smithi122.stdout:Apr 14 14:31:37 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-osd-7[92447]: 2024-04-14T14:31:37.900+0000 7f97c2275700 -1 osd.7 162 heartbeat_check: no reply from 172.21.15.69:6814 osd.1 since back 2024-04-14T14:31:06.635906+0000 front 2024-04-14T14:31:06.635997+0000 (oldest deadline 2024-04-14T14:31:30.135764+0000) 2024-04-14T14:31:38.937 INFO:journalctl@ceph.osd.6.smithi122.stdout:Apr 14 14:31:38 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-osd-6[87451]: 2024-04-14T14:31:38.791+0000 7ff647f73700 -1 osd.6 162 heartbeat_check: no reply from 172.21.15.69:6806 osd.0 since back 2024-04-14T14:30:57.614597+0000 front 2024-04-14T14:30:57.614586+0000 (oldest deadline 2024-04-14T14:31:23.514362+0000) 2024-04-14T14:31:38.937 INFO:journalctl@ceph.osd.6.smithi122.stdout:Apr 14 14:31:38 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-osd-6[87451]: 2024-04-14T14:31:38.791+0000 7ff647f73700 -1 osd.6 162 heartbeat_check: no reply from 172.21.15.69:6814 osd.1 since back 2024-04-14T14:31:05.815477+0000 front 2024-04-14T14:31:05.815418+0000 (oldest deadline 2024-04-14T14:31:30.515245+0000) 2024-04-14T14:31:38.937 INFO:journalctl@ceph.osd.5.smithi122.stdout:Apr 14 14:31:38 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-osd-5[82734]: 2024-04-14T14:31:38.617+0000 7eff07d49700 -1 osd.5 162 heartbeat_check: no reply from 172.21.15.69:6806 osd.0 since back 2024-04-14T14:30:57.287308+0000 front 2024-04-14T14:30:57.287435+0000 (oldest deadline 2024-04-14T14:31:23.187249+0000) 2024-04-14T14:31:38.937 INFO:journalctl@ceph.osd.5.smithi122.stdout:Apr 14 14:31:38 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-osd-5[82734]: 2024-04-14T14:31:38.617+0000 7eff07d49700 -1 osd.5 162 heartbeat_check: no reply from 172.21.15.69:6814 osd.1 since back 2024-04-14T14:31:07.888129+0000 front 2024-04-14T14:31:07.888233+0000 (oldest deadline 2024-04-14T14:31:30.788034+0000) 2024-04-14T14:31:38.937 INFO:journalctl@ceph.osd.5.smithi122.stdout:Apr 14 14:31:38 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-osd-5[82734]: 2024-04-14T14:31:38.617+0000 7eff07d49700 -1 osd.5 162 heartbeat_check: no reply from 172.21.15.69:6822 osd.2 since back 2024-04-14T14:31:11.889113+0000 front 2024-04-14T14:31:11.888947+0000 (oldest deadline 2024-04-14T14:31:37.189036+0000) 2024-04-14T14:31:39.344 INFO:journalctl@ceph.osd.7.smithi122.stdout:Apr 14 14:31:38 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-osd-7[92447]: 2024-04-14T14:31:38.936+0000 7f97c2275700 -1 osd.7 162 heartbeat_check: no reply from 172.21.15.69:6806 osd.0 since back 2024-04-14T14:30:59.635081+0000 front 2024-04-14T14:30:59.635113+0000 (oldest deadline 2024-04-14T14:31:25.535039+0000) 2024-04-14T14:31:39.344 INFO:journalctl@ceph.osd.7.smithi122.stdout:Apr 14 14:31:38 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-osd-7[92447]: 2024-04-14T14:31:38.936+0000 7f97c2275700 -1 osd.7 162 heartbeat_check: no reply from 172.21.15.69:6814 osd.1 since back 2024-04-14T14:31:06.635906+0000 front 2024-04-14T14:31:06.635997+0000 (oldest deadline 2024-04-14T14:31:30.135764+0000) 2024-04-14T14:31:39.981 INFO:journalctl@ceph.osd.6.smithi122.stdout:Apr 14 14:31:39 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-osd-6[87451]: 2024-04-14T14:31:39.768+0000 7ff647f73700 -1 osd.6 162 heartbeat_check: no reply from 172.21.15.69:6806 osd.0 since back 2024-04-14T14:30:57.614597+0000 front 2024-04-14T14:30:57.614586+0000 (oldest deadline 2024-04-14T14:31:23.514362+0000) 2024-04-14T14:31:39.981 INFO:journalctl@ceph.osd.6.smithi122.stdout:Apr 14 14:31:39 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-osd-6[87451]: 2024-04-14T14:31:39.768+0000 7ff647f73700 -1 osd.6 162 heartbeat_check: no reply from 172.21.15.69:6814 osd.1 since back 2024-04-14T14:31:05.815477+0000 front 2024-04-14T14:31:05.815418+0000 (oldest deadline 2024-04-14T14:31:30.515245+0000) 2024-04-14T14:31:39.981 INFO:journalctl@ceph.osd.5.smithi122.stdout:Apr 14 14:31:39 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-osd-5[82734]: 2024-04-14T14:31:39.609+0000 7eff07d49700 -1 osd.5 162 heartbeat_check: no reply from 172.21.15.69:6806 osd.0 since back 2024-04-14T14:30:57.287308+0000 front 2024-04-14T14:30:57.287435+0000 (oldest deadline 2024-04-14T14:31:23.187249+0000) 2024-04-14T14:31:39.981 INFO:journalctl@ceph.osd.5.smithi122.stdout:Apr 14 14:31:39 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-osd-5[82734]: 2024-04-14T14:31:39.609+0000 7eff07d49700 -1 osd.5 162 heartbeat_check: no reply from 172.21.15.69:6814 osd.1 since back 2024-04-14T14:31:07.888129+0000 front 2024-04-14T14:31:07.888233+0000 (oldest deadline 2024-04-14T14:31:30.788034+0000) 2024-04-14T14:31:39.981 INFO:journalctl@ceph.osd.5.smithi122.stdout:Apr 14 14:31:39 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-osd-5[82734]: 2024-04-14T14:31:39.609+0000 7eff07d49700 -1 osd.5 162 heartbeat_check: no reply from 172.21.15.69:6822 osd.2 since back 2024-04-14T14:31:11.889113+0000 front 2024-04-14T14:31:11.888947+0000 (oldest deadline 2024-04-14T14:31:37.189036+0000) 2024-04-14T14:31:40.344 INFO:journalctl@ceph.osd.7.smithi122.stdout:Apr 14 14:31:39 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-osd-7[92447]: 2024-04-14T14:31:39.980+0000 7f97c2275700 -1 osd.7 162 heartbeat_check: no reply from 172.21.15.69:6806 osd.0 since back 2024-04-14T14:30:59.635081+0000 front 2024-04-14T14:30:59.635113+0000 (oldest deadline 2024-04-14T14:31:25.535039+0000) 2024-04-14T14:31:40.344 INFO:journalctl@ceph.osd.7.smithi122.stdout:Apr 14 14:31:39 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-osd-7[92447]: 2024-04-14T14:31:39.980+0000 7f97c2275700 -1 osd.7 162 heartbeat_check: no reply from 172.21.15.69:6814 osd.1 since back 2024-04-14T14:31:06.635906+0000 front 2024-04-14T14:31:06.635997+0000 (oldest deadline 2024-04-14T14:31:30.135764+0000) 2024-04-14T14:31:40.344 INFO:journalctl@ceph.osd.7.smithi122.stdout:Apr 14 14:31:39 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-osd-7[92447]: 2024-04-14T14:31:39.980+0000 7f97c2275700 -1 osd.7 162 heartbeat_check: no reply from 172.21.15.69:6822 osd.2 since back 2024-04-14T14:31:16.338287+0000 front 2024-04-14T14:31:16.338264+0000 (oldest deadline 2024-04-14T14:31:39.238272+0000) 2024-04-14T14:31:40.844 INFO:journalctl@ceph.osd.5.smithi122.stdout:Apr 14 14:31:40 smithi122 podman[101514]: 2024-04-14 14:31:40.541472197 +0000 UTC m=+5.535210543 container died 02b623d44db3e6fbaec10a4e9c5d4247395195112deaba2950c7f364765104fa (image=quay.ceph.io/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888, name=ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-osd-5, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.build-date=20240403, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, RELEASE=main-a971fd5, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-14T14:31:40.844 INFO:journalctl@ceph.osd.6.smithi122.stdout:Apr 14 14:31:40 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-osd-6[87451]: 2024-04-14T14:31:40.756+0000 7ff647f73700 -1 osd.6 162 heartbeat_check: no reply from 172.21.15.69:6806 osd.0 since back 2024-04-14T14:30:57.614597+0000 front 2024-04-14T14:30:57.614586+0000 (oldest deadline 2024-04-14T14:31:23.514362+0000) 2024-04-14T14:31:40.845 INFO:journalctl@ceph.osd.6.smithi122.stdout:Apr 14 14:31:40 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-osd-6[87451]: 2024-04-14T14:31:40.756+0000 7ff647f73700 -1 osd.6 162 heartbeat_check: no reply from 172.21.15.69:6814 osd.1 since back 2024-04-14T14:31:05.815477+0000 front 2024-04-14T14:31:05.815418+0000 (oldest deadline 2024-04-14T14:31:30.515245+0000) 2024-04-14T14:31:40.845 INFO:journalctl@ceph.osd.6.smithi122.stdout:Apr 14 14:31:40 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-osd-6[87451]: 2024-04-14T14:31:40.756+0000 7ff647f73700 -1 osd.6 162 heartbeat_check: no reply from 172.21.15.69:6822 osd.2 since back 2024-04-14T14:31:16.416320+0000 front 2024-04-14T14:31:16.416306+0000 (oldest deadline 2024-04-14T14:31:39.916121+0000) 2024-04-14T14:31:41.344 INFO:journalctl@ceph.osd.7.smithi122.stdout:Apr 14 14:31:40 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-osd-7[92447]: 2024-04-14T14:31:40.994+0000 7f97c2275700 -1 osd.7 162 heartbeat_check: no reply from 172.21.15.69:6806 osd.0 since back 2024-04-14T14:30:59.635081+0000 front 2024-04-14T14:30:59.635113+0000 (oldest deadline 2024-04-14T14:31:25.535039+0000) 2024-04-14T14:31:41.344 INFO:journalctl@ceph.osd.7.smithi122.stdout:Apr 14 14:31:40 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-osd-7[92447]: 2024-04-14T14:31:40.994+0000 7f97c2275700 -1 osd.7 162 heartbeat_check: no reply from 172.21.15.69:6814 osd.1 since back 2024-04-14T14:31:06.635906+0000 front 2024-04-14T14:31:06.635997+0000 (oldest deadline 2024-04-14T14:31:30.135764+0000) 2024-04-14T14:31:41.344 INFO:journalctl@ceph.osd.7.smithi122.stdout:Apr 14 14:31:40 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-osd-7[92447]: 2024-04-14T14:31:40.994+0000 7f97c2275700 -1 osd.7 162 heartbeat_check: no reply from 172.21.15.69:6822 osd.2 since back 2024-04-14T14:31:16.338287+0000 front 2024-04-14T14:31:16.338264+0000 (oldest deadline 2024-04-14T14:31:39.238272+0000) 2024-04-14T14:31:41.344 INFO:journalctl@ceph.osd.5.smithi122.stdout:Apr 14 14:31:40 smithi122 podman[101514]: 2024-04-14 14:31:40.923167404 +0000 UTC m=+5.916905761 container cleanup 02b623d44db3e6fbaec10a4e9c5d4247395195112deaba2950c7f364765104fa (image=quay.ceph.io/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888, name=ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-osd-5, ceph=True, org.label-schema.build-date=20240403, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, RELEASE=main-a971fd5, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True) 2024-04-14T14:31:41.345 INFO:journalctl@ceph.osd.5.smithi122.stdout:Apr 14 14:31:40 smithi122 bash[101514]: ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-osd-5 2024-04-14T14:31:41.345 INFO:journalctl@ceph.osd.5.smithi122.stdout:Apr 14 14:31:41 smithi122 podman[101531]: 2024-04-14 14:31:41.166102606 +0000 UTC m=+0.623276572 container remove 02b623d44db3e6fbaec10a4e9c5d4247395195112deaba2950c7f364765104fa (image=quay.ceph.io/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888, name=ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-osd-5, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=main-a971fd5, org.label-schema.license=GPLv2, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_CLEAN=True, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240403) 2024-04-14T14:31:41.794 INFO:journalctl@ceph.osd.5.smithi122.stdout:Apr 14 14:31:41 smithi122 podman[101623]: 2024-04-14 14:31:41.382147299 +0000 UTC m=+0.017433631 image pull b96c509eef841460b3271f5fdd331860da87b004524f4ed59414aa3680183b53 quay.ceph.io/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888 2024-04-14T14:31:41.795 INFO:journalctl@ceph.osd.5.smithi122.stdout:Apr 14 14:31:41 smithi122 podman[101623]: 2024-04-14 14:31:41.47982923 +0000 UTC m=+0.115115551 container create 38a73023d76552c0850fa512059148e1a20df01fde6596e92ec30f4a6c9b1dc0 (image=quay.ceph.io/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888, name=ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-osd-5-deactivate, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.build-date=20240403, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , RELEASE=main-a971fd5, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2) 2024-04-14T14:31:41.795 INFO:journalctl@ceph.osd.5.smithi122.stdout:Apr 14 14:31:41 smithi122 podman[101623]: 2024-04-14 14:31:41.711466708 +0000 UTC m=+0.346753039 container init 38a73023d76552c0850fa512059148e1a20df01fde6596e92ec30f4a6c9b1dc0 (image=quay.ceph.io/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888, name=ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-osd-5-deactivate, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, org.label-schema.build-date=20240403, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, RELEASE=main-a971fd5, ceph=True) 2024-04-14T14:31:41.795 INFO:journalctl@ceph.osd.5.smithi122.stdout:Apr 14 14:31:41 smithi122 podman[101623]: 2024-04-14 14:31:41.714952942 +0000 UTC m=+0.350239265 container start 38a73023d76552c0850fa512059148e1a20df01fde6596e92ec30f4a6c9b1dc0 (image=quay.ceph.io/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888, name=ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-osd-5-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_BRANCH=HEAD, RELEASE=main-a971fd5, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.build-date=20240403, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, GIT_CLEAN=True) 2024-04-14T14:31:41.795 INFO:journalctl@ceph.osd.5.smithi122.stdout:Apr 14 14:31:41 smithi122 podman[101623]: 2024-04-14 14:31:41.794401386 +0000 UTC m=+0.429687709 container attach 38a73023d76552c0850fa512059148e1a20df01fde6596e92ec30f4a6c9b1dc0 (image=quay.ceph.io/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888, name=ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-osd-5-deactivate, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_CLEAN=True, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.build-date=20240403, RELEASE=main-a971fd5, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-14T14:31:42.094 INFO:journalctl@ceph.osd.5.smithi122.stdout:Apr 14 14:31:42 smithi122 podman[101623]: 2024-04-14 14:31:42.000903939 +0000 UTC m=+0.636190272 container died 38a73023d76552c0850fa512059148e1a20df01fde6596e92ec30f4a6c9b1dc0 (image=quay.ceph.io/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888, name=ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-osd-5-deactivate, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, RELEASE=main-a971fd5, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.build-date=20240403, maintainer=Guillaume Abrioux , GIT_CLEAN=True, io.buildah.version=1.33.5, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0) 2024-04-14T14:31:42.094 INFO:journalctl@ceph.osd.6.smithi122.stdout:Apr 14 14:31:41 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-osd-6[87451]: 2024-04-14T14:31:41.803+0000 7ff647f73700 -1 osd.6 162 heartbeat_check: no reply from 172.21.15.69:6806 osd.0 since back 2024-04-14T14:30:57.614597+0000 front 2024-04-14T14:30:57.614586+0000 (oldest deadline 2024-04-14T14:31:23.514362+0000) 2024-04-14T14:31:42.094 INFO:journalctl@ceph.osd.6.smithi122.stdout:Apr 14 14:31:41 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-osd-6[87451]: 2024-04-14T14:31:41.803+0000 7ff647f73700 -1 osd.6 162 heartbeat_check: no reply from 172.21.15.69:6814 osd.1 since back 2024-04-14T14:31:05.815477+0000 front 2024-04-14T14:31:05.815418+0000 (oldest deadline 2024-04-14T14:31:30.515245+0000) 2024-04-14T14:31:42.095 INFO:journalctl@ceph.osd.6.smithi122.stdout:Apr 14 14:31:41 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-osd-6[87451]: 2024-04-14T14:31:41.803+0000 7ff647f73700 -1 osd.6 162 heartbeat_check: no reply from 172.21.15.69:6822 osd.2 since back 2024-04-14T14:31:16.416320+0000 front 2024-04-14T14:31:16.416306+0000 (oldest deadline 2024-04-14T14:31:39.916121+0000) 2024-04-14T14:31:42.095 INFO:journalctl@ceph.osd.7.smithi122.stdout:Apr 14 14:31:41 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-osd-7[92447]: 2024-04-14T14:31:41.991+0000 7f97c2275700 -1 osd.7 162 heartbeat_check: no reply from 172.21.15.69:6806 osd.0 since back 2024-04-14T14:30:59.635081+0000 front 2024-04-14T14:30:59.635113+0000 (oldest deadline 2024-04-14T14:31:25.535039+0000) 2024-04-14T14:31:42.095 INFO:journalctl@ceph.osd.7.smithi122.stdout:Apr 14 14:31:41 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-osd-7[92447]: 2024-04-14T14:31:41.991+0000 7f97c2275700 -1 osd.7 162 heartbeat_check: no reply from 172.21.15.69:6814 osd.1 since back 2024-04-14T14:31:06.635906+0000 front 2024-04-14T14:31:06.635997+0000 (oldest deadline 2024-04-14T14:31:30.135764+0000) 2024-04-14T14:31:42.095 INFO:journalctl@ceph.osd.7.smithi122.stdout:Apr 14 14:31:41 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-osd-7[92447]: 2024-04-14T14:31:41.991+0000 7f97c2275700 -1 osd.7 162 heartbeat_check: no reply from 172.21.15.69:6822 osd.2 since back 2024-04-14T14:31:16.338287+0000 front 2024-04-14T14:31:16.338264+0000 (oldest deadline 2024-04-14T14:31:39.238272+0000) 2024-04-14T14:31:42.631 DEBUG:teuthology.orchestra.run.smithi122:> sudo pkill -f 'journalctl -f -n 0 -u ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968@osd.5.service' 2024-04-14T14:31:42.671 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-14T14:31:42.672 INFO:tasks.cephadm.osd.5:Stopped osd.5 2024-04-14T14:31:42.672 INFO:tasks.cephadm.osd.6:Stopping osd.6... 2024-04-14T14:31:42.672 DEBUG:teuthology.orchestra.run.smithi122:> sudo systemctl stop ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968@osd.6 2024-04-14T14:31:42.954 INFO:journalctl@ceph.osd.6.smithi122.stdout:Apr 14 14:31:42 smithi122 systemd[1]: Stopping Ceph osd.6 for dc326fc2-fa65-11ee-bc8e-c7b262605968... 2024-04-14T14:31:42.954 INFO:journalctl@ceph.osd.6.smithi122.stdout:Apr 14 14:31:42 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-osd-6[87451]: 2024-04-14T14:31:42.836+0000 7ff647f73700 -1 osd.6 162 heartbeat_check: no reply from 172.21.15.69:6806 osd.0 since back 2024-04-14T14:30:57.614597+0000 front 2024-04-14T14:30:57.614586+0000 (oldest deadline 2024-04-14T14:31:23.514362+0000) 2024-04-14T14:31:42.954 INFO:journalctl@ceph.osd.6.smithi122.stdout:Apr 14 14:31:42 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-osd-6[87451]: 2024-04-14T14:31:42.836+0000 7ff647f73700 -1 osd.6 162 heartbeat_check: no reply from 172.21.15.69:6814 osd.1 since back 2024-04-14T14:31:05.815477+0000 front 2024-04-14T14:31:05.815418+0000 (oldest deadline 2024-04-14T14:31:30.515245+0000) 2024-04-14T14:31:42.954 INFO:journalctl@ceph.osd.6.smithi122.stdout:Apr 14 14:31:42 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-osd-6[87451]: 2024-04-14T14:31:42.836+0000 7ff647f73700 -1 osd.6 162 heartbeat_check: no reply from 172.21.15.69:6822 osd.2 since back 2024-04-14T14:31:16.416320+0000 front 2024-04-14T14:31:16.416306+0000 (oldest deadline 2024-04-14T14:31:39.916121+0000) 2024-04-14T14:31:43.344 INFO:journalctl@ceph.osd.6.smithi122.stdout:Apr 14 14:31:43 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-osd-6[87451]: 2024-04-14T14:31:43.000+0000 7ff64f987700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.6 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-14T14:31:43.344 INFO:journalctl@ceph.osd.6.smithi122.stdout:Apr 14 14:31:43 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-osd-6[87451]: 2024-04-14T14:31:43.000+0000 7ff64f987700 -1 osd.6 162 *** Got signal Terminated *** 2024-04-14T14:31:43.344 INFO:journalctl@ceph.osd.6.smithi122.stdout:Apr 14 14:31:43 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-osd-6[87451]: 2024-04-14T14:31:43.000+0000 7ff64f987700 -1 osd.6 162 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-14T14:31:43.344 INFO:journalctl@ceph.osd.7.smithi122.stdout:Apr 14 14:31:42 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-osd-7[92447]: 2024-04-14T14:31:42.953+0000 7f97c2275700 -1 osd.7 162 heartbeat_check: no reply from 172.21.15.69:6806 osd.0 since back 2024-04-14T14:30:59.635081+0000 front 2024-04-14T14:30:59.635113+0000 (oldest deadline 2024-04-14T14:31:25.535039+0000) 2024-04-14T14:31:43.345 INFO:journalctl@ceph.osd.7.smithi122.stdout:Apr 14 14:31:42 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-osd-7[92447]: 2024-04-14T14:31:42.953+0000 7f97c2275700 -1 osd.7 162 heartbeat_check: no reply from 172.21.15.69:6814 osd.1 since back 2024-04-14T14:31:06.635906+0000 front 2024-04-14T14:31:06.635997+0000 (oldest deadline 2024-04-14T14:31:30.135764+0000) 2024-04-14T14:31:43.345 INFO:journalctl@ceph.osd.7.smithi122.stdout:Apr 14 14:31:42 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-osd-7[92447]: 2024-04-14T14:31:42.953+0000 7f97c2275700 -1 osd.7 162 heartbeat_check: no reply from 172.21.15.69:6822 osd.2 since back 2024-04-14T14:31:16.338287+0000 front 2024-04-14T14:31:16.338264+0000 (oldest deadline 2024-04-14T14:31:39.238272+0000) 2024-04-14T14:31:44.093 INFO:journalctl@ceph.osd.6.smithi122.stdout:Apr 14 14:31:43 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-osd-6[87451]: 2024-04-14T14:31:43.836+0000 7ff647f73700 -1 osd.6 162 heartbeat_check: no reply from 172.21.15.69:6806 osd.0 since back 2024-04-14T14:30:57.614597+0000 front 2024-04-14T14:30:57.614586+0000 (oldest deadline 2024-04-14T14:31:23.514362+0000) 2024-04-14T14:31:44.093 INFO:journalctl@ceph.osd.6.smithi122.stdout:Apr 14 14:31:43 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-osd-6[87451]: 2024-04-14T14:31:43.836+0000 7ff647f73700 -1 osd.6 162 heartbeat_check: no reply from 172.21.15.69:6814 osd.1 since back 2024-04-14T14:31:05.815477+0000 front 2024-04-14T14:31:05.815418+0000 (oldest deadline 2024-04-14T14:31:30.515245+0000) 2024-04-14T14:31:44.093 INFO:journalctl@ceph.osd.6.smithi122.stdout:Apr 14 14:31:43 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-osd-6[87451]: 2024-04-14T14:31:43.836+0000 7ff647f73700 -1 osd.6 162 heartbeat_check: no reply from 172.21.15.69:6822 osd.2 since back 2024-04-14T14:31:16.416320+0000 front 2024-04-14T14:31:16.416306+0000 (oldest deadline 2024-04-14T14:31:39.916121+0000) 2024-04-14T14:31:44.093 INFO:journalctl@ceph.osd.7.smithi122.stdout:Apr 14 14:31:43 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-osd-7[92447]: 2024-04-14T14:31:43.949+0000 7f97c2275700 -1 osd.7 162 heartbeat_check: no reply from 172.21.15.69:6806 osd.0 since back 2024-04-14T14:30:59.635081+0000 front 2024-04-14T14:30:59.635113+0000 (oldest deadline 2024-04-14T14:31:25.535039+0000) 2024-04-14T14:31:44.093 INFO:journalctl@ceph.osd.7.smithi122.stdout:Apr 14 14:31:43 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-osd-7[92447]: 2024-04-14T14:31:43.949+0000 7f97c2275700 -1 osd.7 162 heartbeat_check: no reply from 172.21.15.69:6814 osd.1 since back 2024-04-14T14:31:06.635906+0000 front 2024-04-14T14:31:06.635997+0000 (oldest deadline 2024-04-14T14:31:30.135764+0000) 2024-04-14T14:31:44.093 INFO:journalctl@ceph.osd.7.smithi122.stdout:Apr 14 14:31:43 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-osd-7[92447]: 2024-04-14T14:31:43.949+0000 7f97c2275700 -1 osd.7 162 heartbeat_check: no reply from 172.21.15.69:6822 osd.2 since back 2024-04-14T14:31:16.338287+0000 front 2024-04-14T14:31:16.338264+0000 (oldest deadline 2024-04-14T14:31:39.238272+0000) 2024-04-14T14:31:45.094 INFO:journalctl@ceph.osd.6.smithi122.stdout:Apr 14 14:31:44 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-osd-6[87451]: 2024-04-14T14:31:44.813+0000 7ff647f73700 -1 osd.6 162 heartbeat_check: no reply from 172.21.15.69:6806 osd.0 since back 2024-04-14T14:30:57.614597+0000 front 2024-04-14T14:30:57.614586+0000 (oldest deadline 2024-04-14T14:31:23.514362+0000) 2024-04-14T14:31:45.094 INFO:journalctl@ceph.osd.6.smithi122.stdout:Apr 14 14:31:44 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-osd-6[87451]: 2024-04-14T14:31:44.813+0000 7ff647f73700 -1 osd.6 162 heartbeat_check: no reply from 172.21.15.69:6814 osd.1 since back 2024-04-14T14:31:05.815477+0000 front 2024-04-14T14:31:05.815418+0000 (oldest deadline 2024-04-14T14:31:30.515245+0000) 2024-04-14T14:31:45.094 INFO:journalctl@ceph.osd.6.smithi122.stdout:Apr 14 14:31:44 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-osd-6[87451]: 2024-04-14T14:31:44.813+0000 7ff647f73700 -1 osd.6 162 heartbeat_check: no reply from 172.21.15.69:6822 osd.2 since back 2024-04-14T14:31:16.416320+0000 front 2024-04-14T14:31:16.416306+0000 (oldest deadline 2024-04-14T14:31:39.916121+0000) 2024-04-14T14:31:45.094 INFO:journalctl@ceph.osd.7.smithi122.stdout:Apr 14 14:31:44 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-osd-7[92447]: 2024-04-14T14:31:44.920+0000 7f97c2275700 -1 osd.7 162 heartbeat_check: no reply from 172.21.15.69:6806 osd.0 since back 2024-04-14T14:30:59.635081+0000 front 2024-04-14T14:30:59.635113+0000 (oldest deadline 2024-04-14T14:31:25.535039+0000) 2024-04-14T14:31:45.094 INFO:journalctl@ceph.osd.7.smithi122.stdout:Apr 14 14:31:44 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-osd-7[92447]: 2024-04-14T14:31:44.920+0000 7f97c2275700 -1 osd.7 162 heartbeat_check: no reply from 172.21.15.69:6814 osd.1 since back 2024-04-14T14:31:06.635906+0000 front 2024-04-14T14:31:06.635997+0000 (oldest deadline 2024-04-14T14:31:30.135764+0000) 2024-04-14T14:31:45.094 INFO:journalctl@ceph.osd.7.smithi122.stdout:Apr 14 14:31:44 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-osd-7[92447]: 2024-04-14T14:31:44.920+0000 7f97c2275700 -1 osd.7 162 heartbeat_check: no reply from 172.21.15.69:6822 osd.2 since back 2024-04-14T14:31:16.338287+0000 front 2024-04-14T14:31:16.338264+0000 (oldest deadline 2024-04-14T14:31:39.238272+0000) 2024-04-14T14:31:46.094 INFO:journalctl@ceph.osd.6.smithi122.stdout:Apr 14 14:31:45 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-osd-6[87451]: 2024-04-14T14:31:45.767+0000 7ff647f73700 -1 osd.6 162 heartbeat_check: no reply from 172.21.15.69:6806 osd.0 since back 2024-04-14T14:30:57.614597+0000 front 2024-04-14T14:30:57.614586+0000 (oldest deadline 2024-04-14T14:31:23.514362+0000) 2024-04-14T14:31:46.094 INFO:journalctl@ceph.osd.6.smithi122.stdout:Apr 14 14:31:45 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-osd-6[87451]: 2024-04-14T14:31:45.767+0000 7ff647f73700 -1 osd.6 162 heartbeat_check: no reply from 172.21.15.69:6814 osd.1 since back 2024-04-14T14:31:05.815477+0000 front 2024-04-14T14:31:05.815418+0000 (oldest deadline 2024-04-14T14:31:30.515245+0000) 2024-04-14T14:31:46.094 INFO:journalctl@ceph.osd.6.smithi122.stdout:Apr 14 14:31:45 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-osd-6[87451]: 2024-04-14T14:31:45.767+0000 7ff647f73700 -1 osd.6 162 heartbeat_check: no reply from 172.21.15.69:6822 osd.2 since back 2024-04-14T14:31:16.416320+0000 front 2024-04-14T14:31:16.416306+0000 (oldest deadline 2024-04-14T14:31:39.916121+0000) 2024-04-14T14:31:46.095 INFO:journalctl@ceph.osd.6.smithi122.stdout:Apr 14 14:31:45 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-osd-6[87451]: 2024-04-14T14:31:45.767+0000 7ff647f73700 -1 osd.6 162 heartbeat_check: no reply from 172.21.15.69:6830 osd.3 since back 2024-04-14T14:31:24.017037+0000 front 2024-04-14T14:31:24.017090+0000 (oldest deadline 2024-04-14T14:31:45.116995+0000) 2024-04-14T14:31:46.095 INFO:journalctl@ceph.osd.7.smithi122.stdout:Apr 14 14:31:45 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-osd-7[92447]: 2024-04-14T14:31:45.896+0000 7f97c2275700 -1 osd.7 162 heartbeat_check: no reply from 172.21.15.69:6806 osd.0 since back 2024-04-14T14:30:59.635081+0000 front 2024-04-14T14:30:59.635113+0000 (oldest deadline 2024-04-14T14:31:25.535039+0000) 2024-04-14T14:31:46.095 INFO:journalctl@ceph.osd.7.smithi122.stdout:Apr 14 14:31:45 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-osd-7[92447]: 2024-04-14T14:31:45.896+0000 7f97c2275700 -1 osd.7 162 heartbeat_check: no reply from 172.21.15.69:6814 osd.1 since back 2024-04-14T14:31:06.635906+0000 front 2024-04-14T14:31:06.635997+0000 (oldest deadline 2024-04-14T14:31:30.135764+0000) 2024-04-14T14:31:46.095 INFO:journalctl@ceph.osd.7.smithi122.stdout:Apr 14 14:31:45 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-osd-7[92447]: 2024-04-14T14:31:45.896+0000 7f97c2275700 -1 osd.7 162 heartbeat_check: no reply from 172.21.15.69:6822 osd.2 since back 2024-04-14T14:31:16.338287+0000 front 2024-04-14T14:31:16.338264+0000 (oldest deadline 2024-04-14T14:31:39.238272+0000) 2024-04-14T14:31:46.095 INFO:journalctl@ceph.osd.7.smithi122.stdout:Apr 14 14:31:45 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-osd-7[92447]: 2024-04-14T14:31:45.896+0000 7f97c2275700 -1 osd.7 162 heartbeat_check: no reply from 172.21.15.69:6830 osd.3 since back 2024-04-14T14:31:19.238775+0000 front 2024-04-14T14:31:19.238839+0000 (oldest deadline 2024-04-14T14:31:45.138624+0000) 2024-04-14T14:31:47.094 INFO:journalctl@ceph.osd.6.smithi122.stdout:Apr 14 14:31:46 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-osd-6[87451]: 2024-04-14T14:31:46.784+0000 7ff647f73700 -1 osd.6 162 heartbeat_check: no reply from 172.21.15.69:6806 osd.0 since back 2024-04-14T14:30:57.614597+0000 front 2024-04-14T14:30:57.614586+0000 (oldest deadline 2024-04-14T14:31:23.514362+0000) 2024-04-14T14:31:47.094 INFO:journalctl@ceph.osd.6.smithi122.stdout:Apr 14 14:31:46 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-osd-6[87451]: 2024-04-14T14:31:46.784+0000 7ff647f73700 -1 osd.6 162 heartbeat_check: no reply from 172.21.15.69:6814 osd.1 since back 2024-04-14T14:31:05.815477+0000 front 2024-04-14T14:31:05.815418+0000 (oldest deadline 2024-04-14T14:31:30.515245+0000) 2024-04-14T14:31:47.094 INFO:journalctl@ceph.osd.6.smithi122.stdout:Apr 14 14:31:46 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-osd-6[87451]: 2024-04-14T14:31:46.784+0000 7ff647f73700 -1 osd.6 162 heartbeat_check: no reply from 172.21.15.69:6822 osd.2 since back 2024-04-14T14:31:16.416320+0000 front 2024-04-14T14:31:16.416306+0000 (oldest deadline 2024-04-14T14:31:39.916121+0000) 2024-04-14T14:31:47.094 INFO:journalctl@ceph.osd.6.smithi122.stdout:Apr 14 14:31:46 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-osd-6[87451]: 2024-04-14T14:31:46.784+0000 7ff647f73700 -1 osd.6 162 heartbeat_check: no reply from 172.21.15.69:6830 osd.3 since back 2024-04-14T14:31:24.017037+0000 front 2024-04-14T14:31:24.017090+0000 (oldest deadline 2024-04-14T14:31:45.116995+0000) 2024-04-14T14:31:47.094 INFO:journalctl@ceph.osd.7.smithi122.stdout:Apr 14 14:31:46 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-osd-7[92447]: 2024-04-14T14:31:46.908+0000 7f97c2275700 -1 osd.7 162 heartbeat_check: no reply from 172.21.15.69:6806 osd.0 since back 2024-04-14T14:30:59.635081+0000 front 2024-04-14T14:30:59.635113+0000 (oldest deadline 2024-04-14T14:31:25.535039+0000) 2024-04-14T14:31:47.094 INFO:journalctl@ceph.osd.7.smithi122.stdout:Apr 14 14:31:46 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-osd-7[92447]: 2024-04-14T14:31:46.908+0000 7f97c2275700 -1 osd.7 162 heartbeat_check: no reply from 172.21.15.69:6814 osd.1 since back 2024-04-14T14:31:06.635906+0000 front 2024-04-14T14:31:06.635997+0000 (oldest deadline 2024-04-14T14:31:30.135764+0000) 2024-04-14T14:31:47.094 INFO:journalctl@ceph.osd.7.smithi122.stdout:Apr 14 14:31:46 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-osd-7[92447]: 2024-04-14T14:31:46.908+0000 7f97c2275700 -1 osd.7 162 heartbeat_check: no reply from 172.21.15.69:6822 osd.2 since back 2024-04-14T14:31:16.338287+0000 front 2024-04-14T14:31:16.338264+0000 (oldest deadline 2024-04-14T14:31:39.238272+0000) 2024-04-14T14:31:47.095 INFO:journalctl@ceph.osd.7.smithi122.stdout:Apr 14 14:31:46 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-osd-7[92447]: 2024-04-14T14:31:46.908+0000 7f97c2275700 -1 osd.7 162 heartbeat_check: no reply from 172.21.15.69:6830 osd.3 since back 2024-04-14T14:31:19.238775+0000 front 2024-04-14T14:31:19.238839+0000 (oldest deadline 2024-04-14T14:31:45.138624+0000) 2024-04-14T14:31:48.094 INFO:journalctl@ceph.osd.6.smithi122.stdout:Apr 14 14:31:47 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-osd-6[87451]: 2024-04-14T14:31:47.808+0000 7ff647f73700 -1 osd.6 162 heartbeat_check: no reply from 172.21.15.69:6806 osd.0 since back 2024-04-14T14:30:57.614597+0000 front 2024-04-14T14:30:57.614586+0000 (oldest deadline 2024-04-14T14:31:23.514362+0000) 2024-04-14T14:31:48.094 INFO:journalctl@ceph.osd.6.smithi122.stdout:Apr 14 14:31:47 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-osd-6[87451]: 2024-04-14T14:31:47.808+0000 7ff647f73700 -1 osd.6 162 heartbeat_check: no reply from 172.21.15.69:6814 osd.1 since back 2024-04-14T14:31:05.815477+0000 front 2024-04-14T14:31:05.815418+0000 (oldest deadline 2024-04-14T14:31:30.515245+0000) 2024-04-14T14:31:48.094 INFO:journalctl@ceph.osd.6.smithi122.stdout:Apr 14 14:31:47 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-osd-6[87451]: 2024-04-14T14:31:47.808+0000 7ff647f73700 -1 osd.6 162 heartbeat_check: no reply from 172.21.15.69:6822 osd.2 since back 2024-04-14T14:31:16.416320+0000 front 2024-04-14T14:31:16.416306+0000 (oldest deadline 2024-04-14T14:31:39.916121+0000) 2024-04-14T14:31:48.094 INFO:journalctl@ceph.osd.6.smithi122.stdout:Apr 14 14:31:47 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-osd-6[87451]: 2024-04-14T14:31:47.808+0000 7ff647f73700 -1 osd.6 162 heartbeat_check: no reply from 172.21.15.69:6830 osd.3 since back 2024-04-14T14:31:24.017037+0000 front 2024-04-14T14:31:24.017090+0000 (oldest deadline 2024-04-14T14:31:45.116995+0000) 2024-04-14T14:31:48.094 INFO:journalctl@ceph.osd.7.smithi122.stdout:Apr 14 14:31:47 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-osd-7[92447]: 2024-04-14T14:31:47.926+0000 7f97c2275700 -1 osd.7 162 heartbeat_check: no reply from 172.21.15.69:6806 osd.0 since back 2024-04-14T14:30:59.635081+0000 front 2024-04-14T14:30:59.635113+0000 (oldest deadline 2024-04-14T14:31:25.535039+0000) 2024-04-14T14:31:48.094 INFO:journalctl@ceph.osd.7.smithi122.stdout:Apr 14 14:31:47 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-osd-7[92447]: 2024-04-14T14:31:47.926+0000 7f97c2275700 -1 osd.7 162 heartbeat_check: no reply from 172.21.15.69:6814 osd.1 since back 2024-04-14T14:31:06.635906+0000 front 2024-04-14T14:31:06.635997+0000 (oldest deadline 2024-04-14T14:31:30.135764+0000) 2024-04-14T14:31:48.095 INFO:journalctl@ceph.osd.7.smithi122.stdout:Apr 14 14:31:47 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-osd-7[92447]: 2024-04-14T14:31:47.926+0000 7f97c2275700 -1 osd.7 162 heartbeat_check: no reply from 172.21.15.69:6822 osd.2 since back 2024-04-14T14:31:16.338287+0000 front 2024-04-14T14:31:16.338264+0000 (oldest deadline 2024-04-14T14:31:39.238272+0000) 2024-04-14T14:31:48.095 INFO:journalctl@ceph.osd.7.smithi122.stdout:Apr 14 14:31:47 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-osd-7[92447]: 2024-04-14T14:31:47.926+0000 7f97c2275700 -1 osd.7 162 heartbeat_check: no reply from 172.21.15.69:6830 osd.3 since back 2024-04-14T14:31:19.238775+0000 front 2024-04-14T14:31:19.238839+0000 (oldest deadline 2024-04-14T14:31:45.138624+0000) 2024-04-14T14:31:48.668 INFO:journalctl@ceph.osd.6.smithi122.stdout:Apr 14 14:31:48 smithi122 podman[101729]: 2024-04-14 14:31:48.391321715 +0000 UTC m=+5.489326358 container died 72f85ece3e75f338fd86fb1d159cf293ad4896b0c47290c9634e9324ea9ae4d9 (image=quay.ceph.io/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888, name=ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-osd-6, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.build-date=20240403, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, RELEASE=main-a971fd5, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2) 2024-04-14T14:31:48.958 INFO:journalctl@ceph.osd.6.smithi122.stdout:Apr 14 14:31:48 smithi122 podman[101729]: 2024-04-14 14:31:48.842788956 +0000 UTC m=+5.940793601 container cleanup 72f85ece3e75f338fd86fb1d159cf293ad4896b0c47290c9634e9324ea9ae4d9 (image=quay.ceph.io/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888, name=ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-osd-6, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, RELEASE=main-a971fd5, GIT_CLEAN=True, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240403) 2024-04-14T14:31:48.958 INFO:journalctl@ceph.osd.6.smithi122.stdout:Apr 14 14:31:48 smithi122 bash[101729]: ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-osd-6 2024-04-14T14:31:49.344 INFO:journalctl@ceph.osd.6.smithi122.stdout:Apr 14 14:31:49 smithi122 podman[101748]: 2024-04-14 14:31:49.04448503 +0000 UTC m=+0.642631809 container remove 72f85ece3e75f338fd86fb1d159cf293ad4896b0c47290c9634e9324ea9ae4d9 (image=quay.ceph.io/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888, name=ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-osd-6, ceph=True, org.label-schema.vendor=CentOS, RELEASE=main-a971fd5, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240403, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-14T14:31:49.344 INFO:journalctl@ceph.osd.7.smithi122.stdout:Apr 14 14:31:48 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-osd-7[92447]: 2024-04-14T14:31:48.956+0000 7f97c2275700 -1 osd.7 162 heartbeat_check: no reply from 172.21.15.69:6806 osd.0 since back 2024-04-14T14:30:59.635081+0000 front 2024-04-14T14:30:59.635113+0000 (oldest deadline 2024-04-14T14:31:25.535039+0000) 2024-04-14T14:31:49.344 INFO:journalctl@ceph.osd.7.smithi122.stdout:Apr 14 14:31:48 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-osd-7[92447]: 2024-04-14T14:31:48.957+0000 7f97c2275700 -1 osd.7 162 heartbeat_check: no reply from 172.21.15.69:6814 osd.1 since back 2024-04-14T14:31:06.635906+0000 front 2024-04-14T14:31:06.635997+0000 (oldest deadline 2024-04-14T14:31:30.135764+0000) 2024-04-14T14:31:49.344 INFO:journalctl@ceph.osd.7.smithi122.stdout:Apr 14 14:31:48 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-osd-7[92447]: 2024-04-14T14:31:48.957+0000 7f97c2275700 -1 osd.7 162 heartbeat_check: no reply from 172.21.15.69:6822 osd.2 since back 2024-04-14T14:31:16.338287+0000 front 2024-04-14T14:31:16.338264+0000 (oldest deadline 2024-04-14T14:31:39.238272+0000) 2024-04-14T14:31:49.344 INFO:journalctl@ceph.osd.7.smithi122.stdout:Apr 14 14:31:48 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-osd-7[92447]: 2024-04-14T14:31:48.957+0000 7f97c2275700 -1 osd.7 162 heartbeat_check: no reply from 172.21.15.69:6830 osd.3 since back 2024-04-14T14:31:19.238775+0000 front 2024-04-14T14:31:19.238839+0000 (oldest deadline 2024-04-14T14:31:45.138624+0000) 2024-04-14T14:31:49.748 INFO:journalctl@ceph.osd.6.smithi122.stdout:Apr 14 14:31:49 smithi122 podman[101840]: 2024-04-14 14:31:49.358727967 +0000 UTC m=+0.017912793 image pull b96c509eef841460b3271f5fdd331860da87b004524f4ed59414aa3680183b53 quay.ceph.io/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888 2024-04-14T14:31:49.748 INFO:journalctl@ceph.osd.6.smithi122.stdout:Apr 14 14:31:49 smithi122 podman[101840]: 2024-04-14 14:31:49.462080504 +0000 UTC m=+0.121265316 container create 1d9a9d68216a2745153d55c964b38318f377360f97448b2adaf3cdf8e229012c (image=quay.ceph.io/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888, name=ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-osd-6-deactivate, org.label-schema.vendor=CentOS, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, RELEASE=main-a971fd5, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.build-date=20240403, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-14T14:31:49.748 INFO:journalctl@ceph.osd.6.smithi122.stdout:Apr 14 14:31:49 smithi122 podman[101840]: 2024-04-14 14:31:49.672773029 +0000 UTC m=+0.331957841 container init 1d9a9d68216a2745153d55c964b38318f377360f97448b2adaf3cdf8e229012c (image=quay.ceph.io/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888, name=ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-osd-6-deactivate, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, ceph=True, GIT_CLEAN=True, RELEASE=main-a971fd5, org.label-schema.build-date=20240403, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0) 2024-04-14T14:31:49.749 INFO:journalctl@ceph.osd.6.smithi122.stdout:Apr 14 14:31:49 smithi122 podman[101840]: 2024-04-14 14:31:49.67586793 +0000 UTC m=+0.335052743 container start 1d9a9d68216a2745153d55c964b38318f377360f97448b2adaf3cdf8e229012c (image=quay.ceph.io/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888, name=ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-osd-6-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, ceph=True, org.label-schema.license=GPLv2, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.vendor=CentOS, RELEASE=main-a971fd5, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240403, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD) 2024-04-14T14:31:49.749 INFO:journalctl@ceph.osd.6.smithi122.stdout:Apr 14 14:31:49 smithi122 podman[101840]: 2024-04-14 14:31:49.747817718 +0000 UTC m=+0.407002530 container attach 1d9a9d68216a2745153d55c964b38318f377360f97448b2adaf3cdf8e229012c (image=quay.ceph.io/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888, name=ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-osd-6-deactivate, org.label-schema.build-date=20240403, GIT_CLEAN=True, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, io.buildah.version=1.33.5, RELEASE=main-a971fd5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.vendor=CentOS, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0) 2024-04-14T14:31:50.094 INFO:journalctl@ceph.osd.6.smithi122.stdout:Apr 14 14:31:49 smithi122 podman[101840]: 2024-04-14 14:31:49.957839583 +0000 UTC m=+0.617024387 container died 1d9a9d68216a2745153d55c964b38318f377360f97448b2adaf3cdf8e229012c (image=quay.ceph.io/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888, name=ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-osd-6-deactivate, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, RELEASE=main-a971fd5, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.build-date=20240403, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2) 2024-04-14T14:31:50.094 INFO:journalctl@ceph.osd.7.smithi122.stdout:Apr 14 14:31:49 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-osd-7[92447]: 2024-04-14T14:31:49.979+0000 7f97c2275700 -1 osd.7 162 heartbeat_check: no reply from 172.21.15.69:6806 osd.0 since back 2024-04-14T14:30:59.635081+0000 front 2024-04-14T14:30:59.635113+0000 (oldest deadline 2024-04-14T14:31:25.535039+0000) 2024-04-14T14:31:50.094 INFO:journalctl@ceph.osd.7.smithi122.stdout:Apr 14 14:31:49 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-osd-7[92447]: 2024-04-14T14:31:49.979+0000 7f97c2275700 -1 osd.7 162 heartbeat_check: no reply from 172.21.15.69:6814 osd.1 since back 2024-04-14T14:31:06.635906+0000 front 2024-04-14T14:31:06.635997+0000 (oldest deadline 2024-04-14T14:31:30.135764+0000) 2024-04-14T14:31:50.095 INFO:journalctl@ceph.osd.7.smithi122.stdout:Apr 14 14:31:49 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-osd-7[92447]: 2024-04-14T14:31:49.979+0000 7f97c2275700 -1 osd.7 162 heartbeat_check: no reply from 172.21.15.69:6822 osd.2 since back 2024-04-14T14:31:16.338287+0000 front 2024-04-14T14:31:16.338264+0000 (oldest deadline 2024-04-14T14:31:39.238272+0000) 2024-04-14T14:31:50.095 INFO:journalctl@ceph.osd.7.smithi122.stdout:Apr 14 14:31:49 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-osd-7[92447]: 2024-04-14T14:31:49.979+0000 7f97c2275700 -1 osd.7 162 heartbeat_check: no reply from 172.21.15.69:6830 osd.3 since back 2024-04-14T14:31:19.238775+0000 front 2024-04-14T14:31:19.238839+0000 (oldest deadline 2024-04-14T14:31:45.138624+0000) 2024-04-14T14:31:50.579 DEBUG:teuthology.orchestra.run.smithi122:> sudo pkill -f 'journalctl -f -n 0 -u ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968@osd.6.service' 2024-04-14T14:31:50.620 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-14T14:31:50.620 INFO:tasks.cephadm.osd.6:Stopped osd.6 2024-04-14T14:31:50.620 INFO:tasks.cephadm.osd.7:Stopping osd.7... 2024-04-14T14:31:50.621 DEBUG:teuthology.orchestra.run.smithi122:> sudo systemctl stop ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968@osd.7 2024-04-14T14:31:50.844 INFO:journalctl@ceph.osd.7.smithi122.stdout:Apr 14 14:31:50 smithi122 systemd[1]: Stopping Ceph osd.7 for dc326fc2-fa65-11ee-bc8e-c7b262605968... 2024-04-14T14:31:51.344 INFO:journalctl@ceph.osd.7.smithi122.stdout:Apr 14 14:31:50 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-osd-7[92447]: 2024-04-14T14:31:50.936+0000 7f97c9c89700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.7 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-14T14:31:51.344 INFO:journalctl@ceph.osd.7.smithi122.stdout:Apr 14 14:31:50 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-osd-7[92447]: 2024-04-14T14:31:50.936+0000 7f97c9c89700 -1 osd.7 162 *** Got signal Terminated *** 2024-04-14T14:31:51.344 INFO:journalctl@ceph.osd.7.smithi122.stdout:Apr 14 14:31:50 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-osd-7[92447]: 2024-04-14T14:31:50.936+0000 7f97c9c89700 -1 osd.7 162 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-14T14:31:51.344 INFO:journalctl@ceph.osd.7.smithi122.stdout:Apr 14 14:31:50 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-osd-7[92447]: 2024-04-14T14:31:50.977+0000 7f97c2275700 -1 osd.7 162 heartbeat_check: no reply from 172.21.15.69:6806 osd.0 since back 2024-04-14T14:30:59.635081+0000 front 2024-04-14T14:30:59.635113+0000 (oldest deadline 2024-04-14T14:31:25.535039+0000) 2024-04-14T14:31:51.344 INFO:journalctl@ceph.osd.7.smithi122.stdout:Apr 14 14:31:50 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-osd-7[92447]: 2024-04-14T14:31:50.977+0000 7f97c2275700 -1 osd.7 162 heartbeat_check: no reply from 172.21.15.69:6814 osd.1 since back 2024-04-14T14:31:06.635906+0000 front 2024-04-14T14:31:06.635997+0000 (oldest deadline 2024-04-14T14:31:30.135764+0000) 2024-04-14T14:31:51.344 INFO:journalctl@ceph.osd.7.smithi122.stdout:Apr 14 14:31:50 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-osd-7[92447]: 2024-04-14T14:31:50.977+0000 7f97c2275700 -1 osd.7 162 heartbeat_check: no reply from 172.21.15.69:6822 osd.2 since back 2024-04-14T14:31:16.338287+0000 front 2024-04-14T14:31:16.338264+0000 (oldest deadline 2024-04-14T14:31:39.238272+0000) 2024-04-14T14:31:51.345 INFO:journalctl@ceph.osd.7.smithi122.stdout:Apr 14 14:31:50 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-osd-7[92447]: 2024-04-14T14:31:50.977+0000 7f97c2275700 -1 osd.7 162 heartbeat_check: no reply from 172.21.15.69:6830 osd.3 since back 2024-04-14T14:31:19.238775+0000 front 2024-04-14T14:31:19.238839+0000 (oldest deadline 2024-04-14T14:31:45.138624+0000) 2024-04-14T14:31:52.346 INFO:journalctl@ceph.osd.7.smithi122.stdout:Apr 14 14:31:51 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-osd-7[92447]: 2024-04-14T14:31:51.959+0000 7f97c2275700 -1 osd.7 162 heartbeat_check: no reply from 172.21.15.69:6806 osd.0 since back 2024-04-14T14:30:59.635081+0000 front 2024-04-14T14:30:59.635113+0000 (oldest deadline 2024-04-14T14:31:25.535039+0000) 2024-04-14T14:31:52.346 INFO:journalctl@ceph.osd.7.smithi122.stdout:Apr 14 14:31:51 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-osd-7[92447]: 2024-04-14T14:31:51.959+0000 7f97c2275700 -1 osd.7 162 heartbeat_check: no reply from 172.21.15.69:6814 osd.1 since back 2024-04-14T14:31:06.635906+0000 front 2024-04-14T14:31:06.635997+0000 (oldest deadline 2024-04-14T14:31:30.135764+0000) 2024-04-14T14:31:52.346 INFO:journalctl@ceph.osd.7.smithi122.stdout:Apr 14 14:31:51 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-osd-7[92447]: 2024-04-14T14:31:51.959+0000 7f97c2275700 -1 osd.7 162 heartbeat_check: no reply from 172.21.15.69:6822 osd.2 since back 2024-04-14T14:31:16.338287+0000 front 2024-04-14T14:31:16.338264+0000 (oldest deadline 2024-04-14T14:31:39.238272+0000) 2024-04-14T14:31:52.346 INFO:journalctl@ceph.osd.7.smithi122.stdout:Apr 14 14:31:51 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-osd-7[92447]: 2024-04-14T14:31:51.959+0000 7f97c2275700 -1 osd.7 162 heartbeat_check: no reply from 172.21.15.69:6830 osd.3 since back 2024-04-14T14:31:19.238775+0000 front 2024-04-14T14:31:19.238839+0000 (oldest deadline 2024-04-14T14:31:45.138624+0000) 2024-04-14T14:31:53.245 INFO:journalctl@ceph.osd.7.smithi122.stdout:Apr 14 14:31:52 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-osd-7[92447]: 2024-04-14T14:31:52.953+0000 7f97c2275700 -1 osd.7 162 heartbeat_check: no reply from 172.21.15.69:6806 osd.0 since back 2024-04-14T14:30:59.635081+0000 front 2024-04-14T14:30:59.635113+0000 (oldest deadline 2024-04-14T14:31:25.535039+0000) 2024-04-14T14:31:53.245 INFO:journalctl@ceph.osd.7.smithi122.stdout:Apr 14 14:31:52 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-osd-7[92447]: 2024-04-14T14:31:52.953+0000 7f97c2275700 -1 osd.7 162 heartbeat_check: no reply from 172.21.15.69:6814 osd.1 since back 2024-04-14T14:31:06.635906+0000 front 2024-04-14T14:31:06.635997+0000 (oldest deadline 2024-04-14T14:31:30.135764+0000) 2024-04-14T14:31:53.245 INFO:journalctl@ceph.osd.7.smithi122.stdout:Apr 14 14:31:52 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-osd-7[92447]: 2024-04-14T14:31:52.953+0000 7f97c2275700 -1 osd.7 162 heartbeat_check: no reply from 172.21.15.69:6822 osd.2 since back 2024-04-14T14:31:16.338287+0000 front 2024-04-14T14:31:16.338264+0000 (oldest deadline 2024-04-14T14:31:39.238272+0000) 2024-04-14T14:31:53.245 INFO:journalctl@ceph.osd.7.smithi122.stdout:Apr 14 14:31:52 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-osd-7[92447]: 2024-04-14T14:31:52.953+0000 7f97c2275700 -1 osd.7 162 heartbeat_check: no reply from 172.21.15.69:6830 osd.3 since back 2024-04-14T14:31:19.238775+0000 front 2024-04-14T14:31:19.238839+0000 (oldest deadline 2024-04-14T14:31:45.138624+0000) 2024-04-14T14:31:54.262 INFO:journalctl@ceph.osd.7.smithi122.stdout:Apr 14 14:31:53 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-osd-7[92447]: 2024-04-14T14:31:53.915+0000 7f97c2275700 -1 osd.7 162 heartbeat_check: no reply from 172.21.15.69:6806 osd.0 since back 2024-04-14T14:30:59.635081+0000 front 2024-04-14T14:30:59.635113+0000 (oldest deadline 2024-04-14T14:31:25.535039+0000) 2024-04-14T14:31:54.262 INFO:journalctl@ceph.osd.7.smithi122.stdout:Apr 14 14:31:53 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-osd-7[92447]: 2024-04-14T14:31:53.915+0000 7f97c2275700 -1 osd.7 162 heartbeat_check: no reply from 172.21.15.69:6814 osd.1 since back 2024-04-14T14:31:06.635906+0000 front 2024-04-14T14:31:06.635997+0000 (oldest deadline 2024-04-14T14:31:30.135764+0000) 2024-04-14T14:31:54.262 INFO:journalctl@ceph.osd.7.smithi122.stdout:Apr 14 14:31:53 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-osd-7[92447]: 2024-04-14T14:31:53.915+0000 7f97c2275700 -1 osd.7 162 heartbeat_check: no reply from 172.21.15.69:6822 osd.2 since back 2024-04-14T14:31:16.338287+0000 front 2024-04-14T14:31:16.338264+0000 (oldest deadline 2024-04-14T14:31:39.238272+0000) 2024-04-14T14:31:54.262 INFO:journalctl@ceph.osd.7.smithi122.stdout:Apr 14 14:31:53 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-osd-7[92447]: 2024-04-14T14:31:53.915+0000 7f97c2275700 -1 osd.7 162 heartbeat_check: no reply from 172.21.15.69:6830 osd.3 since back 2024-04-14T14:31:19.238775+0000 front 2024-04-14T14:31:19.238839+0000 (oldest deadline 2024-04-14T14:31:45.138624+0000) 2024-04-14T14:31:54.262 INFO:journalctl@ceph.osd.7.smithi122.stdout:Apr 14 14:31:53 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-osd-7[92447]: 2024-04-14T14:31:53.915+0000 7f97c2275700 -1 osd.7 162 heartbeat_check: no reply from 172.21.15.122:6804 osd.4 since back 2024-04-14T14:31:29.740040+0000 front 2024-04-14T14:31:29.739986+0000 (oldest deadline 2024-04-14T14:31:53.839984+0000) 2024-04-14T14:31:55.094 INFO:journalctl@ceph.osd.7.smithi122.stdout:Apr 14 14:31:54 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-osd-7[92447]: 2024-04-14T14:31:54.867+0000 7f97c2275700 -1 osd.7 162 heartbeat_check: no reply from 172.21.15.69:6806 osd.0 since back 2024-04-14T14:30:59.635081+0000 front 2024-04-14T14:30:59.635113+0000 (oldest deadline 2024-04-14T14:31:25.535039+0000) 2024-04-14T14:31:55.094 INFO:journalctl@ceph.osd.7.smithi122.stdout:Apr 14 14:31:54 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-osd-7[92447]: 2024-04-14T14:31:54.867+0000 7f97c2275700 -1 osd.7 162 heartbeat_check: no reply from 172.21.15.69:6814 osd.1 since back 2024-04-14T14:31:06.635906+0000 front 2024-04-14T14:31:06.635997+0000 (oldest deadline 2024-04-14T14:31:30.135764+0000) 2024-04-14T14:31:55.094 INFO:journalctl@ceph.osd.7.smithi122.stdout:Apr 14 14:31:54 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-osd-7[92447]: 2024-04-14T14:31:54.867+0000 7f97c2275700 -1 osd.7 162 heartbeat_check: no reply from 172.21.15.69:6822 osd.2 since back 2024-04-14T14:31:16.338287+0000 front 2024-04-14T14:31:16.338264+0000 (oldest deadline 2024-04-14T14:31:39.238272+0000) 2024-04-14T14:31:55.094 INFO:journalctl@ceph.osd.7.smithi122.stdout:Apr 14 14:31:54 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-osd-7[92447]: 2024-04-14T14:31:54.867+0000 7f97c2275700 -1 osd.7 162 heartbeat_check: no reply from 172.21.15.69:6830 osd.3 since back 2024-04-14T14:31:19.238775+0000 front 2024-04-14T14:31:19.238839+0000 (oldest deadline 2024-04-14T14:31:45.138624+0000) 2024-04-14T14:31:55.094 INFO:journalctl@ceph.osd.7.smithi122.stdout:Apr 14 14:31:54 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-osd-7[92447]: 2024-04-14T14:31:54.867+0000 7f97c2275700 -1 osd.7 162 heartbeat_check: no reply from 172.21.15.122:6804 osd.4 since back 2024-04-14T14:31:29.740040+0000 front 2024-04-14T14:31:29.739986+0000 (oldest deadline 2024-04-14T14:31:53.839984+0000) 2024-04-14T14:31:56.298 INFO:journalctl@ceph.osd.7.smithi122.stdout:Apr 14 14:31:55 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-osd-7[92447]: 2024-04-14T14:31:55.899+0000 7f97c2275700 -1 osd.7 162 heartbeat_check: no reply from 172.21.15.69:6806 osd.0 since back 2024-04-14T14:30:59.635081+0000 front 2024-04-14T14:30:59.635113+0000 (oldest deadline 2024-04-14T14:31:25.535039+0000) 2024-04-14T14:31:56.298 INFO:journalctl@ceph.osd.7.smithi122.stdout:Apr 14 14:31:55 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-osd-7[92447]: 2024-04-14T14:31:55.899+0000 7f97c2275700 -1 osd.7 162 heartbeat_check: no reply from 172.21.15.69:6814 osd.1 since back 2024-04-14T14:31:06.635906+0000 front 2024-04-14T14:31:06.635997+0000 (oldest deadline 2024-04-14T14:31:30.135764+0000) 2024-04-14T14:31:56.298 INFO:journalctl@ceph.osd.7.smithi122.stdout:Apr 14 14:31:55 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-osd-7[92447]: 2024-04-14T14:31:55.899+0000 7f97c2275700 -1 osd.7 162 heartbeat_check: no reply from 172.21.15.69:6822 osd.2 since back 2024-04-14T14:31:16.338287+0000 front 2024-04-14T14:31:16.338264+0000 (oldest deadline 2024-04-14T14:31:39.238272+0000) 2024-04-14T14:31:56.298 INFO:journalctl@ceph.osd.7.smithi122.stdout:Apr 14 14:31:55 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-osd-7[92447]: 2024-04-14T14:31:55.899+0000 7f97c2275700 -1 osd.7 162 heartbeat_check: no reply from 172.21.15.69:6830 osd.3 since back 2024-04-14T14:31:19.238775+0000 front 2024-04-14T14:31:19.238839+0000 (oldest deadline 2024-04-14T14:31:45.138624+0000) 2024-04-14T14:31:56.298 INFO:journalctl@ceph.osd.7.smithi122.stdout:Apr 14 14:31:55 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-osd-7[92447]: 2024-04-14T14:31:55.899+0000 7f97c2275700 -1 osd.7 162 heartbeat_check: no reply from 172.21.15.122:6804 osd.4 since back 2024-04-14T14:31:29.740040+0000 front 2024-04-14T14:31:29.739986+0000 (oldest deadline 2024-04-14T14:31:53.839984+0000) 2024-04-14T14:31:56.548 INFO:journalctl@ceph.osd.7.smithi122.stdout:Apr 14 14:31:56 smithi122 podman[101947]: 2024-04-14 14:31:56.297754436 +0000 UTC m=+5.462805576 container died 0c53c3cd2bc038aa31f168ce055728fa3b0174f20bda562f0065ca029adc0cb2 (image=quay.ceph.io/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888, name=ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-osd-7, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, io.buildah.version=1.33.5, ceph=True, org.label-schema.vendor=CentOS, RELEASE=main-a971fd5, org.label-schema.build-date=20240403, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True) 2024-04-14T14:31:56.830 INFO:journalctl@ceph.osd.7.smithi122.stdout:Apr 14 14:31:56 smithi122 podman[101947]: 2024-04-14 14:31:56.629412274 +0000 UTC m=+5.794463415 container cleanup 0c53c3cd2bc038aa31f168ce055728fa3b0174f20bda562f0065ca029adc0cb2 (image=quay.ceph.io/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888, name=ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-osd-7, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.build-date=20240403, CEPH_POINT_RELEASE=, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_CLEAN=True, RELEASE=main-a971fd5, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True) 2024-04-14T14:31:56.831 INFO:journalctl@ceph.osd.7.smithi122.stdout:Apr 14 14:31:56 smithi122 bash[101947]: ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-osd-7 2024-04-14T14:31:57.094 INFO:journalctl@ceph.osd.7.smithi122.stdout:Apr 14 14:31:56 smithi122 podman[101976]: 2024-04-14 14:31:56.830701601 +0000 UTC m=+0.527541153 container remove 0c53c3cd2bc038aa31f168ce055728fa3b0174f20bda562f0065ca029adc0cb2 (image=quay.ceph.io/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888, name=ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-osd-7, GIT_CLEAN=True, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, RELEASE=main-a971fd5, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240403) 2024-04-14T14:31:57.526 INFO:journalctl@ceph.osd.7.smithi122.stdout:Apr 14 14:31:57 smithi122 podman[102067]: 2024-04-14 14:31:57.231559559 +0000 UTC m=+0.103775121 container create ba1792b405d7a22c2c39ae542db1729030f652f9d2ab08c20f91ca72a335ef3d (image=quay.ceph.io/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888, name=ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-osd-7-deactivate, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.build-date=20240403, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , RELEASE=main-a971fd5, ceph=True) 2024-04-14T14:31:57.526 INFO:journalctl@ceph.osd.7.smithi122.stdout:Apr 14 14:31:57 smithi122 podman[102067]: 2024-04-14 14:31:57.145771186 +0000 UTC m=+0.017986748 image pull b96c509eef841460b3271f5fdd331860da87b004524f4ed59414aa3680183b53 quay.ceph.io/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888 2024-04-14T14:31:57.526 INFO:journalctl@ceph.osd.7.smithi122.stdout:Apr 14 14:31:57 smithi122 podman[102067]: 2024-04-14 14:31:57.450619995 +0000 UTC m=+0.322835558 container init ba1792b405d7a22c2c39ae542db1729030f652f9d2ab08c20f91ca72a335ef3d (image=quay.ceph.io/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888, name=ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-osd-7-deactivate, ceph=True, io.buildah.version=1.33.5, RELEASE=main-a971fd5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240403, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux ) 2024-04-14T14:31:57.526 INFO:journalctl@ceph.osd.7.smithi122.stdout:Apr 14 14:31:57 smithi122 podman[102067]: 2024-04-14 14:31:57.453603716 +0000 UTC m=+0.325819279 container start ba1792b405d7a22c2c39ae542db1729030f652f9d2ab08c20f91ca72a335ef3d (image=quay.ceph.io/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888, name=ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-osd-7-deactivate, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, ceph=True, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, RELEASE=main-a971fd5, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240403) 2024-04-14T14:31:57.844 INFO:journalctl@ceph.osd.7.smithi122.stdout:Apr 14 14:31:57 smithi122 podman[102067]: 2024-04-14 14:31:57.5255331 +0000 UTC m=+0.397748668 container attach ba1792b405d7a22c2c39ae542db1729030f652f9d2ab08c20f91ca72a335ef3d (image=quay.ceph.io/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888, name=ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-osd-7-deactivate, GIT_CLEAN=True, ceph=True, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, RELEASE=main-a971fd5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.build-date=20240403, org.label-schema.license=GPLv2) 2024-04-14T14:31:57.844 INFO:journalctl@ceph.osd.7.smithi122.stdout:Apr 14 14:31:57 smithi122 podman[102067]: 2024-04-14 14:31:57.751375107 +0000 UTC m=+0.623590663 container died ba1792b405d7a22c2c39ae542db1729030f652f9d2ab08c20f91ca72a335ef3d (image=quay.ceph.io/ceph-ci/ceph:a971fd5f5719038c70b817bafacd6d5400b1d888, name=ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-osd-7-deactivate, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240403, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, RELEASE=main-a971fd5, GIT_COMMIT=12717c0777377369ea674892da98b0d85250f5b0) 2024-04-14T14:31:58.318 DEBUG:teuthology.orchestra.run.smithi122:> sudo pkill -f 'journalctl -f -n 0 -u ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968@osd.7.service' 2024-04-14T14:31:58.359 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-14T14:31:58.359 INFO:tasks.cephadm.osd.7:Stopped osd.7 2024-04-14T14:31:58.359 INFO:tasks.cephadm.prometheus.a:Stopping prometheus.a... 2024-04-14T14:31:58.360 DEBUG:teuthology.orchestra.run.smithi122:> sudo systemctl stop ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968@prometheus.a 2024-04-14T14:31:58.594 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Apr 14 14:31:58 smithi122 systemd[1]: Stopping Ceph prometheus.a for dc326fc2-fa65-11ee-bc8e-c7b262605968... 2024-04-14T14:31:58.991 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Apr 14 14:31:58 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-prometheus-a[64422]: ts=2024-04-14T14:31:58.690Z caller=main.go:840 level=warn msg="Received SIGTERM, exiting gracefully..." 2024-04-14T14:31:58.992 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Apr 14 14:31:58 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-prometheus-a[64422]: ts=2024-04-14T14:31:58.690Z caller=main.go:864 level=info msg="Stopping scrape discovery manager..." 2024-04-14T14:31:58.992 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Apr 14 14:31:58 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-prometheus-a[64422]: ts=2024-04-14T14:31:58.690Z caller=main.go:878 level=info msg="Stopping notify discovery manager..." 2024-04-14T14:31:58.992 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Apr 14 14:31:58 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-prometheus-a[64422]: ts=2024-04-14T14:31:58.690Z caller=manager.go:988 level=info component="rule manager" msg="Stopping rule manager..." 2024-04-14T14:31:58.992 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Apr 14 14:31:58 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-prometheus-a[64422]: ts=2024-04-14T14:31:58.690Z caller=main.go:860 level=info msg="Scrape discovery manager stopped" 2024-04-14T14:31:58.992 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Apr 14 14:31:58 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-prometheus-a[64422]: ts=2024-04-14T14:31:58.690Z caller=main.go:874 level=info msg="Notify discovery manager stopped" 2024-04-14T14:31:58.992 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Apr 14 14:31:58 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-prometheus-a[64422]: ts=2024-04-14T14:31:58.690Z caller=manager.go:998 level=info component="rule manager" msg="Rule manager stopped" 2024-04-14T14:31:58.992 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Apr 14 14:31:58 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-prometheus-a[64422]: ts=2024-04-14T14:31:58.690Z caller=main.go:915 level=info msg="Stopping scrape manager..." 2024-04-14T14:31:58.992 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Apr 14 14:31:58 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-prometheus-a[64422]: ts=2024-04-14T14:31:58.691Z caller=main.go:907 level=info msg="Scrape manager stopped" 2024-04-14T14:31:58.992 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Apr 14 14:31:58 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-prometheus-a[64422]: ts=2024-04-14T14:31:58.691Z caller=notifier.go:606 level=info component=notifier msg="Stopping notification manager..." 2024-04-14T14:31:58.992 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Apr 14 14:31:58 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-prometheus-a[64422]: ts=2024-04-14T14:31:58.691Z caller=main.go:1135 level=info msg="Notifier manager stopped" 2024-04-14T14:31:58.993 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Apr 14 14:31:58 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-prometheus-a[64422]: ts=2024-04-14T14:31:58.691Z caller=main.go:1147 level=info msg="See you next time!" 2024-04-14T14:31:58.993 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Apr 14 14:31:58 smithi122 podman[102172]: 2024-04-14 14:31:58.701021499 +0000 UTC m=+0.123512193 container died 5c1f41d3bbe66fab3fa72f0ed86deea8cdd9704a9d1eb3aeca5a5d6ffcf194a5 (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-prometheus-a, maintainer=The Prometheus Authors ) 2024-04-14T14:31:59.324 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Apr 14 14:31:59 smithi122 podman[102172]: 2024-04-14 14:31:59.098511534 +0000 UTC m=+0.521002232 container cleanup 5c1f41d3bbe66fab3fa72f0ed86deea8cdd9704a9d1eb3aeca5a5d6ffcf194a5 (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-prometheus-a, maintainer=The Prometheus Authors ) 2024-04-14T14:31:59.324 INFO:journalctl@ceph.prometheus.a.smithi122.stdout:Apr 14 14:31:59 smithi122 bash[102172]: ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-prometheus-a 2024-04-14T14:31:59.457 DEBUG:teuthology.orchestra.run.smithi122:> sudo pkill -f 'journalctl -f -n 0 -u ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968@prometheus.a.service' 2024-04-14T14:31:59.492 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-14T14:31:59.492 INFO:tasks.cephadm.prometheus.a:Stopped prometheus.a 2024-04-14T14:31:59.492 DEBUG:teuthology.orchestra.run.smithi069:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid dc326fc2-fa65-11ee-bc8e-c7b262605968 --force --keep-logs 2024-04-14T14:32:01.269 INFO:journalctl@ceph.node-exporter.a.smithi069.stdout:Apr 14 14:32:01 smithi069 systemd[1]: Stopping Ceph node-exporter.a for dc326fc2-fa65-11ee-bc8e-c7b262605968... 2024-04-14T14:32:01.566 INFO:journalctl@ceph.node-exporter.a.smithi069.stdout:Apr 14 14:32:01 smithi069 podman[132000]: 2024-04-14 14:32:01.305831735 +0000 UTC m=+0.127199890 container died 0de0eb2171b9a7fe308795e7e66f4d26c5fb3cf113e7431fd200527a1a20f72c (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-node-exporter-a, maintainer=The Prometheus Authors ) 2024-04-14T14:32:01.927 INFO:journalctl@ceph.node-exporter.a.smithi069.stdout:Apr 14 14:32:01 smithi069 podman[132000]: 2024-04-14 14:32:01.684511881 +0000 UTC m=+0.505880048 container cleanup 0de0eb2171b9a7fe308795e7e66f4d26c5fb3cf113e7431fd200527a1a20f72c (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-node-exporter-a, maintainer=The Prometheus Authors ) 2024-04-14T14:32:01.927 INFO:journalctl@ceph.node-exporter.a.smithi069.stdout:Apr 14 14:32:01 smithi069 bash[132000]: ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-node-exporter-a 2024-04-14T14:32:02.234 INFO:journalctl@ceph.node-exporter.a.smithi069.stdout:Apr 14 14:32:01 smithi069 podman[132015]: 2024-04-14 14:32:01.927323863 +0000 UTC m=+0.622552902 container remove 0de0eb2171b9a7fe308795e7e66f4d26c5fb3cf113e7431fd200527a1a20f72c (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-node-exporter-a, maintainer=The Prometheus Authors ) 2024-04-14T14:32:02.234 INFO:journalctl@ceph.node-exporter.a.smithi069.stdout:Apr 14 14:32:01 smithi069 systemd[1]: ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968@node-exporter.a.service: Main process exited, code=exited, status=143/n/a 2024-04-14T14:32:02.234 INFO:journalctl@ceph.node-exporter.a.smithi069.stdout:Apr 14 14:32:02 smithi069 systemd[1]: ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968@node-exporter.a.service: Failed with result 'exit-code'. 2024-04-14T14:32:02.234 INFO:journalctl@ceph.node-exporter.a.smithi069.stdout:Apr 14 14:32:02 smithi069 systemd[1]: Stopped Ceph node-exporter.a for dc326fc2-fa65-11ee-bc8e-c7b262605968. 2024-04-14T14:32:02.234 INFO:journalctl@ceph.node-exporter.a.smithi069.stdout:Apr 14 14:32:02 smithi069 systemd[1]: ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968@node-exporter.a.service: Consumed 3.141s CPU time. 2024-04-14T14:32:02.521 INFO:journalctl@ceph.alertmanager.a.smithi069.stdout:Apr 14 14:32:02 smithi069 systemd[1]: Stopping Ceph alertmanager.a for dc326fc2-fa65-11ee-bc8e-c7b262605968... 2024-04-14T14:32:02.521 INFO:journalctl@ceph.alertmanager.a.smithi069.stdout:Apr 14 14:32:02 smithi069 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-alertmanager-a[78917]: ts=2024-04-14T14:32:02.480Z caller=main.go:583 level=info msg="Received SIGTERM, exiting gracefully..." 2024-04-14T14:32:02.816 INFO:journalctl@ceph.alertmanager.a.smithi069.stdout:Apr 14 14:32:02 smithi069 podman[132121]: 2024-04-14 14:32:02.521406466 +0000 UTC m=+0.148891550 container died 7524f106e7bd83a750de14b124ac87ed19c0457229c5bce7cd7ef398ce9dda93 (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-alertmanager-a, maintainer=The Prometheus Authors ) 2024-04-14T14:32:03.107 INFO:journalctl@ceph.alertmanager.a.smithi069.stdout:Apr 14 14:32:02 smithi069 podman[132121]: 2024-04-14 14:32:02.830643352 +0000 UTC m=+0.458128446 container cleanup 7524f106e7bd83a750de14b124ac87ed19c0457229c5bce7cd7ef398ce9dda93 (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-alertmanager-a, maintainer=The Prometheus Authors ) 2024-04-14T14:32:03.107 INFO:journalctl@ceph.alertmanager.a.smithi069.stdout:Apr 14 14:32:02 smithi069 bash[132121]: ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-alertmanager-a 2024-04-14T14:32:03.107 INFO:journalctl@ceph.alertmanager.a.smithi069.stdout:Apr 14 14:32:03 smithi069 podman[132135]: 2024-04-14 14:32:03.023383545 +0000 UTC m=+0.494429761 container remove 7524f106e7bd83a750de14b124ac87ed19c0457229c5bce7cd7ef398ce9dda93 (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-alertmanager-a, maintainer=The Prometheus Authors ) 2024-04-14T14:32:03.107 INFO:journalctl@ceph.alertmanager.a.smithi069.stdout:Apr 14 14:32:03 smithi069 podman[132135]: 2024-04-14 14:32:03.107303972 +0000 UTC m=+0.578350188 volume remove e1e5bcaf6468175db501fffa778d8fcc073ffaf5189911d804ed10b9c66c7466 2024-04-14T14:32:03.436 INFO:journalctl@ceph.alertmanager.a.smithi069.stdout:Apr 14 14:32:03 smithi069 systemd[1]: ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968@alertmanager.a.service: Deactivated successfully. 2024-04-14T14:32:03.436 INFO:journalctl@ceph.alertmanager.a.smithi069.stdout:Apr 14 14:32:03 smithi069 systemd[1]: Stopped Ceph alertmanager.a for dc326fc2-fa65-11ee-bc8e-c7b262605968. 2024-04-14T14:32:03.436 INFO:journalctl@ceph.alertmanager.a.smithi069.stdout:Apr 14 14:32:03 smithi069 systemd[1]: ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968@alertmanager.a.service: Consumed 1.457s CPU time. 2024-04-14T14:32:26.378 DEBUG:teuthology.orchestra.run.smithi122:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid dc326fc2-fa65-11ee-bc8e-c7b262605968 --force --keep-logs 2024-04-14T14:32:28.094 INFO:journalctl@ceph.node-exporter.b.smithi122.stdout:Apr 14 14:32:27 smithi122 systemd[1]: Stopping Ceph node-exporter.b for dc326fc2-fa65-11ee-bc8e-c7b262605968... 2024-04-14T14:32:28.509 INFO:journalctl@ceph.node-exporter.b.smithi122.stdout:Apr 14 14:32:28 smithi122 podman[102584]: 2024-04-14 14:32:28.195535059 +0000 UTC m=+0.146892585 container died 67877bf02d8e884b1552ebfe5d91321d808459c31fa43b600784b6cc960cbc27 (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-node-exporter-b, maintainer=The Prometheus Authors ) 2024-04-14T14:32:28.509 INFO:journalctl@ceph.node-exporter.b.smithi122.stdout:Apr 14 14:32:28 smithi122 podman[102584]: 2024-04-14 14:32:28.509484042 +0000 UTC m=+0.460841575 container cleanup 67877bf02d8e884b1552ebfe5d91321d808459c31fa43b600784b6cc960cbc27 (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-node-exporter-b, maintainer=The Prometheus Authors ) 2024-04-14T14:32:28.844 INFO:journalctl@ceph.node-exporter.b.smithi122.stdout:Apr 14 14:32:28 smithi122 bash[102584]: ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-node-exporter-b 2024-04-14T14:32:28.844 INFO:journalctl@ceph.node-exporter.b.smithi122.stdout:Apr 14 14:32:28 smithi122 podman[102598]: 2024-04-14 14:32:28.702628704 +0000 UTC m=+0.511836515 container remove 67877bf02d8e884b1552ebfe5d91321d808459c31fa43b600784b6cc960cbc27 (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-node-exporter-b, maintainer=The Prometheus Authors ) 2024-04-14T14:32:28.844 INFO:journalctl@ceph.node-exporter.b.smithi122.stdout:Apr 14 14:32:28 smithi122 systemd[1]: ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968@node-exporter.b.service: Main process exited, code=exited, status=143/n/a 2024-04-14T14:32:29.339 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Apr 14 14:32:29 smithi122 systemd[1]: Stopping Ceph grafana.a for dc326fc2-fa65-11ee-bc8e-c7b262605968... 2024-04-14T14:32:29.340 INFO:journalctl@ceph.node-exporter.b.smithi122.stdout:Apr 14 14:32:28 smithi122 systemd[1]: ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968@node-exporter.b.service: Failed with result 'exit-code'. 2024-04-14T14:32:29.340 INFO:journalctl@ceph.node-exporter.b.smithi122.stdout:Apr 14 14:32:28 smithi122 systemd[1]: Stopped Ceph node-exporter.b for dc326fc2-fa65-11ee-bc8e-c7b262605968. 2024-04-14T14:32:29.340 INFO:journalctl@ceph.node-exporter.b.smithi122.stdout:Apr 14 14:32:28 smithi122 systemd[1]: ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968@node-exporter.b.service: Consumed 3.137s CPU time. 2024-04-14T14:32:29.594 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Apr 14 14:32:29 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-grafana-a[66394]: logger=server t=2024-04-14T14:32:29.338784949Z level=info msg="Shutdown started" reason="System signal: terminated" 2024-04-14T14:32:29.594 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Apr 14 14:32:29 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-grafana-a[66394]: logger=ticker t=2024-04-14T14:32:29.339129391Z level=info msg=stopped last_tick=2024-04-14T14:32:20Z 2024-04-14T14:32:29.594 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Apr 14 14:32:29 smithi122 ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-grafana-a[66394]: logger=tracing t=2024-04-14T14:32:29.33913148Z level=info msg="Closing tracing" 2024-04-14T14:32:29.594 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Apr 14 14:32:29 smithi122 podman[102705]: 2024-04-14 14:32:29.349272815 +0000 UTC m=+0.123362964 container died 5685178e3eba0255fddc102e698d6275d51cf023aec40b98e018ed2016edb934 (image=quay.io/ceph/grafana:9.4.12, name=ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-grafana-a, maintainer=Grafana Labs ) 2024-04-14T14:32:30.090 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Apr 14 14:32:29 smithi122 podman[102705]: 2024-04-14 14:32:29.688589071 +0000 UTC m=+0.462679208 container cleanup 5685178e3eba0255fddc102e698d6275d51cf023aec40b98e018ed2016edb934 (image=quay.io/ceph/grafana:9.4.12, name=ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-grafana-a, maintainer=Grafana Labs ) 2024-04-14T14:32:30.090 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Apr 14 14:32:29 smithi122 bash[102705]: ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-grafana-a 2024-04-14T14:32:30.090 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Apr 14 14:32:29 smithi122 podman[102719]: 2024-04-14 14:32:29.865091182 +0000 UTC m=+0.517781068 container remove 5685178e3eba0255fddc102e698d6275d51cf023aec40b98e018ed2016edb934 (image=quay.io/ceph/grafana:9.4.12, name=ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968-grafana-a, maintainer=Grafana Labs ) 2024-04-14T14:32:30.576 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Apr 14 14:32:30 smithi122 systemd[1]: ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968@grafana.a.service: Deactivated successfully. 2024-04-14T14:32:30.576 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Apr 14 14:32:30 smithi122 systemd[1]: Stopped Ceph grafana.a for dc326fc2-fa65-11ee-bc8e-c7b262605968. 2024-04-14T14:32:30.577 INFO:journalctl@ceph.grafana.a.smithi122.stdout:Apr 14 14:32:30 smithi122 systemd[1]: ceph-dc326fc2-fa65-11ee-bc8e-c7b262605968@grafana.a.service: Consumed 11.656s CPU time. 2024-04-14T14:32:41.891 DEBUG:teuthology.orchestra.run.smithi069:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-04-14T14:32:41.928 DEBUG:teuthology.orchestra.run.smithi122:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-04-14T14:32:41.958 INFO:tasks.cephadm:Archiving crash dumps... 2024-04-14T14:32:41.961 DEBUG:teuthology.misc:Transferring archived files from smithi069:/var/lib/ceph/dc326fc2-fa65-11ee-bc8e-c7b262605968/crash to /home/teuthworker/archive/teuthology-2024-04-08_20:08:14-orch-main-distro-default-smithi/7646001/remote/smithi069/crash 2024-04-14T14:32:41.962 DEBUG:teuthology.orchestra.run.smithi069:> sudo tar c -f - -C /var/lib/ceph/dc326fc2-fa65-11ee-bc8e-c7b262605968/crash -- . 2024-04-14T14:32:42.039 INFO:teuthology.orchestra.run.smithi069.stderr:tar: /var/lib/ceph/dc326fc2-fa65-11ee-bc8e-c7b262605968/crash: Cannot open: No such file or directory 2024-04-14T14:32:42.040 INFO:teuthology.orchestra.run.smithi069.stderr:tar: Error is not recoverable: exiting now 2024-04-14T14:32:42.041 DEBUG:teuthology.misc:Transferring archived files from smithi122:/var/lib/ceph/dc326fc2-fa65-11ee-bc8e-c7b262605968/crash to /home/teuthworker/archive/teuthology-2024-04-08_20:08:14-orch-main-distro-default-smithi/7646001/remote/smithi122/crash 2024-04-14T14:32:42.042 DEBUG:teuthology.orchestra.run.smithi122:> sudo tar c -f - -C /var/lib/ceph/dc326fc2-fa65-11ee-bc8e-c7b262605968/crash -- . 2024-04-14T14:32:42.084 INFO:teuthology.orchestra.run.smithi122.stderr:tar: /var/lib/ceph/dc326fc2-fa65-11ee-bc8e-c7b262605968/crash: Cannot open: No such file or directory 2024-04-14T14:32:42.084 INFO:teuthology.orchestra.run.smithi122.stderr:tar: Error is not recoverable: exiting now 2024-04-14T14:32:42.085 INFO:tasks.cephadm:Checking cluster log for badness... 2024-04-14T14:32:42.085 DEBUG:teuthology.orchestra.run.smithi069:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/dc326fc2-fa65-11ee-bc8e-c7b262605968/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | head -n 1 2024-04-14T14:32:42.117 INFO:teuthology.orchestra.run.smithi069.stdout:2024-04-14T14:04:54.322417+0000 mon.a (mon.0) 948 : cluster [WRN] Health check failed: 1 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON) 2024-04-14T14:32:42.119 WARNING:tasks.cephadm:Found errors (ERR|WRN|SEC) in cluster log 2024-04-14T14:32:42.119 DEBUG:teuthology.orchestra.run.smithi069:> sudo egrep '\[SEC\]' /var/log/ceph/dc326fc2-fa65-11ee-bc8e-c7b262605968/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | head -n 1 2024-04-14T14:32:42.190 DEBUG:teuthology.orchestra.run.smithi069:> sudo egrep '\[ERR\]' /var/log/ceph/dc326fc2-fa65-11ee-bc8e-c7b262605968/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | head -n 1 2024-04-14T14:32:42.263 DEBUG:teuthology.orchestra.run.smithi069:> sudo egrep '\[WRN\]' /var/log/ceph/dc326fc2-fa65-11ee-bc8e-c7b262605968/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | head -n 1 2024-04-14T14:32:42.331 INFO:teuthology.orchestra.run.smithi069.stdout:2024-04-14T14:04:54.322417+0000 mon.a (mon.0) 948 : cluster [WRN] Health check failed: 1 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON) 2024-04-14T14:32:42.332 INFO:tasks.cephadm:Compressing logs... 2024-04-14T14:32:42.332 DEBUG:teuthology.orchestra.run.smithi069:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-04-14T14:32:42.375 DEBUG:teuthology.orchestra.run.smithi122:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-04-14T14:32:42.400 INFO:teuthology.orchestra.run.smithi069.stderr:find: ‘/var/log/rbd-target-api’gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-04-14T14:32:42.400 INFO:teuthology.orchestra.run.smithi069.stderr:: No such file or directory 2024-04-14T14:32:42.400 INFO:teuthology.orchestra.run.smithi069.stderr:gzip -5 --verbose -- /var/log/ceph/dc326fc2-fa65-11ee-bc8e-c7b262605968/ceph-mon.a.log 2024-04-14T14:32:42.400 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/cephadm.log: gzip -5 --verbose -- /var/log/ceph/dc326fc2-fa65-11ee-bc8e-c7b262605968/ceph.audit.log 2024-04-14T14:32:42.400 INFO:teuthology.orchestra.run.smithi069.stderr:gzip -5 --verbose -- /var/log/ceph/dc326fc2-fa65-11ee-bc8e-c7b262605968/ceph-osd.1.log 2024-04-14T14:32:42.400 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/dc326fc2-fa65-11ee-bc8e-c7b262605968/ceph-mon.a.log: /var/log/ceph/dc326fc2-fa65-11ee-bc8e-c7b262605968/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/dc326fc2-fa65-11ee-bc8e-c7b262605968/ceph-osd.3.log 2024-04-14T14:32:42.402 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/dc326fc2-fa65-11ee-bc8e-c7b262605968/ceph-osd.1.log: gzip -5 --verbose -- /var/log/ceph/dc326fc2-fa65-11ee-bc8e-c7b262605968/ceph-mgr.y.log 2024-04-14T14:32:42.402 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/dc326fc2-fa65-11ee-bc8e-c7b262605968/ceph-osd.3.log: gzip -5 --verbose -- /var/log/ceph/dc326fc2-fa65-11ee-bc8e-c7b262605968/tcmu-runner.log 2024-04-14T14:32:42.402 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/dc326fc2-fa65-11ee-bc8e-c7b262605968/ceph-mgr.y.log: gzip -5 --verbose -- /var/log/ceph/dc326fc2-fa65-11ee-bc8e-c7b262605968/ceph-osd.0.log 2024-04-14T14:32:42.403 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/dc326fc2-fa65-11ee-bc8e-c7b262605968/tcmu-runner.log: 88.8% -- replaced with /var/log/ceph/dc326fc2-fa65-11ee-bc8e-c7b262605968/tcmu-runner.log.gz 2024-04-14T14:32:42.403 INFO:teuthology.orchestra.run.smithi069.stderr:gzip -5 --verbose -- /var/log/ceph/dc326fc2-fa65-11ee-bc8e-c7b262605968/ceph-client.rgw.foo.smithi069.wesedz.log 2024-04-14T14:32:42.404 INFO:teuthology.orchestra.run.smithi122.stderr:find: ‘/var/log/rbd-target-api’gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-04-14T14:32:42.404 INFO:teuthology.orchestra.run.smithi122.stderr:: No such file or directory 2024-04-14T14:32:42.404 INFO:teuthology.orchestra.run.smithi122.stderr:gzip -5 --verbose -- /var/log/ceph/dc326fc2-fa65-11ee-bc8e-c7b262605968/ceph-osd.6.log 2024-04-14T14:32:42.404 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/cephadm.log: gzip -5 --verbose -- /var/log/ceph/dc326fc2-fa65-11ee-bc8e-c7b262605968/ceph.audit.log 2024-04-14T14:32:42.405 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/dc326fc2-fa65-11ee-bc8e-c7b262605968/ceph-osd.6.log: gzip -5 --verbose -- /var/log/ceph/dc326fc2-fa65-11ee-bc8e-c7b262605968/ceph-osd.4.log 2024-04-14T14:32:42.408 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/dc326fc2-fa65-11ee-bc8e-c7b262605968/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/dc326fc2-fa65-11ee-bc8e-c7b262605968/ceph-osd.7.log 2024-04-14T14:32:42.408 INFO:teuthology.orchestra.run.smithi122.stderr:gzip -5 --verbose -- /var/log/ceph/dc326fc2-fa65-11ee-bc8e-c7b262605968/ceph-client.rgw.foo.smithi122.jiobyv.log 2024-04-14T14:32:42.408 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/dc326fc2-fa65-11ee-bc8e-c7b262605968/ceph-osd.4.log: /var/log/ceph/dc326fc2-fa65-11ee-bc8e-c7b262605968/ceph-osd.7.log: gzip -5 --verbose -- /var/log/ceph/dc326fc2-fa65-11ee-bc8e-c7b262605968/ceph-osd.5.log 2024-04-14T14:32:42.408 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/dc326fc2-fa65-11ee-bc8e-c7b262605968/ceph-client.rgw.foo.smithi122.jiobyv.log: gzip -5 --verbose -- /var/log/ceph/dc326fc2-fa65-11ee-bc8e-c7b262605968/ceph-mgr.x.log 2024-04-14T14:32:42.408 INFO:teuthology.orchestra.run.smithi122.stderr: 75.9% -- replaced with /var/log/ceph/dc326fc2-fa65-11ee-bc8e-c7b262605968/ceph-client.rgw.foo.smithi122.jiobyv.log.gz 2024-04-14T14:32:42.408 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/dc326fc2-fa65-11ee-bc8e-c7b262605968/ceph-osd.5.log: gzip -5 --verbose -- /var/log/ceph/dc326fc2-fa65-11ee-bc8e-c7b262605968/ceph.cephadm.log 2024-04-14T14:32:42.408 INFO:teuthology.orchestra.run.smithi122.stderr:gzip -5 --verbose -- /var/log/ceph/dc326fc2-fa65-11ee-bc8e-c7b262605968/ceph.log 2024-04-14T14:32:42.408 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/dc326fc2-fa65-11ee-bc8e-c7b262605968/ceph-mgr.x.log: /var/log/ceph/dc326fc2-fa65-11ee-bc8e-c7b262605968/ceph.cephadm.log: 88.1% -- replaced with /var/log/ceph/dc326fc2-fa65-11ee-bc8e-c7b262605968/ceph.cephadm.log.gz 2024-04-14T14:32:42.408 INFO:teuthology.orchestra.run.smithi122.stderr:gzip -5 --verbose -- /var/log/ceph/dc326fc2-fa65-11ee-bc8e-c7b262605968/ceph-volume.log 2024-04-14T14:32:42.410 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/dc326fc2-fa65-11ee-bc8e-c7b262605968/ceph.log: 91.1% -- replaced with /var/log/ceph/dc326fc2-fa65-11ee-bc8e-c7b262605968/ceph.audit.log.gz 2024-04-14T14:32:42.410 INFO:teuthology.orchestra.run.smithi122.stderr:gzip -5 --verbose -- /var/log/ceph/dc326fc2-fa65-11ee-bc8e-c7b262605968/ceph-mon.b.log 2024-04-14T14:32:42.411 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/dc326fc2-fa65-11ee-bc8e-c7b262605968/ceph-volume.log: 88.8% -- replaced with /var/log/ceph/dc326fc2-fa65-11ee-bc8e-c7b262605968/ceph.log.gz 2024-04-14T14:32:42.412 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/dc326fc2-fa65-11ee-bc8e-c7b262605968/ceph-osd.0.log: 94.4% -- replaced with /var/log/ceph/dc326fc2-fa65-11ee-bc8e-c7b262605968/ceph.audit.log.gz 2024-04-14T14:32:42.413 INFO:teuthology.orchestra.run.smithi069.stderr:gzip -5 --verbose -- /var/log/ceph/dc326fc2-fa65-11ee-bc8e-c7b262605968/ceph.cephadm.log 2024-04-14T14:32:42.413 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/dc326fc2-fa65-11ee-bc8e-c7b262605968/ceph-client.rgw.foo.smithi069.wesedz.log: 76.5% -- replaced with /var/log/ceph/dc326fc2-fa65-11ee-bc8e-c7b262605968/ceph-client.rgw.foo.smithi069.wesedz.log.gz 2024-04-14T14:32:42.414 INFO:teuthology.orchestra.run.smithi069.stderr:gzip -5 --verbose -- /var/log/ceph/dc326fc2-fa65-11ee-bc8e-c7b262605968/ceph.log 2024-04-14T14:32:42.415 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/dc326fc2-fa65-11ee-bc8e-c7b262605968/ceph.cephadm.log: 92.3% -- replaced with /var/log/ceph/cephadm.log.gz 2024-04-14T14:32:42.415 INFO:teuthology.orchestra.run.smithi069.stderr:gzip -5 --verbose -- /var/log/ceph/dc326fc2-fa65-11ee-bc8e-c7b262605968/ceph-osd.2.log 2024-04-14T14:32:42.416 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/dc326fc2-fa65-11ee-bc8e-c7b262605968/ceph.log: 93.1% -- replaced with /var/log/ceph/dc326fc2-fa65-11ee-bc8e-c7b262605968/ceph.cephadm.log.gz 2024-04-14T14:32:42.416 INFO:teuthology.orchestra.run.smithi069.stderr:gzip -5 --verbose -- /var/log/ceph/dc326fc2-fa65-11ee-bc8e-c7b262605968/ceph-volume.log 2024-04-14T14:32:42.423 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/dc326fc2-fa65-11ee-bc8e-c7b262605968/ceph-osd.2.log: 93.8% -- replaced with /var/log/ceph/dc326fc2-fa65-11ee-bc8e-c7b262605968/ceph.log.gz 2024-04-14T14:32:42.423 INFO:teuthology.orchestra.run.smithi069.stderr:gzip -5 --verbose -- /var/log/ceph/dc326fc2-fa65-11ee-bc8e-c7b262605968/ceph-mon.c.log 2024-04-14T14:32:42.426 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/dc326fc2-fa65-11ee-bc8e-c7b262605968/ceph-mon.b.log: 91.2% -- replaced with /var/log/ceph/dc326fc2-fa65-11ee-bc8e-c7b262605968/ceph-volume.log.gz 2024-04-14T14:32:42.436 INFO:teuthology.orchestra.run.smithi122.stderr: 90.8% -- replaced with /var/log/ceph/cephadm.log.gz 2024-04-14T14:32:42.439 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/dc326fc2-fa65-11ee-bc8e-c7b262605968/ceph-volume.log: /var/log/ceph/dc326fc2-fa65-11ee-bc8e-c7b262605968/ceph-mon.c.log: 91.1% -- replaced with /var/log/ceph/dc326fc2-fa65-11ee-bc8e-c7b262605968/ceph-volume.log.gz 2024-04-14T14:32:42.517 INFO:teuthology.orchestra.run.smithi122.stderr: 89.8% -- replaced with /var/log/ceph/dc326fc2-fa65-11ee-bc8e-c7b262605968/ceph-mgr.x.log.gz 2024-04-14T14:32:42.905 INFO:teuthology.orchestra.run.smithi069.stderr: 89.2% -- replaced with /var/log/ceph/dc326fc2-fa65-11ee-bc8e-c7b262605968/ceph-mgr.y.log.gz 2024-04-14T14:32:43.009 INFO:teuthology.orchestra.run.smithi122.stderr: 92.5% -- replaced with /var/log/ceph/dc326fc2-fa65-11ee-bc8e-c7b262605968/ceph-mon.b.log.gz 2024-04-14T14:32:43.119 INFO:teuthology.orchestra.run.smithi069.stderr: 92.6% -- replaced with /var/log/ceph/dc326fc2-fa65-11ee-bc8e-c7b262605968/ceph-mon.c.log.gz 2024-04-14T14:32:44.028 INFO:teuthology.orchestra.run.smithi069.stderr: 91.0% -- replaced with /var/log/ceph/dc326fc2-fa65-11ee-bc8e-c7b262605968/ceph-mon.a.log.gz 2024-04-14T14:32:44.071 INFO:teuthology.orchestra.run.smithi122.stderr: 93.2% -- replaced with /var/log/ceph/dc326fc2-fa65-11ee-bc8e-c7b262605968/ceph-osd.6.log.gz 2024-04-14T14:32:44.208 INFO:teuthology.orchestra.run.smithi122.stderr: 93.4% -- replaced with /var/log/ceph/dc326fc2-fa65-11ee-bc8e-c7b262605968/ceph-osd.5.log.gz 2024-04-14T14:32:44.235 INFO:teuthology.orchestra.run.smithi122.stderr: 93.8% -- replaced with /var/log/ceph/dc326fc2-fa65-11ee-bc8e-c7b262605968/ceph-osd.7.log.gz 2024-04-14T14:32:44.284 INFO:teuthology.orchestra.run.smithi069.stderr: 93.0% -- replaced with /var/log/ceph/dc326fc2-fa65-11ee-bc8e-c7b262605968/ceph-osd.2.log.gz 2024-04-14T14:32:44.540 INFO:teuthology.orchestra.run.smithi069.stderr: 93.4% -- replaced with /var/log/ceph/dc326fc2-fa65-11ee-bc8e-c7b262605968/ceph-osd.1.log.gz 2024-04-14T14:32:44.631 INFO:teuthology.orchestra.run.smithi122.stderr: 93.6% -- replaced with /var/log/ceph/dc326fc2-fa65-11ee-bc8e-c7b262605968/ceph-osd.4.log.gz 2024-04-14T14:32:44.634 INFO:teuthology.orchestra.run.smithi122.stderr: 2024-04-14T14:32:44.634 INFO:teuthology.orchestra.run.smithi122.stderr:real 0m2.244s 2024-04-14T14:32:44.635 INFO:teuthology.orchestra.run.smithi122.stderr:user 0m8.010s 2024-04-14T14:32:44.635 INFO:teuthology.orchestra.run.smithi122.stderr:sys 0m0.303s 2024-04-14T14:32:44.706 INFO:teuthology.orchestra.run.smithi069.stderr: 93.4% -- replaced with /var/log/ceph/dc326fc2-fa65-11ee-bc8e-c7b262605968/ceph-osd.0.log.gz 2024-04-14T14:32:44.891 INFO:teuthology.orchestra.run.smithi069.stderr: 93.4% -- replaced with /var/log/ceph/dc326fc2-fa65-11ee-bc8e-c7b262605968/ceph-osd.3.log.gz 2024-04-14T14:32:44.893 INFO:teuthology.orchestra.run.smithi069.stderr: 2024-04-14T14:32:44.893 INFO:teuthology.orchestra.run.smithi069.stderr:real 0m2.506s 2024-04-14T14:32:44.893 INFO:teuthology.orchestra.run.smithi069.stderr:user 0m11.250s 2024-04-14T14:32:44.893 INFO:teuthology.orchestra.run.smithi069.stderr:sys 0m0.400s 2024-04-14T14:32:44.894 INFO:tasks.cephadm:Archiving logs... 2024-04-14T14:32:44.894 DEBUG:teuthology.misc:Transferring archived files from smithi069:/var/log/ceph to /home/teuthworker/archive/teuthology-2024-04-08_20:08:14-orch-main-distro-default-smithi/7646001/remote/smithi069/log 2024-04-14T14:32:44.895 DEBUG:teuthology.orchestra.run.smithi069:> sudo tar c -f - -C /var/log/ceph -- . 2024-04-14T14:32:46.379 DEBUG:teuthology.misc:Transferring archived files from smithi122:/var/log/ceph to /home/teuthworker/archive/teuthology-2024-04-08_20:08:14-orch-main-distro-default-smithi/7646001/remote/smithi122/log 2024-04-14T14:32:46.380 DEBUG:teuthology.orchestra.run.smithi122:> sudo tar c -f - -C /var/log/ceph -- . 2024-04-14T14:32:47.630 INFO:tasks.cephadm:Removing cluster... 2024-04-14T14:32:47.630 DEBUG:teuthology.orchestra.run.smithi069:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid dc326fc2-fa65-11ee-bc8e-c7b262605968 --force 2024-04-14T14:32:47.924 DEBUG:teuthology.orchestra.run.smithi122:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid dc326fc2-fa65-11ee-bc8e-c7b262605968 --force 2024-04-14T14:32:48.222 INFO:tasks.cephadm:Removing cephadm ... 2024-04-14T14:32:48.223 DEBUG:teuthology.orchestra.run.smithi069:> rm -rf /home/ubuntu/cephtest/cephadm 2024-04-14T14:32:48.238 DEBUG:teuthology.orchestra.run.smithi122:> rm -rf /home/ubuntu/cephtest/cephadm 2024-04-14T14:32:48.252 INFO:tasks.cephadm:Teardown complete 2024-04-14T14:32:48.253 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-04-14T14:32:48.263 INFO:teuthology.task.clock:Checking final clock skew... 2024-04-14T14:32:48.263 DEBUG:teuthology.orchestra.run.smithi069:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-14T14:32:48.280 DEBUG:teuthology.orchestra.run.smithi122:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-14T14:32:48.291 INFO:teuthology.orchestra.run.smithi069.stderr:bash: line 1: ntpq: command not found 2024-04-14T14:32:48.297 INFO:teuthology.orchestra.run.smithi069.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-04-14T14:32:48.297 INFO:teuthology.orchestra.run.smithi069.stdout:=============================================================================== 2024-04-14T14:32:48.297 INFO:teuthology.orchestra.run.smithi069.stdout:^* hv01.front.sepia.ceph.com 3 7 377 113 +854us[ +866us] +/- 48ms 2024-04-14T14:32:48.297 INFO:teuthology.orchestra.run.smithi069.stdout:^+ hv02.front.sepia.ceph.com 3 7 377 116 -1071us[-1060us] +/- 53ms 2024-04-14T14:32:48.297 INFO:teuthology.orchestra.run.smithi069.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 51 -206us[ -206us] +/- 70ms 2024-04-14T14:32:48.297 INFO:teuthology.orchestra.run.smithi069.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2024-04-14T14:32:48.306 INFO:teuthology.orchestra.run.smithi122.stderr:bash: line 1: ntpq: command not found 2024-04-14T14:32:48.310 INFO:teuthology.orchestra.run.smithi122.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-04-14T14:32:48.310 INFO:teuthology.orchestra.run.smithi122.stdout:=============================================================================== 2024-04-14T14:32:48.310 INFO:teuthology.orchestra.run.smithi122.stdout:^* hv01.front.sepia.ceph.com 3 7 377 48 +887us[ +896us] +/- 49ms 2024-04-14T14:32:48.310 INFO:teuthology.orchestra.run.smithi122.stdout:^+ hv02.front.sepia.ceph.com 3 7 377 118 -1009us[-1000us] +/- 53ms 2024-04-14T14:32:48.310 INFO:teuthology.orchestra.run.smithi122.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 49 -184us[ -175us] +/- 70ms 2024-04-14T14:32:48.310 INFO:teuthology.orchestra.run.smithi122.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2024-04-14T14:32:48.310 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-04-14T14:32:48.320 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-04-14T14:32:48.321 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-04-14T14:32:48.358 DEBUG:teuthology.orchestra.run.smithi069:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-04-14T14:32:48.452 DEBUG:teuthology.orchestra.run.smithi122:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-04-14T14:32:48.548 DEBUG:teuthology.orchestra.run.smithi069:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-04-14T14:32:48.579 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T14:32:48.580 DEBUG:teuthology.orchestra.run.smithi122:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-04-14T14:32:48.611 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T14:32:48.611 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-04-14T14:32:48.824 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-04-14T14:32:48.861 INFO:teuthology.task.internal:Duration was 2837.972057 seconds 2024-04-14T14:32:48.862 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-04-14T14:32:48.896 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-04-14T14:32:48.896 DEBUG:teuthology.orchestra.run.smithi069:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-04-14T14:32:48.901 DEBUG:teuthology.orchestra.run.smithi122:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-04-14T14:32:48.943 INFO:teuthology.orchestra.run.smithi069.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-14T14:32:48.945 INFO:teuthology.orchestra.run.smithi122.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-14T14:32:49.385 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-04-14T14:32:49.385 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi069.front.sepia.ceph.com 2024-04-14T14:32:49.385 DEBUG:teuthology.orchestra.run.smithi069:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-04-14T14:32:49.409 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi122.front.sepia.ceph.com 2024-04-14T14:32:49.410 DEBUG:teuthology.orchestra.run.smithi122:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-04-14T14:32:49.453 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-04-14T14:32:49.454 DEBUG:teuthology.orchestra.run.smithi069:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-04-14T14:32:49.457 DEBUG:teuthology.orchestra.run.smithi122:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-04-14T14:32:49.521 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-04-14T14:32:49.522 DEBUG:teuthology.orchestra.run.smithi069:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-04-14T14:32:49.525 DEBUG:teuthology.orchestra.run.smithi122:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-04-14T14:32:50.073 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-04-14T14:32:50.083 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-04-14T14:32:50.084 DEBUG:teuthology.orchestra.run.smithi069:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-04-14T14:32:50.111 DEBUG:teuthology.orchestra.run.smithi122:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-04-14T14:32:50.138 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-04-14T14:32:50.156 DEBUG:teuthology.orchestra.run.smithi069:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-04-14T14:32:50.158 DEBUG:teuthology.orchestra.run.smithi122:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-04-14T14:32:50.182 INFO:teuthology.orchestra.run.smithi069.stdout:kernel.core_pattern = core 2024-04-14T14:32:50.204 INFO:teuthology.orchestra.run.smithi122.stdout:kernel.core_pattern = core 2024-04-14T14:32:50.230 DEBUG:teuthology.orchestra.run.smithi069:> test -e /home/ubuntu/cephtest/archive/coredump 2024-04-14T14:32:50.281 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T14:32:50.282 DEBUG:teuthology.orchestra.run.smithi122:> test -e /home/ubuntu/cephtest/archive/coredump 2024-04-14T14:32:50.299 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-14T14:32:50.299 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-04-14T14:32:50.309 INFO:teuthology.task.internal:Transferring archived files... 2024-04-14T14:32:50.310 DEBUG:teuthology.misc:Transferring archived files from smithi069:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/teuthology-2024-04-08_20:08:14-orch-main-distro-default-smithi/7646001/remote/smithi069 2024-04-14T14:32:50.310 DEBUG:teuthology.orchestra.run.smithi069:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-04-14T14:32:50.374 DEBUG:teuthology.misc:Transferring archived files from smithi122:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/teuthology-2024-04-08_20:08:14-orch-main-distro-default-smithi/7646001/remote/smithi122 2024-04-14T14:32:50.375 DEBUG:teuthology.orchestra.run.smithi122:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-04-14T14:32:50.417 INFO:teuthology.task.internal:Removing archive directory... 2024-04-14T14:32:50.417 DEBUG:teuthology.orchestra.run.smithi069:> rm -rf -- /home/ubuntu/cephtest/archive 2024-04-14T14:32:50.420 DEBUG:teuthology.orchestra.run.smithi122:> rm -rf -- /home/ubuntu/cephtest/archive 2024-04-14T14:32:50.464 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-04-14T14:32:50.476 INFO:teuthology.task.internal:Not uploading archives. 2024-04-14T14:32:50.476 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-04-14T14:32:50.518 INFO:teuthology.task.internal:Tidying up after the test... 2024-04-14T14:32:50.519 DEBUG:teuthology.orchestra.run.smithi069:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-04-14T14:32:50.521 DEBUG:teuthology.orchestra.run.smithi122:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-04-14T14:32:50.533 INFO:teuthology.orchestra.run.smithi069.stdout: 265306 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Apr 14 14:32 /home/ubuntu/cephtest 2024-04-14T14:32:50.535 INFO:teuthology.orchestra.run.smithi122.stdout: 265307 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Apr 14 14:32 /home/ubuntu/cephtest 2024-04-14T14:32:50.536 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-04-14T14:32:50.552 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-04-14T14:32:50.620 INFO:teuthology.nuke:Checking targets against current locks 2024-04-14T14:32:50.648 DEBUG:teuthology.nuke:shortname: smithi069 2024-04-14T14:32:50.649 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-04-14T14:32:50.673 DEBUG:teuthology.nuke:shortname: smithi122 2024-04-14T14:32:50.673 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-04-14T14:32:50.677 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi069.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-04-08_20:08:14-orch-main-distro-default-smithi/7646001', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-14 13:37:21.333171', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBJQaGbtEJpZRB1Ob754T3lFCfEKlgZJ4BF5lU+jqhlWbWzgg878IXsm4Cg2WJdYA925nZ+I/soQQwsXyHC/Vu9g='} 2024-04-14T14:32:50.702 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi122.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-04-08_20:08:14-orch-main-distro-default-smithi/7646001', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-14 13:37:21.334942', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBHlnOxRm8xz7RAh1WohUOHVd7Uuy9q8Gxh3Mg8872snOHa/PP/YXWGJrm+Jtza+hn+pfOAX7nfikFSM/fKmXzfA='} 2024-04-14T14:32:50.726 INFO:teuthology.orchestra.console.smithi069:Power off 2024-04-14T14:32:50.726 DEBUG:teuthology.orchestra.console.smithi069:pexpect command: ipmitool -H smithi069.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-04-14T14:32:50.751 DEBUG:teuthology.orchestra.console.smithi069:power off output: Chassis Power Control: Down/Off 2024-04-14T14:32:50.752 DEBUG:teuthology.orchestra.console.smithi069:pexpect command: ipmitool -H smithi069.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-14T14:32:50.769 INFO:teuthology.orchestra.console.smithi122:Power off 2024-04-14T14:32:50.769 DEBUG:teuthology.orchestra.console.smithi122:pexpect command: ipmitool -H smithi122.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-04-14T14:32:50.779 DEBUG:teuthology.orchestra.console.smithi069:check power output: Chassis Power is on 2024-04-14T14:32:50.792 DEBUG:teuthology.orchestra.console.smithi122:power off output: Chassis Power Control: Down/Off 2024-04-14T14:32:50.793 DEBUG:teuthology.orchestra.console.smithi122:pexpect command: ipmitool -H smithi122.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-14T14:32:50.817 DEBUG:teuthology.orchestra.console.smithi122:check power output: Chassis Power is on 2024-04-14T14:32:54.780 DEBUG:teuthology.orchestra.console.smithi069:pexpect command: ipmitool -H smithi069.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-14T14:32:54.818 DEBUG:teuthology.orchestra.console.smithi122:pexpect command: ipmitool -H smithi122.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-14T14:32:54.891 DEBUG:teuthology.orchestra.console.smithi069:check power output: Chassis Power is on 2024-04-14T14:32:54.929 DEBUG:teuthology.orchestra.console.smithi122:check power output: Chassis Power is on 2024-04-14T14:32:58.896 DEBUG:teuthology.orchestra.console.smithi069:pexpect command: ipmitool -H smithi069.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-14T14:32:58.930 DEBUG:teuthology.orchestra.console.smithi122:pexpect command: ipmitool -H smithi122.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-14T14:32:59.006 DEBUG:teuthology.orchestra.console.smithi069:check power output: Chassis Power is off 2024-04-14T14:32:59.039 DEBUG:teuthology.orchestra.console.smithi122:check power output: Chassis Power is off 2024-04-14T14:32:59.107 INFO:teuthology.orchestra.console.smithi069:Power off completed 2024-04-14T14:32:59.140 INFO:teuthology.orchestra.console.smithi122:Power off completed 2024-04-14T14:32:59.244 INFO:teuthology.run:Summary data: description: orch/cephadm/upgrade/{1-start-distro/1-start-centos_9.stream 2-repo_digest/repo_digest 3-upgrade/staggered 4-wait 5-upgrade-ls agent/off mon_election/connectivity} duration: 2837.9720571041107 failure_reason: '"2024-04-14T14:04:54.322417+0000 mon.a (mon.0) 948 : cluster [WRN] Health check failed: 1 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON)" in cluster log' owner: scheduled_teuthology@teuthology success: false 2024-04-14T14:32:59.244 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-04-14T14:32:59.344 INFO:teuthology.run:FAIL